Now that the dependent spec is up for review, I've refreshed this
build system abstraction one.
Key changes:
- pep-426 -> wheel METADATA files as the distribution description interface
- dropped yaml for JSON as its in the stdlib
I've kept the indirection via a build system rather than adopting
setup.py as-is: I judge the risk for user confusion to be large enough
that its worth doing that - even though we recommend a setuptools
shim, forcing it on everyone is unpleasant.
commit 6de544a6e6aa2d0505e2ac8eb364b8b95e27790d
Author: Robert Collins <rbtcoll...@hp.com>
Date: Wed Oct 28 19:21:48 2015 +1300
PEP for build system abstraction.
diff --git a/build-system-abstraction.rst b/build-system-abstraction.rst
new file mode 100644
index 0000000..b8232c8
--- /dev/null
+++ b/build-system-abstraction.rst
@@ -0,0 +1,419 @@
+:PEP: XX
+:Title: Build system abstraction for pip/conda etc
+:Version: $Revision$
+:Last-Modified: $Date$
+:Author: Robert Collins <rbtcoll...@hp.com>,
+ Nathaniel Smith <n...@pobox.com>
+:BDFL-Delegate: Donald Stufft <don...@stufft.io>
+:Discussions-To: distutils-sig <distutils-sig@python.org>
+:Status: Draft
+:Type: Standards Track
+:Content-Type: text/x-rst
+:Created: 26-Oct-2015
+:Post-History: XX
+:Requires: The new dependency-specification PEP.
+
+
+Abstract
+========
+
+This PEP specifies a programmatic interface for pip [#pip]_ and other
+distribution or installation tools to use when working with Python
+source trees (both the developer tree - e.g. the git tree - and source
+distributions).
+
+The programmatic interface allows decoupling of pip from its current
+hard dependency on setuptools [#setuptools]_ able for two
+key reasons:
+
+1. It enables new build systems that may be much easier to use without
+ requiring them to even appear to be setuptools.
+
+2. It facilitates setuptools itself changing its user interface without
+ breaking pip, giving looser coupling.
+
+The programmatic interface also enables pip to install build time requirements
+for packages which is an important step in getting pip to full feature parity
+with the installation components of easy-install.
+
+As PEP-426 [#pep426]_ is draft, we cannot utilise the metadata format it
+defined. However PEP-427 wheels are in wide use and fairly well specified, so
+we have adopted the METADATA format from that for specifying distribution
+dependencies. However something was needed for communicating bootstrap
+requirements and build requirements - but a thin JSON schema is sufficient.
+
+Motivation
+==========
+
+There is significant pent-up frustration in the Python packaging ecosystem
+around the current lock-in between build system and pip. Breaking that lock-in
+is better for pip, for setuptools, and for other build systems like flit
+[#flit]_.
+
+Specification
+=============
+
+Overview
+--------
+
+Build tools will be located by reading a file ``pypa.json`` from the root
+directory of the source tree. That file describes how to get the build tool.
+The build tool is then self-describing. This avoids having stale descriptions
+of build tools encoded into sdists that have been uploaded to PyPI [#pypi]_.
+
+The interface involves executing processes rather than loading Python modules,
+because tools like pip will often interact with many different versions of the
+same build tool during a single invocation, and in-process APIs are much
+harder to manage in that situation.
+
+Process interface
+-----------------
+
+Where a process needs to be run, we need to be able to assemble the subprocess
+call to make. For this we use a simple command string with in-Python variable
+interpolation. Only variables defined in this PEP will be honoured: the use of
+additional variables is an error. Additional variables can only be added with
+a schema version increase. Basic variable expressions such as ``${PYTHON}``,
+``${PYTHON:-python}``, ``${PYTHON:+PYTHON -m coverage}`` can be used. An
+implementation of this is in shellvars [#shellvars]_.
+
+Processes will be run with the current working directory set to the root of
+the source tree.
+
+Available variables
+-------------------
+
+PYTHON
+ The Python interpreter in use. This is important to enable calling things
+ which are just Python entry points::
+
+ ${PYTHON} -m foo
+
+OUTPUT_DIR
+ Where to create requested output. If not set, outputting to the current
+ working directory is appropriate. This is required by ``pip`` which needs
+ to have wheels output in a known location.
+
+pypa.json
+---------
+
+The yaml file has the following schema. Extra keys are ignored.
+
+schema
+ The version of the schema. This PEP defines version 1.
+ Defaults to 1
+
+bootstrap-requires
+ A list of dependency specifications that must be installed before
+ running the build tool. For instance, if using flit, then the requirements
+ might be::
+
+ bootstrap-requires:
+ - flit
+
+build-tool
+ A command to run to query the build tool for its complete interface.
+ The build tool should output on stdout a build tool description JSON
+ document. Stdin may not be read from, and stderr may be handled however
+ the calling process desires.
+
+build tool description
+----------------------
+
+The build tool description schema. Extra keys are ignored.
+
+schema
+ The version of the schema. This PEP defines version 1.
+ Defaults to 1
+
+build-requires
+ Command to run to query build requirements. Build requirements are
+ returned as a JSON document with one key ``build_requires`` consisting of
+ a list of dependency specifications. Additional keys must be ignored.
+
+metadata
+ Command to run to generate metadata for the project. The metadata should
+ be output on stdout, stdin may not be consumed, and stderr handling is up
+ to the discretion of the calling process. The build-requires for the
+ project will be present in the Python environment when the metadata
+ command is run. pip would run metadata just once to determine what other
+ packages need to be downloaded and installed. The metadata is output as a
+ wheel METADATA file per PEP-427 [#pep427]_.
+
+ Note that the metadata generated by the metadata command, and the metadata
+ present in a generated wheel must be identical.
+
+wheel
+ Command to run to build a wheel of the project. OUTPUT_DIR will be set and
+ point to an existing directory where the wheel should be output. Stdin
+ may not be consumed, stdout and stderr handling is at the discretion of
+ the calling process. The build-requires for the project will be present in
+ the Python environment when the wheel command is run. Only one file
+ should be output - if more are output then pip would pick an arbitrary one
+ to consume.
+
+develop
+ Command to do an in-place 'development' installation of the project.
+ Stdin may not be consumed, stdout and stderr handling is at the discretion
+ of the calling process. The build-requires for the project will be
+ present in the Python environment when the develop command is run.
+
+ Not all build systems will be able to perform develop installs. If a build
+ system cannot do develop installs, then this key can be omitted.
+
+ Note that when it is omitted, commands like ``pip install -e foo``
+ will be unable to complete.
+
+provided-by
+ Optional distribution name that provides this build system.
+ This is used to facilitate caching the build tool description. If absent
+ then the build tool description cannot be cached, which will incure an
+ extra subprocess per package being built (to query the build tool).
+ Specifically, where the resolved bootstrap-requires results in the same
+ version of the named distribution being installed, the build tool
+ description is presumed to be identical.
+
+Python environments and hermetic builds
+---------------------------------------
+
+This specification does not prescribe whether builds should be hermetic or not.
+Existing build tools like setuptools will use installed versions of build time
+requirements (e.g. setuptools_scm) and only install other versions on version
+conflicts or missing dependencies. However its likely that better consistency
+can be created by always isolation builds and using only the
specified dependencies.
+
+However there are nuanced problems there - such as how can users force the
+avoidance of a bad version of a build requirement which meets some packages
+dependencies. Future PEPs may tackle this problem, but it is not currently in
+scope - it does not affect the metadata required to coordinate between build
+systems and things that need to do builds.
+
+Upgrades
+--------
+
+Both 'pypa.json' and the build tool description are versioned to permit future
+incompatible changes. There is a sequence dependency here.
+
+Upgrades to the schemas defined in this specification must proceed with the
+consumers first. To ensure that consumers should refuse to operate when
+'pypa.json' has a schema version that they do not recognise.
+
+Build tools listed in a 'pypa.json' with schema version 1 must not generate a
+build tool description with a version other than 1 (or absent, as the default
+is 1).
+
+Thus the sequence for upgrading either of schemas in a new PEP will be:
+
+1. Issue new PEP defining build tool description and 'pypa.json' schemas. The
+ 'pypa.yaml' schema version must change if either the 'pypa.yaml' schema has
+ changed or the build tool description schema has changed. The build tool
+ description schema version must change if the build tool description schema
+ has changed.
+2. Consumers (e.g. pip) implement support for the new schema version.
+3. Build tool authors implement the new schemas, and publish updated reference
+ 'pypa.json' files for their users. 4. Package authors opt into the new
+ schema when they are happy to introduce a dependency on the version of
+ 'pip' (and potentially other consumers) that introduced support for the new
+ schema version.
+
+The *same* process will take place for the initial deployment of this PEP:-
+the propogation of the capability to use this PEP without a `setuptools shim`_
+will be largely gated by the adoption rate of the first version of pip that
+supports it.
+
+Static metadata in sdists
+-------------------------
+
+This PEP does not tackle the current inability to trust static metadata in
+sdists. That is a separate problem to identifying and consuming the build
+system that is in use in a source tree, whether it came from an sdist or not.
+
+Handling of compiler options
+----------------------------
+
+Handling of different compiler options is out of scope for this specification.
+
+pip currently handles compiler options by appending user supplied strings to
+the command line it runs when running setuptools. This approach is sufficient
+to work with the build system interface defined in this PEP, with the
+exception that globally specified options will stop working globally as
+different build systems evolve. That problem can be solved in pip (or conda or
+other installers).
+
+In the long term, wheels should be able to express the difference between
+wheels built with one compiler or options vs another.
+
+Examples
+========
+
+An example 'pypa.json' for using flit::
+
+ bootstrap-requires:
+ - flit
+ build-tool: flit --dump-build-description
+
+When 'pip' reads this it would prepare an environment with flit in it and
+run `flit --dump-build-description` which would output something like::
+
+ build-requires: flit --dump-build-requires
+ metadata: flit --dump-metadata
+ provided-by: flit
+ wheel: flit wheel -d $OUTPUT_DIR
+
+The `--dump-` switches in this example would be needed to be added to
+flit (or someone else could write an adapter). Because flit doesn't have
+setup-requires support today, `--dump-build-requires` would just output a
+constant string::
+
+ {"build_requires": []}
+
+`--dump-metadata` would interrogate `flit.ini` and marshal the metadata into
+a wheel METADATA file and output that on stdout.
+
+flit wheel would need a `-d` parameter that tells it where to output the
+wheel (pip needs this).
+
+Backwards Compatibility
+=======================
+
+Older pips will remain unable to handle alternative build systems.
+This is no worse than the status quo - and individual build system
+projects can decide whether to include a shim ``setup.py`` or not.
+
+All existing build systems that can product wheels and do develop installs
+should be able to run under this abstraction and will only need a specific
+adapter for them constructed and published on PyPI.
+
+In the absence of a ``pypa.json`` file, tools like pip should assume a
+setuptools build system and use setuptools commands directly.
+
+
+Network effects
+---------------
+
+Projects that adopt build systems that are not setuptools compatible - that
+is that they have no setup.py, or the setup.py doesn't accept commands that
+existing tools try to use - will not be installable by those existing tools.
+
+Where those projects are used by other projects, this effect will cascade.
+
+In particular, because pip does not handle setup-requires today, any project
+(A) that adopts a setuptools-incompatible build system and is consumed as a
+setup-requirement by a second project (B) which has not itself transitioned to
+having a pypa.json will make B uninstallable by any version of pip. This is
+because setup.py in B will trigger easy-install when 'setup.py egg_info' is
+run by pip, and that will try and fail to install A.
+
+As such we recommend that tools which are currently used as setup-requires
+either ensure that they keep a `setuptools shim`_ or find their consumers and
+get them all to upgrade to the use of a `pypa.json` in advance of moving
+themselves. Pragmatically that is impossible, so the advice is to keep a
+setuptools shim indefinitely - both for projects like pbr, setuptools_scm and
+also projects like numpy.
+
+setuptools shim
+---------------
+
+It would be possible to write a generic setuptools shim that looks like
+``setup.py`` and under the hood uses ``pypa.json`` to drive the builds. This
+is not needed for pip to use the system, but would allow package authors to
+use the new features while still retaining compatibility with older pip
+versions.
+
+Rationale
+=========
+
+This PEP started with a long mailing list thread on distutils-sig [#thread]_.
+Subsequent to that a online meeting was held to debug all the positions folk
+had. Minutes from that were posted to the list [#minutes]_.
+
+This specification is a translation of the consensus reached there into PEP
+form, along with some arbitrary choices on the minor remaining questions.
+
+The basic heuristic for the design has to been to focus on introducing an
+abstraction without requiring development not strictly tied to the
+abstraction. Where the gap is small to improvements, or the cost of using the
+existing interface is very high, then we've taken on having the improvement as
+a dependency, but otherwise defered such to future iterations.
+
+We chose wheel METADATA files rather than defining a new specification,
+because pip can already handle wheel .dist-info directories which encode all
+the necessary data in a METADATA file. PEP-426 can't be used as it's still
+draft, and defining a new metadata format, while we should do that, is a
+separate problem. Using a directory on disk would not add any value to the
+interface (pip has to do that today due to limitations in the setuptools
+CLI).
+
+The use of 'develop' as a command is because there is no PEP specifying the
+interoperability of things that do what 'setuptools develop' does - so we'll
+need to define that before pip can take on the responsibility for doing the
+'develop' step. Once thats done we can issue a successor PEP to this one.
+
+The use of a command line API rather than a Python API is a little
+contentious. Fundamentally anything can be made to work, and Robert wants to
+pick something thats sufficiently lowest common denominator that
+implementation is straight forward on all sides. Picking a CLI for that makes
+sense because all build systems will need a CLI for end users to use anyway.
+
+The choice of JSON as a file format is a compromise between several
+constraints. Firstly there is no stdlib YAML interpreter, nor one for any of
+the other low-friction structured file formats. Secondly, INIParser is a poor
+format for a number of reasons, primarily that it has very minimal structure -
+but pip's maintainers are not fond of it. JSON is in the stdlib, has
+sufficient structure to permit embedding anything we want in future without
+requiring embedded DSL's.
+
+Donald suggested using ``setup.cfg`` and the existing setuptools command line
+rather than inventing something new. While that would permit interoperability
+with less visible changes, it requires nearly as much engineering on the pip
+side - looking for the new key in setup.cfg, implementing the non-installed
+environments to run the build in. And the desire from other build system
+authors not to confuse their users by delivering something that looks like but
+behaves quite differently to setuptools seems like a bigger issue than pip
+learning how to invoke a custom build tool.
+
+References
+==========
+
+.. [#pip] pip, the recommended installer for Python packages
+ (http://pip.readthedocs.org/en/stable/)
+
+.. [#setuptools] setuptools, the defacto Python package build system
+ (https://pythonhosted.org/setuptools/)
+
+.. [#flit] flit, a simple way to put packages in PyPI
+ (http://flit.readthedocs.org/en/latest/)
+
+.. [#pypi] PyPI, the Python Package Index
+ (https://pypi.python.org/)
+
+.. [#shellvars] Shellvars, an implementation of shell variable rules
for Python.
+ (https://github.com/testing-cabal/shellvars)
+
+.. [#pep426] PEP-426, Python distribution metadata.
+ (https://www.python.org/dev/peps/pep-0426/)
+
+.. [#pep427] PEP-427, Python distribution metadata.
+ (https://www.python.org/dev/peps/pep-0427/)
+
+.. [#thread] The kick-off thread.
+ (https://mail.python.org/pipermail/distutils-sig/2015-October/026925.html)
+
+.. [#minutes] The minutes.
+ (https://mail.python.org/pipermail/distutils-sig/2015-October/027214.html)
+
+Copyright
+=========
+
+This document has been placed in the public domain.
+
+
+
+..
+ Local Variables:
+ mode: indented-text
+ indent-tabs-mode: nil
+ sentence-end-double-space: t
+ fill-column: 70
+ coding: utf-8
+ End:
-Rob
--
Robert Collins <rbtcoll...@hp.com>
Distinguished Technologist
HP Converged Cloud
_______________________________________________
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
