On 17 July 2017 at 15:41, Thomas Kluyver <tho...@kluyver.me.uk> wrote:
> [I don't have time to properly read & respond now, but I wanted to
> suggest something]
> Would it be better if we said that build_directory must always be
> specified, i.e. passing None is invalid?
> A) Nathaniel has quite rightly expressed concern over requiring two
> different code paths. While I think the extra work is less than his
> wording implies, it is an extra case for things to go wrong.
I don't think that makes sense, as we want a clear "you don't need to
worry about sdist consistency" code path for the normal case where the
frontend has already ensured that the source directory is an unpacked
sdist (hence ensuring that the built wheel will be consistent with
that sdist as long as the backend restricts itself to using the
contents of that directory as inputs).
This core requirement seems to have gotten lost for a lot of folks due
to the recent focus on how best to allow frontends like pip to handle
the "Here's an arbitrary directory, please build it" case, when we
expect the typical end user cases to be ones where the frontend
already knows it has a pristine unpacked sdist and simply needs the
backend to turn that directory into an installable wheel file (either
because it download the sdist from PyPI, or because it just created it
While it would be lovely to be able to go with the "let's just pretend
that problem doesn't exist for now" approach for the arbitrary
directory case, I think you're right that it's a common enough
scenario that we can't ignore it entirely, even in the first iteration
of the API specification - to enable and encourage adoption, we need
to give frontends and backends a reasonable way to exchange
information about what is going on, and preferably do so in a way that
aligns nicely with design concepts that already exist as part of
full-fledged build systems.
That last point is why I specifically *don't* want an idiosyncratic
Python-specific flag as part of the API, and would prefer instead to
address the need as a conventional expectation around the behaviour of
out-of-tree builds based on exemplars like flit and enscons, similar
to the way the semantics of operator overloading are handled in Python
itself (i.e. technically you can do whatever you want with operator
overloading, but if your overloading choices are inconsistent with
other conventional uses of those symbols and hence cause problems for
people and makes your API confusing or unreliable, they may decide not
to use your library because of it).
In-place/out-of-tree is an inherently meaningful concept for arbitrary
build backends, and something that they can't automatically choose for
themselves (since they need the target directory as a user provided
configuration setting). By contrast, whether the starting directory
for build_wheel is an unpacked sdist or not is something a backend
should be able to work out for itself (e.g. by checking for PKG-INFO).
That said, I *might* accept an explicit "unpacked_sdist=True/False"
flag if someone can provide a compelling justification for why it
isn't redundant with "build_directory=None" and/or backends checking
directly for PKG-INFO (or other files they add to the sdist) in the
source directory, and an explanation of what backends should do for
"unpacked_sdist=False, build_directory=None" that they wouldn't do for
the "unpacked_sdist=True, build_directory=None" case. Unlike
build_directory, they certainly couldn't pass such a flag straight on
to a general purpose build backend (or not pass it, in the
And if the proposal is instead for an unpacked_sdist flag as a
*replacement* for the currently proposed build_directory parameter...
well, no. Out-of-tree builds are useful for a lot more than just pip
indicating whether or not it failed to build an sdist, and if the
claim is "there's no evidence that out-of-tree builds are sufficiently
useful to include them in the specification", then I'll point you to
the laments of everyone trying to manage cross-compiling for multiple
platform ABIs without them, both inside and outside the Python
community, as well as the efforts to design sensible ecosystem
independent intermediate artifact caching strategies for automated
build pipelines. Those reasons aren't necessarily part of why the
parameter was initially proposed, but they *are* a big part of why I
was an instant convert to the idea once the suggestion was made (and
why I've actively opposed the notion that the concept hasn't
demonstrated its utility well enough to be incorporated).
> B) The current spec does another 'semantics not specified' for in place
> builds. This should make us uncomfortable - we're asking backends to
> implement something without telling them what. When we did something
> similar for editable installs, we ended up pulling it out again.
Technically the "semantics unspecified" only applies when the source
directory is something other than an unpacked sdist, since such a
build may differ from building via the sdist in unspecified, backend
dependent ways. (This isn't new, it's acknowledging that the handling
of dirty source directories is an inherently backend dependent
behaviour, and that the main problem that pip currently has is with
the way that setuptools/distutils/setup.py in particular deal with it,
*not* necessarily with the way future backends will handle it in
The semantics specification we add for the out-of-tree case is that it
should be as close to "build sdist -> unpack sdist -> in-place build
wheel" as practical in the given environment, while still leaving the
specifics of how that is implemented, and how close it gets to
actually matching the build-via-sdist behaviour up to the backend.
For example, you're free to decide as flit's developer that since
frontends are free to try the "build_sdist" path for themselves, you
*won't* implicitly implement that for out-of-tree builds in flit, and
will instead handle out-of-tree builds exactly the same way as you
handle in-place builds. Whether or not that is sufficient in practice
will then be an implementation discussion between you and your users
(and *maybe* frontend developers if they end up fielding a lot of
flit-specific bug reports), *not* an API specification discussion at
the PyPA/distutils-sig level.
For enscons, Daniel may decide to handle out-of-tree builds by passing
"build_directory" down to the Scons backend as the target "variant
directory", and otherwise not do anything special. Again, an entirely
acceptable way of handling the proposed API specification.
I'm entirely open to the idea that this part of the PEP still needs
clarification - what's there attempts to explain why I'm willing to
accept it as BDFL-Delegate, and why the pip developers are willing to
accept it as sufficient for their needs, but it's also important that
other frontend developers are able to understand how to use it, and
backend developers are able to understand the anticipated expectations
around how they implement it.
> C) We thought we were adding this to satisfy people who want incremental
> builds, but it now looks like we were conflating two different things,
> and this is not actually a good answer for those use cases.
It does give folks a backend independent way to request incremental
builds, but the main use case in practice is as the baseline essential
build step of going from an unpacked sdist directory to a wheel file.
> D) The frontend can specify build_directory=source_dir/build to build in
> a subdirectory of the source tree.
While this is true, I don't think it needs to be called out explicitly
in the PEP.
> E) When we do work out the need and the semantics for in place builds,
> we can write another PEP adding an optional hook for that.
The minimal specification for in-place builds is "Whatever you would
do to build a wheel file from an unpacked sdist". The unspecified part
is how that behaviour may change in the case where the source
directory *isn't* an unpacked sdist and hence may contain other files.
Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia
Distutils-SIG maillist - Distutils-SIG@python.org