On Sat, Jul 15, 2017 at 11:27 PM, Nick Coghlan <ncogh...@gmail.com> wrote: > On 16 July 2017 at 14:56, Nathaniel Smith <n...@pobox.com> wrote: >> But... that is not what the in-place/out-of-place distinction means in >> normal usage, it's not the distinction that any of those build systems >> you were surveying implement, and it's not the distinction specified >> in the current PEP text. >> >> If what we want is a distinction between "please give me a correct >> wheel" and "please give me a wheel but I don't care if it's broken", >> then wouldn't it make more sense to have a simple flag saying *that*? > > No, because pip *also* wants the ability to request that the backend > put the intermediate build artifacts in a particular place,
Say what? Where did they say this? I'm 99% sure this is just not true. > *and* > having that ability will likely prove beneficial given directory based > caching schemes in build automation pipelines (with BitBucket > Pipelines and OpenShift Image Streams being the two I'm personally > familiar with, but it's a logical enough approach to speeding up build > pipelines that I'm sure there are others). Yeah, this is a really neat idea! I'm genuinely enthusiastic about it. Which... is why I think this is a great target for a future PEP, that can adequately address the complications that the current PEP is skimming over. As I argued in a previous email, I think these pipelines would actually *prefer* that out-of-place build be an optional feature, but it's basically impossible to even have the discussion about what they want properly as part of the core PEP 517 discussion. > It just turns out that we can piggy back off that in-place/out-of-tree > distinction to *also* indicate how much the frontend cares about > consistency with sdist builds (which the PEP previously *didn't* say, > but explicit text along those lines was added as part of > https://github.com/python/peps/pull/310/files based on this latest > discussion). Okay, but then this is... bad. You're taking two unrelated distinctions (in-place/out-of-place and sloppy/precise) and smashing them together. In particular, one of the types of build that Donald has said that he considers "sloppy" and worries about avoiding is any kind of incremental build. So if we take Donald's concern and your new PEP text literally, then it rules out incremental out-of-place builds. But incremental out-of-place builds are exactly what you need for the build pipeline case that you're citing as a motivation for this feature. Your PEP is trying to do too many things at once, and that means it's going to do them poorly. >> And in what case would a frontend ever set this >> give_me_a_correct_wheel flag to False? > > When the frontend either genuinely doesn't care (hopefully rare, but > not inconceivable), or else when its building from an unpacked sdist > and hence can be confident that the artifacts will be consistent with > each other regardless of how the backend handles the situation > (expected to be very common, since it's the path that will be followed > for sdists published to PyPI, and when handed an arbitrary PEP 517 > source tree to build, pip will likely try "build_sdist -> in-place > build_wheel" first and only fall back to "out-of-tree build_wheel" if > the initial build_sdist call fails). Ah, the unpacked sdist case is a good point, I neglected that in my discussion of a possible setuptools build_wheel hook. But it's fine -- even if we say that the setuptools build_wheel hook has to produce an "sdist-consistent" wheel in all cases, then it can detect whether it's building from an unpacked sdist (e.g. by keying off the presence of PKG-INFO), and in that case it knows MANIFEST.in has already been taken into account and it can go straight to bdist_wheel. And in fact, this is what we *want* it to key off of, *not* the in-place/out-of-place thing. Consider Debian: they want to do out-of-place builds of unpacked sdists. They're working with pristine unpacked sdists, so they don't want to setuptools to go pack up a new sdist and then unpack it again for every out-of-place build. (In fact, this is probably a less-tested path than building directly from the unpacked sdist.) So if the point of the out-of-place build feature is that it's supposed to tell setuptools that it needs to go via sdist... then in this case it will do the wrong thing. Out-of-place and sdist-consistency are orthogonal concepts. > This is the main reason piggy backing off the in-place/out-of-tree > distinction works so well for this purpose: if the frontend just > unpacked an sdist into a build directory, then the most obvious thing > for it to do is to do an in-place build in that directory. > > It's only when the frontend is handed an arbitrary directory to build > that it doesn't know for sure is an unpacked sdist that the right > thing to do becomes markedly less clear, which is why we're offering > three options: > > 1. build_sdist -> unpack sdist -> in-place build_wheel (same as a PyPI > download) > 2. out-of-tree build_wheel (delegate the decision to the backend) > 3. in-place build_wheel (explicitly decide not to worry about it) > > We *think* 1 & 2 are going to be the most sensible options when given > an arbitrary directory, but allowing for 3 is a natural consequence of > supporting building from an unpacked sdist. > >> In particular, I think we could then say that since for >> distutils/setuptools, MANIFEST.in affects the sdist, then this >> language means that their build_wheel hook MUST also be sensitive to >> MANIFEST.in, and therefore it would need to be implemented internally >> as sdist->unpack->bdist_wheel. (Or if someone's ambitious we could >> even optimize that internally by skipping the pack/unpack step, which >> should make Donald happy :-).) And OTOH other backends that don't do >> this odd MANIFEST.in thing wouldn't have to worry about this. > > We don't want to get into the business of micromanaging how backends > work (that's how we got into the current mess with distutils & > setuptools), we just want to make it possible for frontend developers > and backend developers to collaborate effectively over time. So, uh... you're saying that you don't like this proposal, because it has too much micro-management: - backends are required to produce sdist-consistent wheels and instead prefer this proposal: - backends must support one configuration where they place intermediate artifacts in one specified place, and are not required to produce sdist-consistent wheels - backends must also support another configuration where they place intermediate artifacts in another place that's specified in a different way, and in this case they are required to produce sdist-consistent wheels I mean... the second proposal encompasses the first proposal, and then adds extra requirements, and those extra requirements are about fiddly internal things like where intermediate artifacts should be cached... surely the first proposal is less micromanage-y? > That > turned out to be the core problem with the previous > "prepare_input_for_build_wheel" hook: while it technically addressed > Paul & Donald's concerns as frontend developers, it placed too many > arbitrary constraints on how backend implementations worked and didn't > align well with the way full-fledged build systems handle requests for > out of tree builds. > > By contrast, `please do an out-of-tree build using this directory` > handles the same scenario (build_sdist failed, but build_wheel can > still be made to work) in a more elegant fashion by allowing the front > end to state *what* it wants (i.e. something that gets as close as is > practical to the "build_sdist -> unpack sdist -> in-place build_wheel" > path given the limitations of the current environment), We totally agree the frontend should state "*what* it wants". This is exactly my problem with the PEP -- it doesn't do this! Instead, it has frontends specify something they don't care about (in-place/out-of-place) and then overloads that to mean this unrelated thing. My draft at the start of this thread was exactly designed by trying to find the intersection of (1) satisfying everyone's requirements, (2) restricting myself to operations whose semantics I could define in a generic, future-proof way. > while > delegating the precise details of *how* that is done to the backend. > > Some backends will implement those requests literally as "build_sdist > -> unpack sdist -> in-place build_wheel" (which is what the example in > the PEP does). > > Some won't offer any more artifact consistency guarantees than they do > for the in-place build_wheel case (this is the current plan for flit > and presumably for enscons as well) > > Some will be able to take their sdist manifest data into account > without actually preparing a full sdist archive (this might make sense > for a setuptools/distutils backend). > > All 3 of those options are fine from an ecosystem level perspective, > and are ultimately a matter to be resolved between backend developers > and their users. > > So we're now at a point where: > > - key frontend developers agree the current spec allows them to > request what they need/want from backends > - key backend developers agree the current spec can be readily implemented > - there are still some open questions around exactly when its > reasonable for hooks to fail, but we're only going to answer those > through real world experience, not further hypothetical speculation > - we have the ability to evolve the API in the future if some aspects > turn out to be particularly problematic > > That means I'm going to *explicitly* ask that you accept that the PEP > is going to be accepted, and it's going to be accepted with the API in > its current form, even if you personally don't agree with our > reasoning for all of the technical details. If your level of concern > around the build_directory parameter specifically is high enough that > you don't want to be listed as a co-author on PEP 517 anymore, then > that's entirely reasonable (we can add a separate Acknowledgments > section to recognise your significant input to the process without > implying your endorsement of the final result), but as long as the > accepted API ends up being supported in at least pip, flit, and > enscons, it honestly doesn't really matter all that much in practice > what the rest of us think of the design (we're here as design > advisors, rather than being the ones that will necessarily need to > cope with the bug reports arising from any interoperability > challenges). Clearly neither of us have convinced each other here. If you want to take over authorship of PEP 517, that's fine -- it's basically just acknowledging the current reality. But in that case I think you should step down as BDFL-delegate; even Guido doesn't accept his own PEPs. Here's another possible way forward: you mold PEP 517 into what you think is best, I take my text from the beginning of this thread and base a new PEP off it, and then Donald gets stuck as BDFL-delegate and has to pick. (Sorry Donald.) What do you think? It would at least have the advantage that everyone else gets a chance to catch up and look things over -- I think at the moment it's basically only you and me who actually have strong opinions, and everyone else is barely following. -n -- Nathaniel J. Smith -- https://vorpus.org _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
