On 13 June 2017 at 08:55, Nathaniel Smith <n...@pobox.com> wrote: > On Mon, Jun 12, 2017 at 3:34 PM, Paul Moore <p.f.mo...@gmail.com> wrote: >> But honestly, I think we're at the point where someone just needs to >> make a decision - there's very little compelling evidence either way. > > I was the original PEP author, but Thomas has mostly taken it over at > this point, so I'm not sure how much you should listen to me :-). But > if you pointed at me and told me to make some decisions, then right > now this is what I'd do: > > 1) Go back to having the wheel generation hook generate a .whl file > Rationale: the optimization benefits of generating an unpacked wheel > are unclear, but we know that reproducible builds are important, and > filename encoding is tricky and important, and that having a common > well-understood standard with tooling around it is important, and on > those axes .whl unambiguously wins. And if there do later turn out to > be compelling optimization benefits to generating unpacked wheels, > then we can add an optional generate_unpacked_wheel hook in the > future.
Despite being the one to originally propose standardisation on passing directory paths around, I'm starting to lean back towards this approach. My rationale for this doesn't really have a lot to do with topics we've discussed so far, and instead asks the question: what would work best for an installation frontend that wanted to keep the actual build tools off the system doing the installation, while still allowing for transparent "from sdist" installations? And that's where cross-platform archive formats really shine: since they're incredibly convenient for network transfers, they make fewer implicit assumptions about *where* the work is being done. Whereas if we rely on directories in the baseline interface specification, we're making a lot of assumptions about how all future frontends are going to work. It should also be relatively straightforward to add cross-platform validators to frontends that check that archives are well-formed, but it's harder to write such validators that work cross-platform on arbitrary directories. > 2) Specify that the wheel generation hook, metadata hook, and sdist > hook return the name of path that they created as a unicode string. > Rationale: fixes a point of ambiguity in the current spec. And still leaves the door open to supporting multiple wheels in the future by returning a list instead of string. > 3) Go back to having the sdist generation hook generate an archive > file instead of an unpacked directory > Rationale: Essentially the same as for point (1). The additional > complication here is that we know that pip plans to use this as part > of its standard build-from-unpacked-source pipeline, so if we're > trying to optimize this case for developers with their fast SSD drives > etc. then the compress/decompress cycle actually does matter. I think you also raise a good point in bringing up the "First make it work, then make it right, then make it fast" priority order. We're currently in the "make it right" phase, and I think you're correct that that's much easier to achieve cross platform with archive based data exchange - otherwise we run the risk of running into surprises with NTFS, HDFS+, etc (let alone folks running builds over NFS or CIFS). So the core "make it right" API would be "build_sdist" and "build_wheel" as mandatory backend hooks, with "prepare_wheel_metadata" and "prepare_build_files" as optional "make it faster" helpers. We'd then reserve "prepare_sdist_content" and "prepare_wheel_content" as possible future "make it faster" APIs, rather than adding them now. > 4) Specify that the sdist archive file must be .tar.gz > Rationale: we only need one and it reduces variation, and dstufft > likes .tar.gz better than .zip. +1 > 5) Drop the prepare_files hook > Rationale: it's purpose is somewhat unclear at this point, and it > gives up the main advantage of going through sdist (the guarantee that > building from sdist and building from unpacked tree give the same > results) while still having the main disadvantages (you have to copy > everything and lose incremental rebuilds). The main reason this made it into the PEP is that build_sdist may have additional dependencies that prepare_build_files doesn't (e.g. Thomas has indicated that flit needs to call out to VCS tools to build a full sdist, but can export its own input files directly). So while it's technically a "make it faster" hook rather than an essential "make it right" hook, I think just smoothing out the interaction between pip & flit specifically provides sufficient value to make it worth including in the initial version of the specification. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
