Re: [Distutils] PEP 517: Open questions around artifact export directories

On 16 June 2017 at 14:25, Nathaniel Smith wrote: > On Thu, Jun 15, 2017 at 7:58 PM, Donald Stufft wrote: >> >> >> On Jun 15, 2017, at 10:51 PM, Nathaniel Smith wrote: >> >> I can see some benefit to standardizing on a single format instead of >>

Re: [Distutils] PEP 517: Open questions around artifact export directories

On Thu, Jun 15, 2017 at 7:58 PM, Donald Stufft wrote: > > On Jun 15, 2017, at 10:51 PM, Nathaniel Smith wrote: > > I can see some benefit to standardizing on a single format instead of > making every backend author learn e.g. the weird quirks required to > get

Re: [Distutils] PEP 517: Open questions around artifact export directories

The format you want is zstandard or brotli. Maybe fastly be interested in sponsoring bandwidth efficient file formats? :-) It would be cool to have a way to do signatures and hashing before compression. On Thu, Jun 15, 2017, 22:59 Donald Stufft wrote: > > On Jun 15, 2017, at

Re: [Distutils] PEP 517: Open questions around artifact export directories

> On Jun 15, 2017, at 10:51 PM, Nathaniel Smith wrote: > > I can see some benefit to standardizing on a single format instead of > making every backend author learn e.g. the weird quirks required to > get unicode filenames correct in two different formats, and I'm not > sure why

Re: [Distutils] PEP 517: Open questions around artifact export directories

On Thu, Jun 15, 2017 at 1:55 PM, Brett Cannon wrote: > Then shouldn't we be pushing for .tar.xz instead? (The Rust community is > actually moving to .tar.xz for distributing Rust itself: > https://users.rust-lang.org/t/rustup-1-4-0-released/11268 ; I don't know > what their

Re: [Distutils] PEP 517: Open questions around artifact export directories

> On Jun 15, 2017, at 5:05 PM, Paul Moore wrote: > > On 15 June 2017 at 21:49, Daniel Holth wrote: >> Nothing wrong with sdist of course, but one thing I hear from the emphasis >> on sdist (something that pip can build automatically) is that maybe we are

Re: [Distutils] PEP 517: Open questions around artifact export directories

> On Jun 15, 2017, at 4:55 PM, Brett Cannon wrote: > > > > On Thu, 15 Jun 2017 at 08:25 Donald Stufft > wrote: > >> On Jun 15, 2017, at 10:10 AM, C Anthony Risinger >

Re: [Distutils] PEP 517: Open questions around artifact export directories

> On Jun 15, 2017, at 4:29 PM, Nathaniel Smith wrote: > > - However, I am more dubious about hacking stuff like prepare_build_files > into the standard to support this specific interaction between two specific > tools, especially since it's not at all obvious to me that this

Re: [Distutils] PEP 517: Open questions around artifact export directories

On Thu, 15 Jun 2017 at 08:25 Donald Stufft wrote: > > On Jun 15, 2017, at 10:10 AM, C Anthony Risinger > wrote: > > On Tue, Jun 13, 2017 at 8:53 PM, Nick Coghlan wrote: > >> On 13 June 2017 at 19:44, Thomas Kluyver

Re: [Distutils] PEP 517: Open questions around artifact export directories

On 15 June 2017 at 21:49, Daniel Holth wrote: > Nothing wrong with sdist of course, but one thing I hear from the emphasis > on sdist (something that pip can build automatically) is that maybe we are > worried that a publisher, who gave us the software for free, will unduly >

Re: [Distutils] PEP 517: Open questions around artifact export directories

Nothing wrong with sdist of course, but one thing I hear from the emphasis on sdist (something that pip can build automatically) is that maybe we are worried that a publisher, who gave us the software for free, will unduly inconvenience the consumer. But that would require an obligation from the

Re: [Distutils] PEP 517: Open questions around artifact export directories

On Wed, Jun 14, 2017 at 1:49 PM, Donald Stufft wrote: > > > On Jun 12, 2017, at 9:27 PM, Nick Coghlan wrote: > > > 5) Drop the prepare_files hook > Rationale: it's purpose is somewhat unclear at this point, and it > gives up the main advantage of going

Re: [Distutils] PEP 517: Open questions around artifact export directories

> On Jun 15, 2017, at 10:10 AM, C Anthony Risinger > wrote: > > On Tue, Jun 13, 2017 at 8:53 PM, Nick Coghlan > wrote: > On 13 June 2017 at 19:44, Thomas Kluyver >

Re: [Distutils] PEP 517: Open questions around artifact export directories

On Tue, Jun 13, 2017 at 8:53 PM, Nick Coghlan wrote: > On 13 June 2017 at 19:44, Thomas Kluyver wrote: > > On Tue, Jun 13, 2017, at 02:27 AM, Nick Coghlan wrote: > > > I've updated the PR to specify zip archives for build_wheel and .tar.gz > > archives

Re: [Distutils] PEP 517: Open questions around artifact export directories

On Thu, Jun 15, 2017, at 03:10 PM, C Anthony Risinger wrote: > I'm not trying to open a bikeshedding opportunity here -- and I tried > to ignore it, honest! -- but why are tarballs preferable to zipfiles > for sdists?> > ... > > Just seems a little odd/arbitrary to me that wheel is zip, python >

Re: [Distutils] PEP 517: Open questions around artifact export directories

Sounds like an unfortunate side effect of using distutils instead of a better build system. If they were using enscons the wheel generation would be entirely driven by a manifest or a series of glob patterns, and it would not matter if there were any extra files in the build directory. As long as

Re: [Distutils] PEP 517: Open questions around artifact export directories

> On Jun 12, 2017, at 9:27 PM, Nick Coghlan wrote: > >> >> 5) Drop the prepare_files hook >> Rationale: it's purpose is somewhat unclear at this point, and it >> gives up the main advantage of going through sdist (the guarantee that >> building from sdist and building from

Re: [Distutils] PEP 517: Open questions around artifact export directories

I have updated the PR as requested: 1. Specify the PAX format, and note that it is not yet the default for the tarfile module in Python. 2. Updated the self-contained backend example with Nick's suggestions. Thomas On Wed, Jun 14, 2017, at 08:53 AM, Paul Moore wrote: > On 14 June 2017 at 02:53,

Re: [Distutils] PEP 517: Open questions around artifact export directories

On 14 June 2017 at 02:53, Nick Coghlan wrote: > While the > standard library does still default to GNU_FORMAT in general, the > stated rationale for doing so (it being more widely supported than > PAX_FORMAT) was last updated more than 10 years ago, and I don't think > it

Re: [Distutils] PEP 517: Open questions around artifact export directories

On 14 June 2017 at 11:53, Nick Coghlan wrote: > def build_sdist(sdist_dir, config_settings): > sdist_subdir = "mypackage-0.1" > sdist_path = pathlib.Path(sdist_dir) / (sdist_subdir + ".tar.gz") > sdist = tarfile.open(sdist_path, "w:gz",

Re: [Distutils] PEP 517: Open questions around artifact export directories

On 13 June 2017 at 19:44, Thomas Kluyver wrote: > On Tue, Jun 13, 2017, at 02:27 AM, Nick Coghlan wrote: >> Despite being the one to originally propose standardisation on passing >> directory paths around, I'm starting to lean back towards this >> approach. >> >> My

Re: [Distutils] PEP 517: Open questions around artifact export directories

Took a peek at the warehouse source code to try to understand its sdist requirements, it appears to be less strict than I thought. Probably missed something but it looks like warehouse currently requires a .zip sdist to contain PKG-INFO one level deep but doesn't care about the contents of .tar.gz

Re: [Distutils] PEP 517: Open questions around artifact export directories

On Tue, Jun 13, 2017, at 02:27 AM, Nick Coghlan wrote: > Despite being the one to originally propose standardisation on passing > directory paths around, I'm starting to lean back towards this > approach. > > My rationale for this doesn't really have a lot to do with topics > we've discussed so

Re: [Distutils] PEP 517: Open questions around artifact export directories

On Tue, Jun 13, 2017 at 12:59 AM, Paul Moore wrote: > > On 12 June 2017 at 23:42, Donald Stufft wrote: > > > > As always, it’s complicated — > > https://marcosc.com/2008/12/zip-files-and-encoding-i-hate-you/ > > Lol. But I still consider falling over on

Re: [Distutils] PEP 517: Open questions around artifact export directories

On 12 June 2017 at 23:42, Donald Stufft wrote: > > As always, it’s complicated — > https://marcosc.com/2008/12/zip-files-and-encoding-i-hate-you/ Lol. But I still consider falling over on normalisation rules an improvement over falling over on unrepresentable characters ;-) And

Re: [Distutils] PEP 517: Open questions around artifact export directories

On 13 June 2017 at 08:55, Nathaniel Smith wrote: > On Mon, Jun 12, 2017 at 3:34 PM, Paul Moore wrote: >> But honestly, I think we're at the point where someone just needs to >> make a decision - there's very little compelling evidence either way. > > I was

Re: [Distutils] PEP 517: Open questions around artifact export directories

On Mon, Jun 12, 2017 at 4:41 PM, Donald Stufft wrote: > > On Jun 12, 2017, at 7:18 PM, Nathaniel Smith wrote: > > On Mon, Jun 12, 2017 at 3:49 PM, Donald Stufft wrote: > >> >> On Jun 12, 2017, at 6:36 PM, Nathaniel Smith

Re: [Distutils] PEP 517: Open questions around artifact export directories

> On Jun 12, 2017, at 7:18 PM, Nathaniel Smith wrote: > > On Mon, Jun 12, 2017 at 3:49 PM, Donald Stufft > wrote: > >> On Jun 12, 2017, at 6:36 PM, Nathaniel Smith > > wrote >> Another point

Re: [Distutils] PEP 517: Open questions around artifact export directories

On Mon, Jun 12, 2017 at 3:49 PM, Donald Stufft wrote: > > On Jun 12, 2017, at 6:36 PM, Nathaniel Smith wrote > > Another point is that tools that you might have in your build pipeline > -- like auditwheel -- currently use wheel files as their interchange >

Re: [Distutils] PEP 517: Open questions around artifact export directories

On Mon, Jun 12, 2017 at 3:34 PM, Paul Moore wrote: > But honestly, I think we're at the point where someone just needs to > make a decision - there's very little compelling evidence either way. I was the original PEP author, but Thomas has mostly taken it over at this point,

Re: [Distutils] PEP 517: Open questions around artifact export directories

> On Jun 12, 2017, at 6:36 PM, Nathaniel Smith wrote: > > On Mon, Jun 12, 2017 at 1:57 PM, Thomas Kluyver wrote: >> On Mon, Jun 12, 2017, at 09:45 PM, Daniel Holth wrote: >> >> I think all my wheel generators except bdist_wheel build the zipfile >>

Re: [Distutils] PEP 517: Open questions around artifact export directories

> On Jun 12, 2017, at 6:34 PM, Paul Moore wrote: > > On 12 June 2017 at 21:57, Thomas Kluyver wrote: >> There is a certain appeal to using the zipped .whl file as the canonical >> format for all tools that produce or consume wheels, rather than

Re: [Distutils] PEP 517: Open questions around artifact export directories

On Mon, Jun 12, 2017 at 1:57 PM, Thomas Kluyver wrote: > On Mon, Jun 12, 2017, at 09:45 PM, Daniel Holth wrote: > > I think all my wheel generators except bdist_wheel build the zipfile > directly. > > > There is a certain appeal to using the zipped .whl file as the canonical

Re: [Distutils] PEP 517: Open questions around artifact export directories

On 12 June 2017 at 21:57, Thomas Kluyver wrote: > There is a certain appeal to using the zipped .whl file as the canonical > format for all tools that produce or consume wheels, rather than defining a > closely related but distinct 'unpacked wheel' format. A directory and a

Re: [Distutils] PEP 517: Open questions around artifact export directories

> On Jun 12, 2017, at 6:10 PM, Thomas Kluyver wrote: > > On Mon, Jun 12, 2017, at 10:23 PM, Donald Stufft wrote: >> because it allows the front ends more flexibility in how they use the wheels > > I don't get this? Why is it more flexible? I went into some detail here:

Re: [Distutils] PEP 517: Open questions around artifact export directories

On Mon, Jun 12, 2017, at 10:23 PM, Donald Stufft wrote: > because it allows the front ends more flexibility in how they use > the wheels I don't get this? Why is it more flexible? > and allows us to avoid doing work, making the process involved faster > for everyone. This is true so long as

Re: [Distutils] PEP 517: Open questions around artifact export directories

> On Jun 12, 2017, at 5:21 PM, Thomas Kluyver wrote: > > On Mon, Jun 12, 2017, at 10:09 PM, Donald Stufft wrote: >> It’s pretty hard to screw up zipping up a directory. > > If you want reproducible builds, it's very easy to screw up, and your > response doesn't inspire

Re: [Distutils] PEP 517: Open questions around artifact export directories

On Mon, Jun 12, 2017, at 10:09 PM, Donald Stufft wrote: > It’s pretty hard to screw up zipping up a directory. If you want reproducible builds, it's very easy to screw up, and your response doesn't inspire confidence that frontends will do it carefully. But I see flit mainly as something you use

Re: [Distutils] PEP 517: Open questions around artifact export directories

> On Jun 12, 2017, at 5:05 PM, Daniel Holth wrote: > > Yes, and I worry that certain front ends will generate the zipfile > incorrectly. Better to do it in the back end. It’s pretty hard to screw up zipping up a directory. — Donald Stufft

Re: [Distutils] PEP 517: Open questions around artifact export directories

Yes, and I worry that certain front ends will generate the zipfile incorrectly. Better to do it in the back end. On Mon, Jun 12, 2017 at 4:57 PM Thomas Kluyver wrote: > On Mon, Jun 12, 2017, at 09:45 PM, Daniel Holth wrote: > > I think all my wheel generators except

Re: [Distutils] PEP 517: Open questions around artifact export directories

On Mon, Jun 12, 2017, at 09:45 PM, Daniel Holth wrote: > I think all my wheel generators except bdist_wheel build the zipfile > directly. There is a certain appeal to using the zipped .whl file as the canonical format for all tools that produce or consume wheels, rather than defining a closely

Re: [Distutils] PEP 517: Open questions around artifact export directories

On Mon, Jun 12, 2017 at 4:41 PM Donald Stufft wrote: > > On Jun 12, 2017, at 4:36 PM, Daniel Holth wrote: > > It's certainly easier to build a zipfile correctly than to build a > directory tree. Might even be faster if your filesystem is slow. Surely if >

Re: [Distutils] PEP 517: Open questions around artifact export directories

It's certainly easier to build a zipfile correctly than to build a directory tree. Might even be faster if your filesystem is slow. Surely if there are multiple *.dist-info it is an error? On Mon, Jun 12, 2017 at 4:15 PM Donald Stufft wrote: > On Jun 12, 2017, at 4:01 PM,

Re: [Distutils] PEP 517: Open questions around artifact export directories

Once I thought it might be useful to define 'wheel internal manifest' (whim) format which would just be a list or mapping that looks like [('category', source path or filelike, destination path), ... ], since that is basically what wheel is doing, putting paths in categories. Then you get the data

Re: [Distutils] PEP 517: Open questions around artifact export directories

> On Jun 12, 2017, at 4:36 PM, Daniel Holth wrote: > > It's certainly easier to build a zipfile correctly than to build a directory > tree. Might even be faster if your filesystem is slow. Surely if there are > multiple *.dist-info it is an error? > Sure, but it’s an

Re: [Distutils] PEP 517: Open questions around artifact export directories

> On Jun 12, 2017, at 4:01 PM, Thomas Kluyver wrote: > > On Sat, Jun 10, 2017, at 06:14 PM, Nick Coghlan wrote: >> Thomas - I agree with Donald's reasoning here, so would you mind >> updating the PEP accordingly? > > I've done so here: >

Re: [Distutils] PEP 517: Open questions around artifact export directories

On Sat, Jun 10, 2017, at 06:14 PM, Nick Coghlan wrote: > Thomas - I agree with Donald's reasoning here, so would you mind > updating the PEP accordingly? I've done so here: https://github.com/python/peps/pull/290 There are still a couple of questions on which I wasn't quite sure what the

Re: [Distutils] PEP 517: Open questions around artifact export directories

> On Jun 10, 2017, at 1:31 PM, Paul Moore wrote: > > On 10 June 2017 at 18:14, Nick Coghlan wrote: >>> Keeping it in the backend doesn’t really buy us much of anything, except >>> that a handful of backend authors don’t have to make relatively minor >>>

Re: [Distutils] PEP 517: Open questions around artifact export directories

On 10 June 2017 at 18:14, Nick Coghlan wrote: >> Keeping it in the backend doesn’t really buy us much of anything, except >> that a handful of backend authors don’t have to make relatively minor >> adjustments to their code base. In a vacuum I can’t see any compelling >>

Re: [Distutils] PEP 517: Open questions around artifact export directories

On 11 June 2017 at 02:59, Donald Stufft wrote: > We have a few possible cases where the build-the-wheel backend is going to > be called: > > 1) We’re creating a wheel ahead of time to publish somewhere. > 2) We’re creating a wheel JIT that we’re going to cache. > 3) We’re

Re: [Distutils] PEP 517: Open questions around artifact export directories

> On Jun 10, 2017, at 12:23 PM, Nick Coghlan wrote: > > The fact this is also true for both "setup.py bdist_wheel" and for > "enscons" would then be the strongest argument in favour of keeping > the current "build_wheel" API: existing backends are already building > wheels

Re: [Distutils] PEP 517: Open questions around artifact export directories

On 10 June 2017 at 19:12, Thomas Kluyver wrote: > On Sat, Jun 10, 2017, at 04:32 AM, Nick Coghlan wrote: >> 1. Move all artifact naming responsibilities to the frontend. All >> backend hooks would just write files in the defined format into the >> directory supplied by the

Re: [Distutils] PEP 517: Open questions around artifact export directories

On Sat, Jun 10, 2017, at 04:32 AM, Nick Coghlan wrote: > 1. Move all artifact naming responsibilities to the frontend. All > backend hooks would just write files in the defined format into the > directory supplied by the frontend (and those directories would have > no defined naming scheme -

Re: [Distutils] PEP 517: Open questions around artifact export directories

On 10 June 2017 at 04:32, Nick Coghlan wrote: > > While I'm open to being persuaded otherwise, my current thinking is > that these concerns could be suitably addressed by making the > following amendments: > > 1. Move all artifact naming responsibilities to the frontend. All >

[Distutils] PEP 517: Open questions around artifact export directories

Hi folks, Thanks for the recent discussions around the PEP 517 build hooks PEP - I think they've clarified and improved the PEP in several areas and are setting us up well for an interoperability API that is both flexible and straightforward for frontends and backends to support. However, I