Re: [Distutils] distlib and wheel metadata

On Thursday, February 23, 2017, Xavier Fernandez wrote: > +1 also. > This whole double requirement feels over-complicated for what seems like a > rather small usecase: it would be interesting to have a few stats on the > number of packages concerned by this pinning

Re: [Distutils] distlib and wheel metadata

On 24 February 2017 at 08:21, Donald Stufft wrote: > > On Feb 23, 2017, at 4:51 PM, C Anthony Risinger wrote: > > The distlib solver would bind up from impossible constraints, yet every > time, pip found a way to "power through" the exact same configuration >

Re: [Distutils] distlib and wheel metadata

On Thursday, February 23, 2017, Donald Stufft wrote: > > On Feb 23, 2017, at 5:31 PM, C Anthony Risinger > wrote: > > Yeah I'm pretty sure the bigger problem was version-less reqs eagerly > selecting a version

Re: [Distutils] distlib and wheel metadata

> On Feb 23, 2017, at 5:31 PM, C Anthony Risinger wrote: > > Yeah I'm pretty sure the bigger problem was version-less reqs eagerly > selecting a version (eg. latest) incompatible with later requirements > provided by a different package, but then treating them as hard reqs by

Re: [Distutils] distlib and wheel metadata

On Thu, Feb 23, 2017 at 4:21 PM, Donald Stufft wrote: > > On Feb 23, 2017, at 4:51 PM, C Anthony Risinger wrote: > > The distlib solver would bind up from impossible constraints, yet every > time, pip found a way to "power through" the exact same configuration

Re: [Distutils] distlib and wheel metadata

> On Feb 23, 2017, at 4:51 PM, C Anthony Risinger wrote: > > The distlib solver would bind up from impossible constraints, yet every time, > pip found a way to "power through" the exact same configuration despite > blatantly incompatible metadata at times. I never looked into

Re: [Distutils] distlib and wheel metadata

On Thu, Feb 23, 2017 at 1:56 PM, Xavier Fernandez wrote: > +1 also. > This whole double requirement feels over-complicated for what seems like a > rather small usecase: it would be interesting to have a few stats on the > number of packages concerned by this pinning

Re: [Distutils] distlib and wheel metadata

On Feb 23, 2017 7:46 AM, "Donald Stufft" wrote: On Feb 23, 2017, at 6:49 AM, Nathaniel Smith wrote: (Here's an example I've just run into that involves a == dependency on a public package: I have a library that needs to access some C API calls on Windows, but

Re: [Distutils] distlib and wheel metadata

+1 also. This whole double requirement feels over-complicated for what seems like a rather small usecase: it would be interesting to have a few stats on the number of packages concerned by this pinning (maybe just scan all the last uploaded wheels of each package ?). And if one needs to classify

Re: [Distutils] distlib and wheel metadata

On 23Feb2017 0914, Donald Stufft wrote: On Feb 23, 2017, at 11:04 AM, Nick Coghlan > wrote: Redistributors may *ask* a publisher to reclassify their project as a library or a devtool (and hence also avoid pinning their dependencies in order to

Re: [Distutils] distlib and wheel metadata

Another way to look at the problem is that it is just too hard to override what the package says. For example in buildout you can provide a patch for any package that does not do exactly what you want, and it is applied during installation. This could include patching the dependencies. On Thu,

Re: [Distutils] distlib and wheel metadata

> On Feb 23, 2017, at 11:04 AM, Nick Coghlan wrote: > > Redistributors may *ask* a publisher to reclassify their project as a library > or a devtool (and hence also avoid pinning their dependencies in order to > make integration easier), but publishers will always have the

Re: [Distutils] distlib and wheel metadata

On 24 February 2017 at 01:27, Paul Moore wrote: > On 23 February 2017 at 15:09, Nick Coghlan wrote: > > The difference is that: > > > > * tool = you typically want at least one copy per Python interpreter > (like a > > library) > > * application = you

Re: [Distutils] distlib and wheel metadata

> On Feb 23, 2017, at 6:49 AM, Nathaniel Smith wrote: > > (Here's an example I've just run into that involves a == dependency on > a public package: I have a library that needs to access some C API > calls on Windows, but not on other platforms. The natural way to do > this is

Re: [Distutils] distlib and wheel metadata

On 23 February 2017 at 15:09, Nick Coghlan wrote: > The difference is that: > > * tool = you typically want at least one copy per Python interpreter (like a > library) > * application = you typically only want one copy per system > > It may be clearer to make the former

Re: [Distutils] distlib and wheel metadata

On 24 February 2017 at 00:28, Paul Moore wrote: > Specifically, I doubt people will make a firm distinction between > "tool" and "library". In many cases it'll be a matter of opinion. Is > py.test a tool or a library? It has a command line interface after > all. I'd also

Re: [Distutils] distlib and wheel metadata

On 23 February 2017 at 14:49, Thomas Kluyver wrote: > On Thu, Feb 23, 2017, at 02:28 PM, Paul Moore wrote: >> I'd also drop "used to develop and deploy Python libraries, >> applications, and scripts" - why does what it's used for affect its >> category? > > Things for

Re: [Distutils] distlib and wheel metadata

On Thu, Feb 23, 2017, at 02:28 PM, Paul Moore wrote: > I'd also drop "used to develop and deploy Python libraries, > applications, and scripts" - why does what it's used for affect its > category? Things for working on & with Python code often have installation requirements a bit different from

Re: [Distutils] distlib and wheel metadata

On 23 February 2017 at 14:24, Petr Viktorin wrote: > On 02/23/2017 02:47 PM, Nick Coghlan wrote: >> >> >> >> library: a software component used to build Python applications. >> Users will mainly interact with the component via a Python API. >> Libraries are

Re: [Distutils] distlib and wheel metadata

On 23 February 2017 at 13:47, Nick Coghlan wrote: > > Slight amendment here to use the term "library" rather than the generic > component (freeing up the latter for its usual meaning in referring to > arbitrary software components). I also realised that we need a separate >

Re: [Distutils] distlib and wheel metadata

On 02/23/2017 02:47 PM, Nick Coghlan wrote: library: a software component used to build Python applications. Users will mainly interact with the component via a Python API. Libraries are essentially dynamic plugins for a Python runtime. Examples: requests, numpy, pytz

Re: [Distutils] distlib and wheel metadata

On 23 February 2017 at 22:32, Nick Coghlan wrote: > On 23 February 2017 at 18:53, Paul Moore wrote: > >> On 23 February 2017 at 08:44, Nick Coghlan wrote: >> > That gets us back into the world of defining what the various package >>

Re: [Distutils] distlib and wheel metadata

On 23 February 2017 at 13:04, Freddy Rietdijk wrote: >> Here's an example I've just run into that involves a == dependency on >> a public package: I have a library that needs to access some C API >> calls on Windows, but not on other platforms. The natural way to do >>

Re: [Distutils] distlib and wheel metadata

On 23 February 2017 at 12:32, Nick Coghlan wrote: > component: a library or framework used to build Python applications. Users > will mainly interact with the component via a Python API. Examples: > requests, numpy, pytz Sorry to nitpick, but why is "component" better than

Re: [Distutils] distlib and wheel metadata

On 23 February 2017 at 23:04, Freddy Rietdijk wrote: > > Here's an example I've just run into that involves a == dependency on > a public package: I have a library that needs to access some C API > calls on Windows, but not on other platforms. The natural way to do >

Re: [Distutils] distlib and wheel metadata

> Here's an example I've just run into that involves a == dependency on a public package: I have a library that needs to access some C API calls on Windows, but not on other platforms. The natural way to do this is to split out the CFFI code into its own package, _mylib_windows_helper or whatever,

Re: [Distutils] distlib and wheel metadata

On 23 February 2017 at 18:53, Paul Moore wrote: > On 23 February 2017 at 08:44, Nick Coghlan wrote: > > That gets us back into the world of defining what the various package > types > > mean, and I really don't want to go there :) > > And yet I still

Re: [Distutils] distlib and wheel metadata

On Thu, Feb 23, 2017 at 12:44 AM, Nick Coghlan wrote: > On 23 February 2017 at 18:37, Paul Moore wrote: >> >> On 23 February 2017 at 08:18, Nick Coghlan wrote: >> > I'm not a huge fan of having simple boolean toggles in metadata >> >

Re: [Distutils] distlib and wheel metadata

On 23 February 2017 at 08:44, Nick Coghlan wrote: > That gets us back into the world of defining what the various package types > mean, and I really don't want to go there :) And yet I still don't understand what's wrong with "application", "library", and "metapackage" (the

Re: [Distutils] distlib and wheel metadata

On 23 February 2017 at 18:37, Paul Moore wrote: > On 23 February 2017 at 08:18, Nick Coghlan wrote: > > I'm not a huge fan of having simple boolean toggles in metadata > definitions > > (hence the more elaborate idea of two different kinds of dependency

Re: [Distutils] distlib and wheel metadata

On 23 February 2017 at 08:18, Nick Coghlan wrote: > I'm not a huge fan of having simple boolean toggles in metadata definitions > (hence the more elaborate idea of two different kinds of dependency > declaration), but this may be a case where that's a good way to go, since it

Re: [Distutils] distlib and wheel metadata

On 17 February 2017 at 23:18, Jeremy Stanley wrote: > On 2017-02-17 09:56:04 +0100 (+0100), Nick Coghlan wrote: > [...] > > So if we rely on a manual "publish with pinned dependencies", "get bug > > report from redistributor or app developer", "republish with unpinned > >

Re: [Distutils] distlib and wheel metadata

On 23 February 2017 at 18:03, Nick Coghlan wrote: > > > On 15 Feb 2017 23:40, "Nathaniel Smith" wrote: > > On Feb 15, 2017 07:41, "Nick Coghlan" wrote: > > > > Ah-hah, this does make sense as a problem, thanks! > > However, your solution

Re: [Distutils] distlib and wheel metadata

On 15 Feb 2017 23:40, "Nathaniel Smith" wrote: On Feb 15, 2017 07:41, "Nick Coghlan" wrote: Ah-hah, this does make sense as a problem, thanks! However, your solution seems very odd to me :-). If the goal is to put an "are you sure/yes I'm sure" UX

Re: [Distutils] distlib and wheel metadata

> On 15 Feb 2017, at 15:11, Nathaniel Smith wrote: > > >> In the vast majority of publication-to-PyPi cases people won't need >> the "integrates" field, since what they're publishing on PyPI will >> just be their abstract dependencies, and any warning against using >> "==" will

Re: [Distutils] distlib and wheel metadata

On 2017-02-17 09:56:04 +0100 (+0100), Nick Coghlan wrote: [...] > So if we rely on a manual "publish with pinned dependencies", "get bug > report from redistributor or app developer", "republish with unpinned > dependencies", we'll be in a situation where: > > - the affected app developer or

Re: [Distutils] distlib and wheel metadata

On 17 February 2017 at 08:56, Nick Coghlan wrote: > - we retain full control over the tone of the error notification I tried to formulate a long response to this email, and got completely bogged down. So I'm going to give a brief[1] response for now and duck out until the

Re: [Distutils] distlib and wheel metadata

On Fri, Feb 17, 2017 at 12:56 AM, Nick Coghlan wrote: > By contrast, if we design the metadata format such that *PyPI* can provide a > suitable error message, then: But all these benefits you're describing also work if you s/PyPI/setuptools/, no? And that doesn't require any

Re: [Distutils] distlib and wheel metadata

On 15 Feb 2017 23:28, "Paul Moore" wrote: So, in summary, * I agree that libraries pinning dependencies too tightly is bad. * Distributions can easily enough report such pins upstream when the library is initially packaged, so there's no ongoing cost here (just possibly a

Re: [Distutils] distlib and wheel metadata

> I notice here (and in a few other places) you talk about "Applications". From what I understand of Nick's position, applications absolutely should pin their dependencies - so if I'm understanding correctly, those applications will (and should) continue to pin exact versions. Application

Re: [Distutils] distlib and wheel metadata

> On Feb 15, 2017, at 11:44 AM, Donald Stufft wrote: > > >> On Feb 15, 2017, at 1:15 PM, Daniel Holth > > wrote: >> >> I also get a little frustrated with this kind of proposal "no pins" which I >> read as "annoy the publisher to

Re: [Distutils] distlib and wheel metadata

> On Feb 15, 2017, at 1:15 PM, Daniel Holth wrote: > > I also get a little frustrated with this kind of proposal "no pins" which I > read as "annoy the publisher to try to prevent them from annoying the > consumer". As a free software publisher I feel entitled to annoy the

Re: [Distutils] distlib and wheel metadata

I also get a little frustrated with this kind of proposal "no pins" which I read as "annoy the publisher to try to prevent them from annoying the consumer". As a free software publisher I feel entitled to annoy the consumer, an activity I will indulge in inversely proportional to my desire for

Re: [Distutils] distlib and wheel metadata

On Feb 15, 2017 07:41, "Nick Coghlan" wrote: >> pipenv borrows the Ruby solution to modeling this by having Pipfile >> for abstract dependency declarations and Pipfile.lock for concrete >> integration testing ones, so the idea here is to propagate that model >> to pydist.json

Re: [Distutils] distlib and wheel metadata

Thanks for your reply, it was very helpful. On 15 February 2017 at 16:55, Freddy Rietdijk wrote: > Larger applications that have many dependencies that are fixed have been > kept out of Nixpkgs for now. I notice here (and in a few other places) you talk about

Re: [Distutils] distlib and wheel metadata

On Wed, Feb 15, 2017 at 11:55 AM, Freddy Rietdijk wrote: > > Sort of repeating my earlier question, but how often does this happen > in reality? > > From a quick check in our repo we have patched about 1% of our packages to > remove the constraints. We have close to 2000

Re: [Distutils] distlib and wheel metadata

> Sort of repeating my earlier question, but how often does this happen in reality? >From a quick check in our repo we have patched about 1% of our packages to remove the constraints. We have close to 2000 Python packages. We don't necessarily patch all the constraints, only when they collide

Re: [Distutils] distlib and wheel metadata

On 15 February 2017 at 15:50, Freddy Rietdijk wrote: > It's quite frustrating as a downstream having to deal with packages where > versions are pinned unnecessarily and therefore I've also requested on the > Setuptools tracker a flag that ignores constraints [1] (though I

Re: [Distutils] distlib and wheel metadata

On Wed, Feb 15, 2017, at 03:40 PM, Daniel Holth wrote: > It would make sense to go ahead and delete the obsolete fields, I'm > sure they were overlooked because they are not common in the wild. > > From PEP 345: > * Deprecated fields: >* Requires (in favor of Requires-Dist) >*

Re: [Distutils] distlib and wheel metadata

> Maybe it would help if you have a concrete example of a scenario where they would benefit from having this distinction? In the Nix package manager (source distribution with binary substitutes) and Nixpkgs package set we typically require the filename and hash of a package. In our expressions we

Re: [Distutils] distlib and wheel metadata

On 15 February 2017 at 15:41, Nick Coghlan wrote: > My goal with the split is to get to a state where: > > - exactly zero projects on PyPI use "==" or "===" in their requires > metadata (because PyPI explicitly prohibits it) > - the vast majority of projects on PyPI *don't*

Re: [Distutils] distlib and wheel metadata

On 15 February 2017 at 15:58, Paul Moore wrote: > On 15 February 2017 at 14:11, Nathaniel Smith wrote: >>> It's mainly a matter of incorporating >>> https://caremad.io/posts/2013/07/setup-vs-requirement/ into the core >>> data model, as this distinction

Re: [Distutils] distlib and wheel metadata

On 15 February 2017 at 15:11, Nathaniel Smith wrote: > On Wed, Feb 15, 2017 at 5:27 AM, Nick Coghlan wrote: >> It's mainly a matter of incorporating >> https://caremad.io/posts/2013/07/setup-vs-requirement/ into the core >> data model, as this distinction

Re: [Distutils] distlib and wheel metadata

IIUC PEP 345, the predecessor of PEP 426, replaced Requires with Requires-Dist because the former was never very well specified, easier to re-name the field rather than redefine it. bdist_wheel's egg-info conversion assumes the only useful requirements are in the setuptools requires.txt. It would

Re: [Distutils] distlib and wheel metadata

> the full METADATA format is documented in the pre-JSON revision of PEP 426. Can you confirm which exact revision in the PEPs repo you mean? I could guess at 0451397. That version does not refer to a field "Requires" (rather, the more recent "Requires-Dist"). Your conversion function reads

Re: [Distutils] distlib and wheel metadata

On 15 February 2017 at 14:11, Nathaniel Smith wrote: >> It's mainly a matter of incorporating >> https://caremad.io/posts/2013/07/setup-vs-requirement/ into the core >> data model, as this distinction between abstract development >> dependencies and concrete deployment

Re: [Distutils] distlib and wheel metadata

On 15 February 2017 at 14:00, Wes Turner wrote: > On Wed, Feb 15, 2017 at 5:33 AM, Nick Coghlan wrote: >> I asked Daniel to *stop* using pydist.json, since wheel was emitting a >> point-in-time snapshot of PEP 426 (which includes a lot of >>

Re: [Distutils] distlib and wheel metadata

Wheel puts everything important in METADATA, except entry_points.txt. The requirements expressed there under 'Requires-Dist' are reliable, and the full METADATA format is documented in the pre-JSON revision of PEP 426. At runtime, once pkg_resources parses it, *.egg-info and *.dist-info look

Re: [Distutils] distlib and wheel metadata

On Wed, Feb 15, 2017 at 5:27 AM, Nick Coghlan wrote: > On 15 February 2017 at 12:58, Nathaniel Smith wrote: >> On Wed, Feb 15, 2017 at 3:33 AM, Nick Coghlan wrote: >>> - "requires": list where entries are either a string containing a PEP

Re: [Distutils] distlib and wheel metadata

On 15 February 2017 at 12:58, Nathaniel Smith wrote: > On Wed, Feb 15, 2017 at 3:33 AM, Nick Coghlan wrote: >> - "requires": list where entries are either a string containing a PEP >> 508 dependency specifier or else a hash map contain a "requires" key >> plus

Re: [Distutils] distlib and wheel metadata

On Wed, Feb 15, 2017 at 5:33 AM, Nick Coghlan wrote: > On 14 February 2017 at 21:21, Vinay Sajip via Distutils-SIG > wrote: > > > > > >> I thought the current status was that it's called metadata.json > >> exactly *because* it's not standardized,

Re: [Distutils] distlib and wheel metadata

On Wed, Feb 15, 2017 at 3:33 AM, Nick Coghlan wrote: > - "requires": list where entries are either a string containing a PEP > 508 dependency specifier or else a hash map contain a "requires" key > plus "extra" or "environment" fields as qualifiers > - "integrates":

Re: [Distutils] distlib and wheel metadata

On 14 February 2017 at 21:21, Vinay Sajip via Distutils-SIG wrote: > > >> I thought the current status was that it's called metadata.json >> exactly *because* it's not standardized, and you *shouldn't* look at >> it? > > > Well, it was work-in-progress-standardised

Re: [Distutils] distlib and wheel metadata

> Managing backwards compatibility is probably the single most important thing > we can do here. > There are almost 800,000 files on PyPI that someone can download and install, > telling all > of them they need to switch to some new system or things are going to break > for them is > simply

Re: [Distutils] distlib and wheel metadata

> The technical problem with PEP 426 is that unless you want to throw away pypi > and start over, > all tools need to understand the old METADATA files regardless. It might not be as bad as that. For example, that IMO was the mistake behind the original concept of distutils2 - it was never

Re: [Distutils] distlib and wheel metadata

On Tue, Feb 14, 2017 at 1:36 PM, Donald Stufft wrote: > WRT to “standard implementations” versus “standards”, the idea of a > “standard implementation” being the source of truth and no longer needing to > do all the work to define standards is a nice idea, but I think it is an >

Re: [Distutils] distlib and wheel metadata

> On Feb 14, 2017, at 1:15 PM, Nathaniel Smith wrote: > > It's too bad that the JSON thing didn't work out, but I think we're > better off working on better specifying the one source of truth > everything already uses (METADATA) instead of bringing in *new* >

Re: [Distutils] distlib and wheel metadata

On Feb 14, 2017 12:21, "Vinay Sajip" wrote: > I thought the current status was that it's called metadata.json > exactly *because* it's not standardized, and you *shouldn't* look at > it? Well, it was work-in-progress-standardised according to PEP 426 (since sometimes

Re: [Distutils] distlib and wheel metadata

On Tue, Feb 14, 2017 at 12:15 PM, Nathaniel Smith wrote: > On Tue, Feb 14, 2017 at 10:10 AM, Vinay Sajip via Distutils-SIG > wrote: > >> humpty in term uses uses distlib which seems to mishandle wheel > >> metadata. (For example, it chokes if there's

Re: [Distutils] distlib and wheel metadata

> I thought the current status was that it's called metadata.json > exactly *because* it's not standardized, and you *shouldn't* look at > it? Well, it was work-in-progress-standardised according to PEP 426 (since sometimes implementations have to work in parallel with working out the details

Re: [Distutils] distlib and wheel metadata

On Tue, Feb 14, 2017 at 2:40 PM, Vinay Sajip wrote: > > Nope. Honestly, though, I wish there was *one* *library* that defined > the standard, > > which was the case for setuptools for a while (yeah, I know, the warts, > really, I know) > > because I really don't think

Re: [Distutils] distlib and wheel metadata

> Nope. Honestly, though, I wish there was *one* *library* that defined the > standard, > which was the case for setuptools for a while (yeah, I know, the warts, > really, I know) > because I really don't think there's a desire to innovate or a reason for > competition > at this level. In the

Re: [Distutils] distlib and wheel metadata

On 14 February 2017 at 18:36, Jim Fulton wrote: > I wish there was *one* *library* that defined the standard packaging should be that library, but it doesn't cover metadata precisely because that PEP 426 hasn't been accepted (it doesn't try to cover the historical metadata

Re: [Distutils] distlib and wheel metadata

I would accept a pull request to stop generating metadata.json in bdist_wheel. On Tue, Feb 14, 2017 at 1:16 PM Nathaniel Smith wrote: > On Tue, Feb 14, 2017 at 10:10 AM, Vinay Sajip via Distutils-SIG > wrote: > >> humpty in term uses uses distlib which

Re: [Distutils] distlib and wheel metadata

On Tue, Feb 14, 2017 at 1:10 PM, Vinay Sajip wrote: > > humpty in term uses uses distlib which seems to mishandle wheel > > metadata. (For example, it chokes if there's extra distribution meta and > > makes it impossible for buildout to install python-dateutil from a >

Re: [Distutils] distlib and wheel metadata

On Tue, Feb 14, 2017 at 10:10 AM, Vinay Sajip via Distutils-SIG wrote: >> humpty in term uses uses distlib which seems to mishandle wheel >> metadata. (For example, it chokes if there's extra distribution meta and >> makes it impossible for buildout to install

[Distutils] distlib and wheel metadata

> humpty in term uses uses distlib which seems to mishandle wheel> metadata. >(For example, it chokes if there's extra distribution meta and > makes it impossible for buildout to install python-dateutil from a wheel.) I looked into the "mishandling". It's that the other tools don't adhere to [the