[Distutils] What are reasonable compatibility tags? (was: Three clarification questions about PEP 425 and PyPy3
On Fri, 31 Aug 2018 at 01:41 Paul Moore wrote: > On Fri, 31 Aug 2018 at 07:15, Nathaniel Smith wrote: > > > > On Thu, Aug 30, 2018 at 6:52 PM, Brett Cannon wrote: > > > > > > > > > On Thu, 30 Aug 2018 at 11:21 Nathaniel Smith wrote: > > > >> * Make the 3 tag categories totally independent. Compute a separate > set > > >> for each, and then take the full cross product. > > > > > > > > > I think Paul was hinting at this as part of his "wildcard" idea (and I > > > honestly thought of this initially as well as it greatly simplifies > things). > > > So what would cp36-cp36m-plat expand to? > > > > I don't know what it means to "expand" a wheel tag. > > To me, the confusion really lies in the interaction of "what tags do > producers set" and "what tags to consumers accept". Yes, it's the fact that some tags seem to have very strict requirements for compatibility while others are fairy lenient (e.g. cp36m versus abi3). > Nearly all of the > odd corner cases of tags on wheels that we see in the wild don't come > from bdist_wheel without manual project intervention, and there's no > validation on what they do in that case. So when we're trying to say > "what should we accept" we're faced with a dilemma, because we don't > really know what the options are. > > As a starting point, I think it would be useful to focus on what tools > are allowed to *generate*. > > The current situation seems to me to be as follows (mostly from the PEP): > > 1. Python tag - can be absolutely anything (--python-tag doesn't > appear to validate the value at all). In practice, py2, py3, and cpXY > are the most common values AFAIK. I'd consider most other values as > having had "limited testing" :-) > Let's find out how common. :) Using the following query (and adjusting as appropriate): ``` SELECT COUNT(DISTINCT file.project) as projects FROM [the-psf:pypi.downloads20180824] WHERE file.filename LIKE '%-py3-%.whl' ``` You get the following results: - *: 120,079 - cp36: 1,088 - cp3: 2 - py36: 79 - py3: 15,639 So yes, there is a definite skew towards cp36 and py3 for at least Python 3.6. :) I also talked with Barry Warsaw about this and he mentioned how Debian shares pure Python installs across environments and while there was initial worry about e.g. py36 versus py35 wheels needing to be taken into account, the problem never really came up. > 2. ABI tag - I don't know. But I'm pretty sure there are variations in > how this works between Windows and Unix, that should be considered > (IIRC, for C extensions older versions of Python on Windows produced > wheels with a "none" ABI tag, but newer ones produce "cpXYm" which was > a change to match Linux behaviour). > I believe this is correct. The other tricky bit is abi3 as that compatibility is tied to the python tag. I think this is where Nathaniel's idea of moving the Python tag to just be py3 and then being tighter on the ABI tag comes in. > 3. Platform tag - distutils.util.get_platform() (but what about > manylinux, how does that end up in there?) If you mean how does 'wheel' decide to put it in there, I don't know. For checking if a wheel works it comes from a check of system details. > Is there a list of valid > values that get_platform() can produce? > Not that I'm specifically aware of. The tricky bit is really when a platform is compatible with more than a single platform tag like Linux and macOS are. > > IMO, before worrying about what consumers should match, we should tie > down the valid values *producers* are allowed to generate, and from > that consider whether PyPI should enforce a particular set of allowed > tags. Obviously, projects can do what they like in terms of randomly > renaming wheels, and users can do "pip install any-old-junk.whl", but > we need a baseline of what counts as "sensible" tagging schemes if > we're going to enumerate acceptable tags in a consumer. > > Essentially this is a variation on the principle of "be strict in what > you produce, and lenient in what you consume", focused on the > "produce" side of the equation. > > My comment about wildcarding is basically a forlorn hope that if we > can't be sure what we're getting, maybe the best we can do is say > "well, py3-*-* sounds like it might work, and there's nothing better, > so let's give it a go". But if we can tie down a definition of what > constitutes a "reasonable" set of tags, enumerating what we'll accept > seems more likely to succeed. > OK, so let's look at what we're trying to support. If we have pure Python code there's very likely going to be a bottom Python version that's supported and then forward-compatibility is assumed. This is specified today through python-requires, so having a specific Python version in the wheel itself isn't totally critical. So 'py3-none-any' combined with python-requires takes care of this the vast majority of the time. Then you have extension modules. At a CPython level there's something like cp36m today (and its
[Distutils] Re: What are reasonable compatibility tags? (was: Three clarification questions about PEP 425 and PyPy3
Le ven. 31 août 2018 à 9:25 PM, Brett Cannon a écrit : > > OK, so let's look at what we're trying to support. If we have pure Python > code there's very likely going to be a bottom Python version that's > supported and then forward-compatibility is assumed. This is specified > today through python-requires, so having a specific Python version in the > wheel itself isn't totally critical. So 'py3-none-any' combined with > python-requires takes care of this the vast majority of the time. > Hello, while python-requires is indeed nice, it is only usable by pip to select a wheel file with a fully compliant PEP 503 repository. Otherwise, like in a local wheelhouse (wheel files in a directory), it is invisible to pip until pip tries to install the wheel so a more precise python tag can still be somewhat useful... (I agree that pip could access the wheel metadata to check the python-requires without needing to unzip the whole file). Xavier > -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/MHLIO7KC47NHQJ42LDHMHN53PIF7KIMN/
[Distutils] Re: What are reasonable compatibility tags? (was: Three clarification questions about PEP 425 and PyPy3
On Fri, 31 Aug 2018 at 13:33 Xavier Fernandez wrote: > > > > Le ven. 31 août 2018 à 9:25 PM, Brett Cannon a écrit : > >> >> OK, so let's look at what we're trying to support. If we have pure Python >> code there's very likely going to be a bottom Python version that's >> supported and then forward-compatibility is assumed. This is specified >> today through python-requires, so having a specific Python version in the >> wheel itself isn't totally critical. So 'py3-none-any' combined with >> python-requires takes care of this the vast majority of the time. >> > > Hello, > > while python-requires is indeed nice, it is only usable by pip to select a > wheel file with a fully compliant PEP 503 repository. > Otherwise, like in a local wheelhouse (wheel files in a directory), it is > invisible to pip until pip tries to install the wheel so a more precise > python tag can still be somewhat useful... > (I agree that pip could access the wheel metadata to check the > python-requires without needing to unzip the whole file). > You can make your pure Python wheel have a py3-cp36m-* wheel name and you get the version specification you want; there is nothing saying a wheel must have extension modules if it specifies an ABI tag. But more importantly, pip still has to resolve what version of a package to use, so regardless of what's cached in your wheelhouse there will be a check against python-requires somewhere, else you're no worse off than just installing the wheel directly. IOW your wheelhouse isn't going to short-circuit pip's resolver, it's just going to save you from having to download some bits from somewhere else (unless I understand how pip utilizes the wheelhouse on disk). -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/RGDXTBACIZR3MU2ZP4YEU4CIE3MG34NH/
[Distutils] Re: Three clarification questions about PEP 425 and PyPy3
On Thu, 30 Aug 2018 at 23:13 Nathaniel Smith wrote:
> On Thu, Aug 30, 2018 at 6:52 PM, Brett Cannon wrote:
> >
> >
> > On Thu, 30 Aug 2018 at 11:21 Nathaniel Smith wrote:
> >>
> >> If we're going to rethink this,
> >
> >
> > Well, I didn't want to "rethink" so much as "fill in". :)
> >
> >>
> >> then I would really like to move away from assigning special meaning to
> >> specific combinations of tags. The thing where if you use cp36m as you
> >> Python ABI tag then you're forced to use cp36 as your python dialect tag
> >> doesn't make sense. And the thing where if you have a wheel with an
> >> extension module tagged as cp36-abi3, then that works fine on 3.7,
> >
> >
> > I wouldn't expect that to if the stable ABI was expanded in Python 3.7
> and
> > you used the expanded part.
>
> In my example, you have a wheel using the 3.6 ABI, running on 3.7.
> That's case that's supposed to work :-).
>
> >>
> >> but if you *remove* the extension module then it *stops* working on 3.7?
> >> That's just bizarre...
> >
> >
> > I don't quite follow what you mean by "remove the extension module then
> it
> > stops". What stops by removing the extension module(s)?
>
> If I'm reading your proposal right, it says that when running on
> Python 3.7, pip should be willing to install wheels tagged cp36-abi3,
> but not wheels tagged cp36-none. But conceptually, if you take the
> extension modules out of a cp36-abi3 wheel, then you're left with a
> cp36-none wheel. So this is weird. Anywhere we're willing to install a
> cp36-abi3 wheel, we should also be willing to install a cp36-none
> wheel.
>
Sure. I took it out because it complicated the logic in my head and only 7
projects have such a wheel.
>
> >>
> >>
> >> So my suggestions:
> >>
> >> * Make the 3 tag categories totally independent. Compute a separate set
> >> for each, and then take the full cross product.
> >
> >
> > I think Paul was hinting at this as part of his "wildcard" idea (and I
> > honestly thought of this initially as well as it greatly simplifies
> things).
> > So what would cp36-cp36m-plat expand to?
>
> I don't know what it means to "expand" a wheel tag. Are you punning
> the tag as also describing a Python installation ("CPython 3.6, with a
> certain soabi tag and platform"), and then asking to find all the
> wheel tags that could go in that installation?
Potentially because this comes up if you're trying to create a
multi-platform lock file for dependencies or you want to download the
dependencies for your cloud deployment which differs from your dev machine.
> You can't actually do
> this in general – for example, determining whether a target
> interpreter is compatible with manylinux wheels requires running some
> special sniffing code on that interpreter.
>
Maybe not entirely in general, but you can come pretty close.
>
> > cp36: cp3N where N is any positive digit(s)? And then toss in py3 and
> py3N?
> > Prefer exact, then generic '3', then older, and finally newer?
>
> Given that we don't have any real use cases for cpXY and pyXY, I'd
> rather not expand the options – just preserve what we do now... so for
> CPython 3.6, I'd say py3, py3N for N <= 6, cp3N for N <= 6, and I
> don't really care about the exact order – some sort of
> more-specific-before-less-specific makes sense, and whatever we do now
> is probably fine. If someone goes wild and starts distributing py35
> and py36 wheels for the same package (via a "36to35" tool, I guess?)
> then that's probably what you want? But I don't imagine this will ever
> be an important use case. We tried it with 2to3, and everyone decided
> they'd rather write in the subset language for a decade instead.
>
> > cp36m: that, abi3, and then 'none'? Do we care if someone has some crazy
> ABI
> > that no one understands like 'b' (and if so should it only be applied to
> > 'py' interpreter versions which break your nice cross product
> simplicity)?
> > plat: depends on platform.
>
> Do you mean, what happens if we find ourselves running on an
> interpreter we don't recognize (not cpython/pypy/jython/...), and that
> interpreter returns something wacky from
> sysconfig.get_config_var("SOABI")?
That and people who have weird ABIs for CPython like 'b' as I found on PyPI.
It's hard to get a clear signal of what people want because people seem to
be signalling a desire to be fairly permissive, but then people bring up
examples where they don't. ;)
> I think there's a reasonable
> argument that in that case we should only accept 'none' as the tag.
> (And then maybe whoever invented this new interpreter gets the job of
> adding sysconfig.get_supported_abi_tags() as a standard stdlib feature
> :-).)
>
> > And the match preference goes to platform, interpreter version, and then
> > ABI? I think that would work in terms of ignoring C ABIs that have very
> > little chance of linking while being the broadest in terms of accepting a
> > wheel that has some semblance of a chance of running.
>
> I don't think
[Distutils] Environment markers for GPU/CUDA availibility
Hi all, trying to pull together a few separate discussions into a single thread here. The main issue is that currently PEP 508 does not provide environment markers for GPU/CUDA availability, which leads to problems for projects that want to provide distributions for environments with and without GPU support. As far as I can tell, there's been multiple suggestions to bring this issue to distutils-sig, but no one has actually done it. Relevant issues: (closed) "How should Python packages depending on TensorFlow structure their requirements?" https://github.com/tensorflow/tensorflow/issues/7166 (closed) "Adding gpu or cuda specification in PEP 508" https://github.com/python/peps/issues/581 (closed) "More support for conditional installation" https://github.com/pypa/pipenv/issues/1353 (no response) "Adding gpu or cuda markers in PEP 508" https://github.com/pypa/interoperability-peps/issues/68 There is now a third-party project which attempts to amend this for tensorflow (https://github.com/akatrevorjay/tensorflow-auto-detect) but this approach is somewhat fragile (depends on version numbers being in sync), doesn't directly scale to all similar projects, and would require maintainers for a given project to maintain _three_ separate projects, instead of just one. I'm not intimately familiar with PEP 508, so my questions for this list: * Is the demand sufficient to justify supporting this use case? * Is it possible to add support for GPU Environment markers? * If so, what would need to be done? * If implemented, what should the transition look like for projects like tensorflow? Thanks! D. -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/LXLF4YSC4WUZOYRX65DW7CESIX7UUBK5/
[Distutils] Re: Environment markers for GPU/CUDA availibility
I’m not knowledgable about GPUs, but from limited conversations with others, it is important to first decide what exactly the problem area is. Unlike currently available environment markers, there’s currently not a very reliable way to programmatically determine even if there is a GPU, let alone what that GPU can actually do (not every GPU can be used by Tensorflow, for example). IMO it would likely be a good route to first implement some interface for GPU environment detection in Python. This interface can then be used in projects like tensorflow-auto-detect. Projects like Tensorflow can also detect directly what implementation it should use, like many projects do platform-specific things by detection os.name of sys.platform. Once we’re sure we have all the things needed for detection, markers can be drafted based on the detection interface. TP > On 01/9/2018, at 03:57, Dustin Ingram wrote: > > Hi all, trying to pull together a few separate discussions into a > single thread here. > > The main issue is that currently PEP 508 does not provide environment > markers for GPU/CUDA availability, which leads to problems for > projects that want to provide distributions for environments with and > without GPU support. > > As far as I can tell, there's been multiple suggestions to bring this > issue to distutils-sig, but no one has actually done it. > > Relevant issues: > > (closed) "How should Python packages depending on TensorFlow structure > their requirements?" > https://github.com/tensorflow/tensorflow/issues/7166 > > (closed) "Adding gpu or cuda specification in PEP 508" > https://github.com/python/peps/issues/581 > > (closed) "More support for conditional installation" > https://github.com/pypa/pipenv/issues/1353 > > (no response) "Adding gpu or cuda markers in PEP 508" > https://github.com/pypa/interoperability-peps/issues/68 > > There is now a third-party project which attempts to amend this for > tensorflow (https://github.com/akatrevorjay/tensorflow-auto-detect) > but this approach is somewhat fragile (depends on version numbers > being in sync), doesn't directly scale to all similar projects, and > would require maintainers for a given project to maintain _three_ > separate projects, instead of just one. > > I'm not intimately familiar with PEP 508, so my questions for this list: > > * Is the demand sufficient to justify supporting this use case? > * Is it possible to add support for GPU Environment markers? > * If so, what would need to be done? > * If implemented, what should the transition look like for projects > like tensorflow? > > Thanks! > D. > -- > Distutils-SIG mailing list -- distutils-sig@python.org > To unsubscribe send an email to distutils-sig-le...@python.org > https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ > Message archived at > https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/LXLF4YSC4WUZOYRX65DW7CESIX7UUBK5/ -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/3VLRDFVQS7E7BBAYF6WQQ5TM2QJMVEDX/
[Distutils] Re: Three clarification questions about PEP 425 and PyPy3
> On Aug 30, 2018, at 9:54 PM, Brett Cannon wrote: > > I initially thought 'packaging', but I also don't want to care about Python > 2. :) So I haven't decided yet. At worst this just leads to a clearer > understanding of how tools should do wheel compatibility matching. If you want to get rid of all of the bundled pep425tags, you’re going to have to care about Python 2 anyways, since I think all of those projects still support it. But if the only output is a better understanding of what the semantics should be, I still think that is super valuable.-- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/I4NTBI5MFWOL5S2ERMAODCCPCJQERPHP/
[Distutils] Re: Three clarification questions about PEP 425 and PyPy3
On Thu, Aug 30, 2018 at 6:52 PM, Brett Cannon wrote:
>
>
> On Thu, 30 Aug 2018 at 11:21 Nathaniel Smith wrote:
>>
>> If we're going to rethink this,
>
>
> Well, I didn't want to "rethink" so much as "fill in". :)
>
>>
>> then I would really like to move away from assigning special meaning to
>> specific combinations of tags. The thing where if you use cp36m as you
>> Python ABI tag then you're forced to use cp36 as your python dialect tag
>> doesn't make sense. And the thing where if you have a wheel with an
>> extension module tagged as cp36-abi3, then that works fine on 3.7,
>
>
> I wouldn't expect that to if the stable ABI was expanded in Python 3.7 and
> you used the expanded part.
In my example, you have a wheel using the 3.6 ABI, running on 3.7.
That's case that's supposed to work :-).
>>
>> but if you *remove* the extension module then it *stops* working on 3.7?
>> That's just bizarre...
>
>
> I don't quite follow what you mean by "remove the extension module then it
> stops". What stops by removing the extension module(s)?
If I'm reading your proposal right, it says that when running on
Python 3.7, pip should be willing to install wheels tagged cp36-abi3,
but not wheels tagged cp36-none. But conceptually, if you take the
extension modules out of a cp36-abi3 wheel, then you're left with a
cp36-none wheel. So this is weird. Anywhere we're willing to install a
cp36-abi3 wheel, we should also be willing to install a cp36-none
wheel.
>>
>>
>> So my suggestions:
>>
>> * Make the 3 tag categories totally independent. Compute a separate set
>> for each, and then take the full cross product.
>
>
> I think Paul was hinting at this as part of his "wildcard" idea (and I
> honestly thought of this initially as well as it greatly simplifies things).
> So what would cp36-cp36m-plat expand to?
I don't know what it means to "expand" a wheel tag. Are you punning
the tag as also describing a Python installation ("CPython 3.6, with a
certain soabi tag and platform"), and then asking to find all the
wheel tags that could go in that installation? You can't actually do
this in general – for example, determining whether a target
interpreter is compatible with manylinux wheels requires running some
special sniffing code on that interpreter.
> cp36: cp3N where N is any positive digit(s)? And then toss in py3 and py3N?
> Prefer exact, then generic '3', then older, and finally newer?
Given that we don't have any real use cases for cpXY and pyXY, I'd
rather not expand the options – just preserve what we do now... so for
CPython 3.6, I'd say py3, py3N for N <= 6, cp3N for N <= 6, and I
don't really care about the exact order – some sort of
more-specific-before-less-specific makes sense, and whatever we do now
is probably fine. If someone goes wild and starts distributing py35
and py36 wheels for the same package (via a "36to35" tool, I guess?)
then that's probably what you want? But I don't imagine this will ever
be an important use case. We tried it with 2to3, and everyone decided
they'd rather write in the subset language for a decade instead.
> cp36m: that, abi3, and then 'none'? Do we care if someone has some crazy ABI
> that no one understands like 'b' (and if so should it only be applied to
> 'py' interpreter versions which break your nice cross product simplicity)?
> plat: depends on platform.
Do you mean, what happens if we find ourselves running on an
interpreter we don't recognize (not cpython/pypy/jython/...), and that
interpreter returns something wacky from
sysconfig.get_config_var("SOABI")? I think there's a reasonable
argument that in that case we should only accept 'none' as the tag.
(And then maybe whoever invented this new interpreter gets the job of
adding sysconfig.get_supported_abi_tags() as a standard stdlib feature
:-).)
> And the match preference goes to platform, interpreter version, and then
> ABI? I think that would work in terms of ignoring C ABIs that have very
> little chance of linking while being the broadest in terms of accepting a
> wheel that has some semblance of a chance of running.
I don't think the preference order matters that much as long as its
well-defined. The only cases where it would affect things are like...
Suppose in the future we add support for platform tags based on
different ISA levels, like x86_64_avx2 vs x86_64_sse3 vs x86_64. Then
you could find yourself in a situation where examplepkg v1.2.3 has the
following wheels available:
cp36-cp36m-manylinux1_x86_64
cp36-abi3-manylinux1_x86_64_avx2
Our environment is CPython 3.6, running on a manylinux1-compatible OS
and we do have AVX2 support available, so either of these wheels could
work. The first wheel has a "better" ABI (cp36m > abi3), but a "worse"
platform (x86_64 < x86_64_avx2). So which one should we pick?
I have zero intuition here. Whoever compiled these wheels is clearly
perverse, and they should stop doing silly things like this.
> And how would that apply to PyPy3? Same ranging on 'pp36N` and drop
[Distutils] Re: Three clarification questions about PEP 425 and PyPy3
On Fri, 31 Aug 2018 at 07:15, Nathaniel Smith wrote: > > On Thu, Aug 30, 2018 at 6:52 PM, Brett Cannon wrote: > > > > > > On Thu, 30 Aug 2018 at 11:21 Nathaniel Smith wrote: > >> * Make the 3 tag categories totally independent. Compute a separate set > >> for each, and then take the full cross product. > > > > > > I think Paul was hinting at this as part of his "wildcard" idea (and I > > honestly thought of this initially as well as it greatly simplifies things). > > So what would cp36-cp36m-plat expand to? > > I don't know what it means to "expand" a wheel tag. To me, the confusion really lies in the interaction of "what tags do producers set" and "what tags to consumers accept". Nearly all of the odd corner cases of tags on wheels that we see in the wild don't come from bdist_wheel without manual project intervention, and there's no validation on what they do in that case. So when we're trying to say "what should we accept" we're faced with a dilemma, because we don't really know what the options are. As a starting point, I think it would be useful to focus on what tools are allowed to *generate*. The current situation seems to me to be as follows (mostly from the PEP): 1. Python tag - can be absolutely anything (--python-tag doesn't appear to validate the value at all). In practice, py2, py3, and cpXY are the most common values AFAIK. I'd consider most other values as having had "limited testing" :-) 2. ABI tag - I don't know. But I'm pretty sure there are variations in how this works between Windows and Unix, that should be considered (IIRC, for C extensions older versions of Python on Windows produced wheels with a "none" ABI tag, but newer ones produce "cpXYm" which was a change to match Linux behaviour). 3. Platform tag - distutils.util.get_platform() (but what about manylinux, how does that end up in there?) Is there a list of valid values that get_platform() can produce? IMO, before worrying about what consumers should match, we should tie down the valid values *producers* are allowed to generate, and from that consider whether PyPI should enforce a particular set of allowed tags. Obviously, projects can do what they like in terms of randomly renaming wheels, and users can do "pip install any-old-junk.whl", but we need a baseline of what counts as "sensible" tagging schemes if we're going to enumerate acceptable tags in a consumer. Essentially this is a variation on the principle of "be strict in what you produce, and lenient in what you consume", focused on the "produce" side of the equation. My comment about wildcarding is basically a forlorn hope that if we can't be sure what we're getting, maybe the best we can do is say "well, py3-*-* sounds like it might work, and there's nothing better, so let's give it a go". But if we can tie down a definition of what constitutes a "reasonable" set of tags, enumerating what we'll accept seems more likely to succeed. Paul -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/IZ2HMXYXRSIFMJNNKZGXJATDPNHHSY5K/
[Distutils] Editable requirement parsing in pip
Hi, I was verifying my Pipenv resolver work, and found a quirk in pip when handling -e requirements. With a requirements.txt: # Works as expected colorama ; os_name == "nt" # Works as expected ./colorama ; os_name == "nt" # Drops markers; always installs disregarding the OS (!) -e ./colorama ; os_name == "nt" This by itself is okay, since it doesn’t really make sense to me to specify a requirement as editable without intending it to always work, although somehow surprising and (AFAICT) undocumented. The behavior in the command line is a little more preplexing: # Works as expected $ pip install "./colorama ; os_name == 'nt'" # Does not work $ pip install -e "./colorama ; os_name == 'nt'" ./colorama ; os_name == "nt" should either be a path to a local project or a VCS url beginning with svn+, git+, hg+, or bzr+ # This either $ pip install "-e ./colorama ; os_name == 'nt'" ./colorama ; os_name == "nt" should either be a path to a local project or a VCS url beginning with svn+, git+, hg+, or bzr+ Since pip’s documentation says a requirements.txt “is just a list of pip install arguments placed in a file” [1], this is surprising to me. [1]: https://pip.pypa.io/en/stable/user_guide/#id1 So my questions are * Is this behaviour intentional, or is it an oversight? * Is there specification or documentation on how pip is supposed to work? Given that PEP 508 (although pip does not actually use its URL lookup syntax) allows requirements to specify markers in all cases, I would expect pip to allow and honour them in all cases. Or, if that’s not the case (since editable is not really a standardised thing), I’d expect pip to at least be consistent with itself :) -- Tzu-ping Chung (@uranusjr) uranu...@gmail.com https://uranusjr.com -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/3B4W7ZCANHBQLYHLHYF5Y6FG757HVYX6/
