[Distutils] Re: pipenv and pip
There was a specific bug related to pipenv-only functionality which is why the vcs ref is obtained. Pip now by default prefers ephemeral wheels, while pipenv maintains a copy of editable repos currently. The work Tzu Ping has been discussing and the work in pipenv currently are separate. You can also not simply read some lines of pipenv and assume there should be a 1-1 functional equivalence. Sometimes we will have overlap and sometimes we will have bugs. In the specific case you mentioned, we simply make sure to check out whatever version is requested in the Pipfile before doing resolution and handing off the results to pip for installation. So while it may seem like we are simply doing things over again that pip already handles, we have different motivations and while we very likely have plenty of bugs, there is more context than that we did something that pip also does. In any event, I’m not sure this mailing list is the right place to do code reviews; we would happily accept any feedback if you make it on the project itself. Pipenv has a lot of cleanup work to do. We are in the process of tackling that now. If you have ideas for how to tackle that, we would love to hear them :) With regard to zazo — it’s been mentioned to us many times but we’ve been in touch with Pradyun as I mentioned and he said he was very busy and couldn’t coordinate efforts at all on the resolver front. Zazo isn’t actually an implementation, it’s an abstraction layer. We built a directed graph and layered a resolver on top of it. Since this is a primary functionality pipenv has always wanted to offer (as far as my understanding went) and has always basically failed at because of various tooling issues, we have been working for the last 4-8 weeks pretty much in seclusion trying to tackle this. Dan Ryan // pipenv maintainer gh: @techalchemy > On Aug 21, 2018, at 9:02 AM, Tzu-ping Chung wrote: > > Hi Chris, > > From my understanding (it is totally possible I could miss something), > get_vcs_deps() > is specifically used only for resolution (to populate Git hash etc. in the > lock file, if I’m > not mistaken), not installation. This is why I mentioned the two main aspects > Pipenv > interact with pip for in the very beginning—the situation of the two parts > are very > different; the latter (resolution) is significantly worse than the former. > > I admit Pipenv has not been a good citizen in that regard. I do hope to clean > most > (if not all) of those up with the new resolver implementation. > > Thanks for the feedback, and sorry for the disturbance. > > TP > > > Sent from Mail for Windows 10 > > From: Chris Jerdonek > Sent: 21 August 2018 19:58 > To: Tzu-ping Chung > Cc: distutils sig > Subject: Re: [Distutils] Re: pipenv and pip > > On Tue, Aug 21, 2018 at 4:02 AM, Tzu-ping Chung wrote: > > > > Pipenv mainly interacts with pip for two things: install/uninstall/upgrade > > packages, and to gain information > > about a package (what versions are available, what dependencies does a > > particular version has, etc.). > > For the former case, we are currently using it with subprocesses, and it is > > likely the intended way of > > interaction. > > I just want to say that this didn't appear to be the case when I > looked at the code. For example, the pipenv function obtain_vcs_req() > called inside get_vcs_deps() uses internal pip API's directly -- > apparently for its installs of "VCS dependencies" in editable mode: > https://github.com/pypa/pipenv/blob/5da09fd24e3b63f28f77454594649bd2912fb17d/pipenv/utils.py#L1187-L1196 > > The function obtain_vcs_req() seems to bypass a lot of the logic > inside VersionControl.obtain() (which I know because this is an area > of the code that I've actively been working on improving). I also > noticed that pipenv's code here seems to result in the installation > routine unnecessarily being called twice in succession in some cases, > since it calls update() even after obtain() is called (and a > RevOptions object shouldn't be getting passed to is_commit_id_equal() > there -- that method invocation will always return False). > > It was a little frustrating to see these methods being called in this > way, because it made it appear to me that new, different problems > might be getting introduced in pipenv, even as bugs are getting fixed > in pip. > > --Chris > > > > I have to say, however, that the experience is not flawless. > > pip has a significant startup time, > > and does not offer chances for interaction once it is started on running, so > > we really don’t have a good > > way to, for example, provide installation progress bar for the user, unless > > we parse pip’s stdou
[Distutils] Re: pipenv and pip
To be honest, I don’t feel like I have enough background in the larger ecosystem to speak broadly enough about what goes on the agenda for the core sprints or even what holes there are in library support. Should pipenv’s usage be considered niche? If you want to skip the listing below, a lot of user issues can be summarized as “why don’t things work the way they do in ?” There’s some validity to the questions. Whenever we get away from having to execute code to figure out dependencies, it will help a lot · Why can’t we pin our packages the way npm and yarn do by default? o I mean we *could*, but we (as in python) don’t enforce any kind of versioning, so pipenv doesn’t do this by default because it doesn’t imply any kind of actual guarantee (not that things work better for JS doing this anyway necessarily) · Why do things resolve differently on than they do on ? · Why do things resolve differently on than they do on ? · Why do I get only the hashes for an sdist on my windows machine but when I go to my mac I get a hash checking failure because it downloads a wheel · Marker parsing, marker merging · Basically anything related to having to execute setup.py/build wheels to figure out dependencies · Parsing/normalizing/Querying the index ** this is how I got involved with pipenv in the first place · Shell specific configuration issues (locale/encoding issues, bashrc/fish config/etc) > (For example, I think it would be entirely reasonable for pipenv to say "we're not going to respect an environment variable PIP_FIND_LINKS", but conversely it's a reasonable user request to say "we'd like a standard way to specify local package directories that all tools will respect"). Maybe so, but our users expect pipenv to provide all of the functionality pip does, plus resolution + virtualenv creation. Since we just drop to pip on installation anyway, we might as well honor pip’s configuration for things like this. Looking at it from a user experience perspective, I wouldn’t want to set the same configuration values in 2 different places just because we can’t share an internal API for parsing configurations – that’s why we are just using whatever you guys are doing, because we don’t want to force people to do this twice, and the information is already available. But I guess this is the problem :) We haven’t really taken the time to survey this information, but it’s certainly possible that a standardized configuration would make sense. That said, we’re still finding our feet, and a lot of our issues are just bugs on our end. Our biggest pain point has been some combination of sanitizing and parsing inputs/markers -- pip does a LOT of internal parsing which is not in any other library (besides requirementslib, somewhat). Dan Ryan gh: @techalchemy <https://github.com/techalchemy> // e: d...@danryan.co From: Brett Cannon [mailto:br...@python.org] Sent: Tuesday, August 21, 2018 1:22 PM To: Paul Moore Cc: Distutils Subject: [Distutils] Re: pipenv and pip Since this ties into what's being discussed, I'll mention that on pypa-dev I created an outline of where I saw holes in library support and specs in order to be able to re-constitute pip just from libraries (mostly for the wheel case): https://groups.google.com/forum/#!topic/pypa-dev/91QdZ1vxLT8 . I'll also mention I already have a design done for PEP 425 compatibility tags that I hope to work on at the Python core dev sprints next month so I can try to get it added to 'packaging'. On Tue, 21 Aug 2018 at 05:11 Paul Moore wrote: On Tue, 21 Aug 2018 at 12:04, Tzu-ping Chung wrote: > > Hi, > > Dan and I had been doing most of the maintenance work for Pipenv recently, > and as Dan mentioned, > we have been working on some related projects that poke into pip internals > significantly, so I feel I > should voice some opinions. I have significantly less experience messing with > pip than Dan, and might > be able to offer a slightly different perspective. Thanks, this is really useful. > Pipenv mainly interacts with pip for two things: install/uninstall/upgrade > packages, and to gain information > about a package (what versions are available, what dependencies does a > particular version has, etc.). > For the former case, we are currently using it with subprocesses, and it is > likely the intended way of > interaction. I have to say, however, that the experience is not flawless. pip > has a significant startup time, > and does not offer chances for interaction once it is started on running, so > we really don’t have a good > way to, for example, provide installation progress bar for the user, unless > we parse pip’s stdout directly. > These are not essential to Pipenv’s functionality, however, so they are more >
[Distutils] Re: pipenv and pip
> > A quick skim at the abstractions of ResolveLib, it does abstract in > lot of the same > ways that I intended for zazo. It is a pleasant surprise to me too. > To be fair I have been throwing zazo at everyone as an abstraction layer the same way Paul is (and like Paul I didn't have a chance to do anything with it) -- TP and I kind of had a synchronous brainstorming session about how to abstract our specific need and we iteratively abstracted from a direct implementation like the one in pip-tools (which I basically keep current at this point) through requirementslib (which is our sort of pip <=> pipenv interface library which provides interop to pipfile, requirements.txt, InstallRequirement, etc), and landed on a directed graph for the actual resolver. This probably accommodates my development style more than his or yours, I am super iterative. > While I didn't really say that, I understand you might have been > interpreted what I > said as that. None the less, I'll be happy to coordinate on this front > moving forward. :) Oooh sorry for the miscommunication, I think I only saw the part about limited time. TP and I operate on a 12 hour time shift already, so we will just pick a good platform to continue the more detailed discussions on. I will follow up on the other thread we had (sorry again!) Dan Ryan gh: @techalchemy // e: d...@danryan.co > -Original Message- > From: Pradyun Gedam [mailto:pradyu...@gmail.com] > Sent: Tuesday, August 21, 2018 1:26 PM > To: Tzu-ping Chung > Cc: Dan Ryan; DistUtils Mailing List > Subject: Re: [Distutils] Re: pipenv and pip > > Hey! > > On Tue, Aug 21, 2018 at 7:51 PM Tzu-ping Chung > wrote: > > > > Speaking of Zazo[1], I actually found its abstraction extremely similar to > > our > current > > > > abstraction, ResolveLib[2]. I’m not sure it’s coincidence, we took > > inspiration > from > > > > similar sources, or this is simply the right way to do it, but how the logic > matches > > > > is quite surprising to me. > > A quick skim at the abstractions of ResolveLib, it does abstract in > lot of the same > ways that I intended for zazo. It is a pleasant surprise to me too. > > I have more thoughts on this but I'll avoid going into any more detail > here. I'll > be happy to have a chat with you elsewhere (off list, GH tracker etc) > on this and > collaborate here. > > > I’m quite sure we would be able to find some way to consolidate efforts > once we > > > > find the chance, but for the moment, progress in ResolveLib (and by > extension > > > > Pipenv) would likely benefit pip developing a proper resolver (be it Zazo or > > > > otherwise). > > > > > > TP > > > > > > [1]: https://github.com/pradyunsg/zazo > > [2]: https://github.com/sarugaku/resolvelib > > > > > > > > Sent from Mail for Windows 10 > > > > > > > > From: Dan Ryan > > Sent: 21 August 2018 22:03 > > To: Tzu-ping Chung > > Cc: Chris Jerdonek; distutils sig > > Subject: Re: [Distutils] Re: pipenv and pip > > > > > > > > There was a specific bug related to pipenv-only functionality which is why > the vcs ref is obtained. Pip now by default prefers ephemeral wheels, while > pipenv maintains a copy of editable repos currently. The work Tzu Ping has > been discussing and the work in pipenv currently are separate. You can also > not simply read some lines of pipenv and assume there should be a 1-1 > functional equivalence. Sometimes we will have overlap and sometimes we > will have bugs. In the specific case you mentioned, we simply make sure to > check out whatever version is requested in the Pipfile before doing > resolution and handing off the results to pip for installation. So while it > may > seem like we are simply doing things over again that pip already handles, we > have different motivations and while we very likely have plenty of bugs, > there is more context than that we did something that pip also does. > > > > > > > > In any event, I’m not sure this mailing list is the right place to do code > reviews; we would happily accept any feedback if you make it on the project > itself. Pipenv has a lot of cleanup work to do. We are in the process of > tackling that now. If you have ideas for how to tackle that, we would love to > hear them :) > > > > > > With regard to zazo — it’s been mentioned to us many times but we’ve > been in touch with Pradyun as I mentioned and he said he was very busy and > couldn’t coordinate efforts at all on the resolver front. > > While I didn't really say that, I understand you might hav
[Distutils] Re: pipenv and pip
Hey! On Tue, Aug 21, 2018 at 7:51 PM Tzu-ping Chung wrote: > > Speaking of Zazo[1], I actually found its abstraction extremely similar to > our current > > abstraction, ResolveLib[2]. I’m not sure it’s coincidence, we took > inspiration from > > similar sources, or this is simply the right way to do it, but how the logic > matches > > is quite surprising to me. A quick skim at the abstractions of ResolveLib, it does abstract in lot of the same ways that I intended for zazo. It is a pleasant surprise to me too. I have more thoughts on this but I'll avoid going into any more detail here. I'll be happy to have a chat with you elsewhere (off list, GH tracker etc) on this and collaborate here. > I’m quite sure we would be able to find some way to consolidate efforts once > we > > find the chance, but for the moment, progress in ResolveLib (and by extension > > Pipenv) would likely benefit pip developing a proper resolver (be it Zazo or > > otherwise). > > > TP > > > [1]: https://github.com/pradyunsg/zazo > [2]: https://github.com/sarugaku/resolvelib > > > > Sent from Mail for Windows 10 > > > > From: Dan Ryan > Sent: 21 August 2018 22:03 > To: Tzu-ping Chung > Cc: Chris Jerdonek; distutils sig > Subject: Re: [Distutils] Re: pipenv and pip > > > > There was a specific bug related to pipenv-only functionality which is why > the vcs ref is obtained. Pip now by default prefers ephemeral wheels, while > pipenv maintains a copy of editable repos currently. The work Tzu Ping has > been discussing and the work in pipenv currently are separate. You can also > not simply read some lines of pipenv and assume there should be a 1-1 > functional equivalence. Sometimes we will have overlap and sometimes we will > have bugs. In the specific case you mentioned, we simply make sure to check > out whatever version is requested in the Pipfile before doing resolution and > handing off the results to pip for installation. So while it may seem like > we are simply doing things over again that pip already handles, we have > different motivations and while we very likely have plenty of bugs, there is > more context than that we did something that pip also does. > > > > In any event, I’m not sure this mailing list is the right place to do code > reviews; we would happily accept any feedback if you make it on the project > itself. Pipenv has a lot of cleanup work to do. We are in the process of > tackling that now. If you have ideas for how to tackle that, we would love to > hear them :) > > > With regard to zazo — it’s been mentioned to us many times but we’ve been in > touch with Pradyun as I mentioned and he said he was very busy and couldn’t > coordinate efforts at all on the resolver front. While I didn't really say that, I understand you might have been interpreted what I said as that. None the less, I'll be happy to coordinate on this front moving forward. :) > Zazo isn’t actually an implementation, it’s an abstraction layer. We built a > directed graph and layered a resolver on top of it. zazo is intended to be something along those lines, an abstraction that allows multiple libraries to interface with it, while providing a single well tested core provided the resolution capabilities. This separated the resolution logic from the rest of the codebase pretty well, which was a concern since pip didn't really do that (at all) earlier. Cleaning up the technical debt that has piled up over the years in pip is something I've been doing in the time I'm have had to work on pip. > Since this is a primary functionality pipenv has always wanted to offer (as > far as my understanding went) and has always basically failed at because of > various tooling issues, we have been working for the last 4-8 weeks pretty > much in seclusion trying to tackle this. That's the sort of time I've not really had to work on zazo since after GSoC, for various non-technical reasons. I'm glad someone has made the time to look into this. I'll be happy to help any way I can. Pradyun > > Dan Ryan // pipenv maintainer > > gh: @techalchemy > -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/UH6KMBA4FOVQEHHAWGHWKRORICY722FH/
[Distutils] Re: pipenv and pip
Since this ties into what's being discussed, I'll mention that on pypa-dev I created an outline of where I saw holes in library support and specs in order to be able to re-constitute pip just from libraries (mostly for the wheel case): https://groups.google.com/forum/#!topic/pypa-dev/91QdZ1vxLT8 . I'll also mention I already have a design done for PEP 425 compatibility tags that I hope to work on at the Python core dev sprints next month so I can try to get it added to 'packaging'. On Tue, 21 Aug 2018 at 05:11 Paul Moore wrote: > On Tue, 21 Aug 2018 at 12:04, Tzu-ping Chung wrote: > > > > Hi, > > > > Dan and I had been doing most of the maintenance work for Pipenv > recently, and as Dan mentioned, > > we have been working on some related projects that poke into pip > internals significantly, so I feel I > > should voice some opinions. I have significantly less experience messing > with pip than Dan, and might > > be able to offer a slightly different perspective. > > Thanks, this is really useful. > > > Pipenv mainly interacts with pip for two things: > install/uninstall/upgrade packages, and to gain information > > about a package (what versions are available, what dependencies does a > particular version has, etc.). > > For the former case, we are currently using it with subprocesses, and it > is likely the intended way of > > interaction. I have to say, however, that the experience is not > flawless. pip has a significant startup time, > > and does not offer chances for interaction once it is started on > running, so we really don’t have a good > > way to, for example, provide installation progress bar for the user, > unless we parse pip’s stdout directly. > > These are not essential to Pipenv’s functionality, however, so they are > more like an annoyance rather > > than glaring problems. > > Yes, that's a good point. A programmatic API to do installs would > presumably give much better means of progress reporting, etc. > Unfortunately, it's nowhere near as simple as we'd like, because pip > messes with global state all over the place, so if you just called the > old "pip.main" (before we moved it) you got things like your logging > config, your IO streams and such messed up. It also broke if used in > the presence of threads. Using a subprocess isn't just to protect our > internal APIs, it's also to protect the caller's global state. So > there's more work there than it would seem, and it's likely to affect > the fundamental assumptions of a lot of pip's internal code, but I > agree it would help with a lot of use cases. > > The subprocess overhead is also something I can relate to. I'm heavily > running the pip test suite at the moment, and I'm royally sick of the > runtime from all the process spawning :-( > > But as you say, it's something we can live with for now. > > > The other thing Pipenv uses pip for—getting package information—is more > troubling (to me, personally). > > Pipenv has a slightly different need from pip regarding dependency > resolution. pip can (and does) freely > > drop dependencies that does not match the current environment, but > Pipenv needs to generate a lock file > > for an abstract platform that works for, say, both macOS and Windows. > This means pip’s resolver is not > > useful for us, and we need to implement our own. Our own resolver, > however, still needs to know about > > packages it gets, and we are left with two choices: a. try re-implement > the same logic, or b. use pip internals > > to cobble something together. > > > > We tried to go for a. for a while, but as you’d easily imagine, our own > implementation is buggy, cannot > > handle edge cases nearly as well, and fielded a lot of complaints along > the lines of “I can do this in pip, why > > can’t I do the same in Pipenv”. One example is how package artifacts are > discovered. At my own first > > glance, I thought to myself this wouldn’t be that hard—we have a simple > API, and the naming conventions are > > there, so as long as we specify sources in Pipfile (we do), we should be > able to discover them no problem. > > I couldn’t be more wrong. There are find_links, dependency_links, > pip.conf for the user, for the machine, all > > sorts of things, and for everything quirk in pip we don’t replicate > 100%, issues are filed urging use to fix it. > > In the end we gave up and use pip’s internal PackageFinder instead. > > This is exactly the reason a common library/API and clear spec would > be worth working on. In effect you're having to treat "how pip works" > as a de facto standard that people expect you to follow, and that's > not practical. I've hit this issue as well (luckily, only in adhoc > code) where I want to "get files like pip does" but doing anything > beyond a basic minimum is a nightmare. > > One thought on the package finder - distlib implements a finder, and > while it doesn't include a lot of the things you mention, it does > represent a competing implementation, and there's likely
[Distutils] Re: pipenv and pip
Speaking of Zazo[1], I actually found its abstraction extremely similar to our current abstraction, ResolveLib[2]. I’m not sure it’s coincidence, we took inspiration from similar sources, or this is simply the right way to do it, but how the logic matches is quite surprising to me. I’m quite sure we would be able to find some way to consolidate efforts once we find the chance, but for the moment, progress in ResolveLib (and by extension Pipenv) would likely benefit pip developing a proper resolver (be it Zazo or otherwise). TP [1]: https://github.com/pradyunsg/zazo [2]: https://github.com/sarugaku/resolvelib Sent from Mail for Windows 10 From: Dan Ryan Sent: 21 August 2018 22:03 To: Tzu-ping Chung Cc: Chris Jerdonek; distutils sig Subject: Re: [Distutils] Re: pipenv and pip There was a specific bug related to pipenv-only functionality which is why the vcs ref is obtained. Pip now by default prefers ephemeral wheels, while pipenv maintains a copy of editable repos currently. The work Tzu Ping has been discussing and the work in pipenv currently are separate. You can also not simply read some lines of pipenv and assume there should be a 1-1 functional equivalence. Sometimes we will have overlap and sometimes we will have bugs. In the specific case you mentioned, we simply make sure to check out whatever version is requested in the Pipfile before doing resolution and handing off the results to pip for installation. So while it may seem like we are simply doing things over again that pip already handles, we have different motivations and while we very likely have plenty of bugs, there is more context than that we did something that pip also does. In any event, I’m not sure this mailing list is the right place to do code reviews; we would happily accept any feedback if you make it on the project itself. Pipenv has a lot of cleanup work to do. We are in the process of tackling that now. If you have ideas for how to tackle that, we would love to hear them :) With regard to zazo — it’s been mentioned to us many times but we’ve been in touch with Pradyun as I mentioned and he said he was very busy and couldn’t coordinate efforts at all on the resolver front. Zazo isn’t actually an implementation, it’s an abstraction layer. We built a directed graph and layered a resolver on top of it. Since this is a primary functionality pipenv has always wanted to offer (as far as my understanding went) and has always basically failed at because of various tooling issues, we have been working for the last 4-8 weeks pretty much in seclusion trying to tackle this. Dan Ryan // pipenv maintainer gh: @techalchemy On Aug 21, 2018, at 9:02 AM, Tzu-ping Chung wrote: Hi Chris, >From my understanding (it is totally possible I could miss something), >get_vcs_deps() is specifically used only for resolution (to populate Git hash etc. in the lock file, if I’m not mistaken), not installation. This is why I mentioned the two main aspects Pipenv interact with pip for in the very beginning—the situation of the two parts are very different; the latter (resolution) is significantly worse than the former. I admit Pipenv has not been a good citizen in that regard. I do hope to clean most (if not all) of those up with the new resolver implementation. Thanks for the feedback, and sorry for the disturbance. TP Sent from Mail for Windows 10 From: Chris Jerdonek Sent: 21 August 2018 19:58 To: Tzu-ping Chung Cc: distutils sig Subject: Re: [Distutils] Re: pipenv and pip On Tue, Aug 21, 2018 at 4:02 AM, Tzu-ping Chung wrote: > > Pipenv mainly interacts with pip for two things: install/uninstall/upgrade > packages, and to gain information > about a package (what versions are available, what dependencies does a > particular version has, etc.). > For the former case, we are currently using it with subprocesses, and it is > likely the intended way of > interaction. I just want to say that this didn't appear to be the case when I looked at the code. For example, the pipenv function obtain_vcs_req() called inside get_vcs_deps() uses internal pip API's directly -- apparently for its installs of "VCS dependencies" in editable mode: https://github.com/pypa/pipenv/blob/5da09fd24e3b63f28f77454594649bd2912fb17d/pipenv/utils.py#L1187-L1196 The function obtain_vcs_req() seems to bypass a lot of the logic inside VersionControl.obtain() (which I know because this is an area of the code that I've actively been working on improving). I also noticed that pipenv's code here seems to result in the installation routine unnecessarily being called twice in succession in some cases, since it calls update() even after obtain() is called (and a RevOptions object shouldn't be getting passed to is_commit_id_equal() there -- that method invocation will always return False). It was a little frustrating to see these methods being called in this way, because it made it appear
[Distutils] Re: pipenv and pip
Hi Chris, >From my understanding (it is totally possible I could miss something), >get_vcs_deps() is specifically used only for resolution (to populate Git hash etc. in the lock file, if I’m not mistaken), not installation. This is why I mentioned the two main aspects Pipenv interact with pip for in the very beginning—the situation of the two parts are very different; the latter (resolution) is significantly worse than the former. I admit Pipenv has not been a good citizen in that regard. I do hope to clean most (if not all) of those up with the new resolver implementation. Thanks for the feedback, and sorry for the disturbance. TP Sent from Mail for Windows 10 From: Chris Jerdonek Sent: 21 August 2018 19:58 To: Tzu-ping Chung Cc: distutils sig Subject: Re: [Distutils] Re: pipenv and pip On Tue, Aug 21, 2018 at 4:02 AM, Tzu-ping Chung wrote: > > Pipenv mainly interacts with pip for two things: install/uninstall/upgrade > packages, and to gain information > about a package (what versions are available, what dependencies does a > particular version has, etc.). > For the former case, we are currently using it with subprocesses, and it is > likely the intended way of > interaction. I just want to say that this didn't appear to be the case when I looked at the code. For example, the pipenv function obtain_vcs_req() called inside get_vcs_deps() uses internal pip API's directly -- apparently for its installs of "VCS dependencies" in editable mode: https://github.com/pypa/pipenv/blob/5da09fd24e3b63f28f77454594649bd2912fb17d/pipenv/utils.py#L1187-L1196 The function obtain_vcs_req() seems to bypass a lot of the logic inside VersionControl.obtain() (which I know because this is an area of the code that I've actively been working on improving). I also noticed that pipenv's code here seems to result in the installation routine unnecessarily being called twice in succession in some cases, since it calls update() even after obtain() is called (and a RevOptions object shouldn't be getting passed to is_commit_id_equal() there -- that method invocation will always return False). It was a little frustrating to see these methods being called in this way, because it made it appear to me that new, different problems might be getting introduced in pipenv, even as bugs are getting fixed in pip. --Chris > I have to say, however, that the experience is not flawless. > pip has a significant startup time, > and does not offer chances for interaction once it is started on running, so > we really don’t have a good > way to, for example, provide installation progress bar for the user, unless > we parse pip’s stdout directly. > These are not essential to Pipenv’s functionality, however, so they are more > like an annoyance rather > than glaring problems. > > The other thing Pipenv uses pip for—getting package information—is more > troubling (to me, personally). > Pipenv has a slightly different need from pip regarding dependency > resolution. pip can (and does) freely > drop dependencies that does not match the current environment, but Pipenv > needs to generate a lock file > for an abstract platform that works for, say, both macOS and Windows. This > means pip’s resolver is not > useful for us, and we need to implement our own. Our own resolver, however, > still needs to know about > packages it gets, and we are left with two choices: a. try re-implement the > same logic, or b. use pip internals > to cobble something together. > > We tried to go for a. for a while, but as you’d easily imagine, our own > implementation is buggy, cannot > handle edge cases nearly as well, and fielded a lot of complaints along the > lines of “I can do this in pip, why > can’t I do the same in Pipenv”. One example is how package artifacts are > discovered. At my own first > glance, I thought to myself this wouldn’t be that hard—we have a simple API, > and the naming conventions are > there, so as long as we specify sources in Pipfile (we do), we should be > able to discover them no problem. > I couldn’t be more wrong. There are find_links, dependency_links, pip.conf > for the user, for the machine, all > sorts of things, and for everything quirk in pip we don’t replicate 100%, > issues are filed urging use to fix it. > In the end we gave up and use pip’s internal PackageFinder instead. > > This is a big problem going forward, and we are fully aware of that. The > strategy we are taking at the > moment is to try to limit the surface area of pip internals usage. Dan > mentioned we have been building a > resolver for Pipenv[1], and we took the chance to work toward centralising > things interfacing with pip > internals. Those are still internals, of course, but we now have a > relatively good idea what we actually need > from pip, and
[Distutils] Re: pipenv and pip
On Tue, 21 Aug 2018 at 12:04, Tzu-ping Chung wrote: > > Hi, > > Dan and I had been doing most of the maintenance work for Pipenv recently, > and as Dan mentioned, > we have been working on some related projects that poke into pip internals > significantly, so I feel I > should voice some opinions. I have significantly less experience messing with > pip than Dan, and might > be able to offer a slightly different perspective. Thanks, this is really useful. > Pipenv mainly interacts with pip for two things: install/uninstall/upgrade > packages, and to gain information > about a package (what versions are available, what dependencies does a > particular version has, etc.). > For the former case, we are currently using it with subprocesses, and it is > likely the intended way of > interaction. I have to say, however, that the experience is not flawless. pip > has a significant startup time, > and does not offer chances for interaction once it is started on running, so > we really don’t have a good > way to, for example, provide installation progress bar for the user, unless > we parse pip’s stdout directly. > These are not essential to Pipenv’s functionality, however, so they are more > like an annoyance rather > than glaring problems. Yes, that's a good point. A programmatic API to do installs would presumably give much better means of progress reporting, etc. Unfortunately, it's nowhere near as simple as we'd like, because pip messes with global state all over the place, so if you just called the old "pip.main" (before we moved it) you got things like your logging config, your IO streams and such messed up. It also broke if used in the presence of threads. Using a subprocess isn't just to protect our internal APIs, it's also to protect the caller's global state. So there's more work there than it would seem, and it's likely to affect the fundamental assumptions of a lot of pip's internal code, but I agree it would help with a lot of use cases. The subprocess overhead is also something I can relate to. I'm heavily running the pip test suite at the moment, and I'm royally sick of the runtime from all the process spawning :-( But as you say, it's something we can live with for now. > The other thing Pipenv uses pip for—getting package information—is more > troubling (to me, personally). > Pipenv has a slightly different need from pip regarding dependency > resolution. pip can (and does) freely > drop dependencies that does not match the current environment, but Pipenv > needs to generate a lock file > for an abstract platform that works for, say, both macOS and Windows. This > means pip’s resolver is not > useful for us, and we need to implement our own. Our own resolver, however, > still needs to know about > packages it gets, and we are left with two choices: a. try re-implement the > same logic, or b. use pip internals > to cobble something together. > > We tried to go for a. for a while, but as you’d easily imagine, our own > implementation is buggy, cannot > handle edge cases nearly as well, and fielded a lot of complaints along the > lines of “I can do this in pip, why > can’t I do the same in Pipenv”. One example is how package artifacts are > discovered. At my own first > glance, I thought to myself this wouldn’t be that hard—we have a simple API, > and the naming conventions are > there, so as long as we specify sources in Pipfile (we do), we should be able > to discover them no problem. > I couldn’t be more wrong. There are find_links, dependency_links, pip.conf > for the user, for the machine, all > sorts of things, and for everything quirk in pip we don’t replicate 100%, > issues are filed urging use to fix it. > In the end we gave up and use pip’s internal PackageFinder instead. This is exactly the reason a common library/API and clear spec would be worth working on. In effect you're having to treat "how pip works" as a de facto standard that people expect you to follow, and that's not practical. I've hit this issue as well (luckily, only in adhoc code) where I want to "get files like pip does" but doing anything beyond a basic minimum is a nightmare. One thought on the package finder - distlib implements a finder, and while it doesn't include a lot of the things you mention, it does represent a competing implementation, and there's likely some mileage in trying to have the two implementations converge on a reasonable split between "standard finder behaviour" and "application (pip) specific details". (For example, I think it would be entirely reasonable for pipenv to say "we're not going to respect an environment variable PIP_FIND_LINKS", but conversely it's a reasonable user request to say "we'd like a standard way to specify local package directories that all tools will respect"). I've said it before but it bears repeating - I'd fully support someone pulling chunks of pip's code out and making them into supported 3rd party libraries that we could use as a client.
[Distutils] Re: pipenv and pip
ns to > distutils-sig@python.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ > or, via email, send a message with subject or body 'help' to > distutils-sig-requ...@python.org > > You can reach the person managing the list at > distutils-sig-ow...@python.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Distutils-SIG digest..."Today's Topics: > > 1. Re: pipenv and pip (Dan Ryan) > 2. Re: pipenv and pip (Dan Ryan) > > From: Dan Ryan > Subject: [Distutils] Re: pipenv and pip > Date: 20 August 2018 at 22:04:11 GMT+8 > To: Chris Jerdonek > Cc: distutils sig > > > The truth is that it’s basically impossible to gauge bugs in pip vs bugs in > our patches to it which are often a lot more likely — reproductions of edge > cases can be impossible but there are specific things I know we broke (like > parsing certain kinds of extras, previously) — mostly bugs land in pips > issue tracker before we report them or we will direct people there. We have > like 2 active maintainers and we are maintaining like 15 pipenv related > projects so we normally just point people at pip rather than file an issue. > I am usually on irc as well if needed, and I often ask for clarification > there > > Dan Ryan // pipenv maintainer > gh: @techalchemy > > On Aug 20, 2018, at 4:32 AM, Chris Jerdonek > wrote: > > Thanks. Is the state of affairs as you described them what you're > planning for the future as well, or do you anticipate any changes > worthy of note? > > Also, are any of the bugs filed in pipenv's tracker due to bugs or > rough spots in pip -- is there a way to find those, like by using a > label? It would be good to be able to know about those so pip can > improve and become more useful. It doesn't seem like any bugs have > been filed in pip's tracker in the past year by any of pipenv's top > contributors. That seems a bit surprising to me given pipenv's heavy > reliance on pip (together with the fact that I know pip has its share > of issues), or is there another way you have of communicating > regarding things that interconnect with pip? > > Thanks, > --Chris > > > > On Mon, Aug 20, 2018 at 12:51 AM, Dan Ryan wrote: > Sure I can grab that— we patch pip because we use some internals to handle > resolution and we have some bugs around that currently. They aren’t > upstreamed because they aren’t actually present in pip, only in pipenv. > Pipenv crosses back and forth across the virtualenv boundary during the > process. Pipenv relies on piptools and vendors a patched version of pip to > ensure consistency as well as to provide a few hacks around querying the > index. We do have a bit of reimplementation around some kinds of logic, > with the largest overlap being in parsing of requirements. > > As we handle some resolution, which isn’t really something pip does, there > is no cli interface to achieve this. I maintain a library (as of last week) > which provides compatibility shims between pip versions 8-current. It is a > good idea to use the cli, but we already spend enough resources forking > subprocesses into the background that it is a lot more efficient to use the > internals, which I track quite closely. The preference toward cli > interaction is largely to allow internal api breakage which we don’t mind. > > For the most part, we have open channels of communication as necessary. We > rely as heavily as we can on pip, packaging, and setuptools to connect the > dots, retrieve package info, etc. > > Dan Ryan // pipenv maintainer > gh: @techalchemy > > On Aug 20, 2018, at 2:41 AM, Chris Jerdonek > wrote: > > Hi, > > Can someone explain to me the relationship between pipenv and pip, > from the perspective of pipenv's maintainers? > > For example, does pipenv currently reimplement anything that pip tries > to do, or does it simply call out to pip through the CLI or through > its internal API's? Does it have any preferences or future plans in > this regard? How about upstreaming to pip fixes or things that would > help pipenv? > > I've been contributing to pip more lately, and I had a look at > pipenv's repository for the first time today. > https://github.com/pypa/pipenv > > Given that pip's code was recently made internal, I was a bit > surprised to see that pipenv vendors and patches pip: > https://github.com/pypa/pipenv/tree/master/pipenv/patched/notpip > Before I had always assumed that pipenv used pip's CLI (because that's > what pip says you should do). > > I also noticed that some bugs in pipenv's tracker seem closely related > to pip's behavior, b
[Distutils] Re: pipenv and pip
Hi, Dan and I had been doing most of the maintenance work for Pipenv recently, and as Dan mentioned, we have been working on some related projects that poke into pip internals significantly, so I feel I should voice some opinions. I have significantly less experience messing with pip than Dan, and might be able to offer a slightly different perspective. Pipenv mainly interacts with pip for two things: install/uninstall/upgrade packages, and to gain information about a package (what versions are available, what dependencies does a particular version has, etc.). For the former case, we are currently using it with subprocesses, and it is likely the intended way of interaction. I have to say, however, that the experience is not flawless. pip has a significant startup time, and does not offer chances for interaction once it is started on running, so we really don’t have a good way to, for example, provide installation progress bar for the user, unless we parse pip’s stdout directly. These are not essential to Pipenv’s functionality, however, so they are more like an annoyance rather than glaring problems. The other thing Pipenv uses pip for—getting package information—is more troubling (to me, personally). Pipenv has a slightly different need from pip regarding dependency resolution. pip can (and does) freely drop dependencies that does not match the current environment, but Pipenv needs to generate a lock file for an abstract platform that works for, say, both macOS and Windows. This means pip’s resolver is not useful for us, and we need to implement our own. Our own resolver, however, still needs to know about packages it gets, and we are left with two choices: a. try re-implement the same logic, or b. use pip internals to cobble something together. We tried to go for a. for a while, but as you’d easily imagine, our own implementation is buggy, cannot handle edge cases nearly as well, and fielded a lot of complaints along the lines of “I can do this in pip, why can’t I do the same in Pipenv”. One example is how package artifacts are discovered. At my own first glance, I thought to myself this wouldn’t be that hard—we have a simple API, and the naming conventions are there, so as long as we specify sources in Pipfile (we do), we should be able to discover them no problem. I couldn’t be more wrong. There are find_links, dependency_links, pip.conf for the user, for the machine, all sorts of things, and for everything quirk in pip we don’t replicate 100%, issues are filed urging use to fix it. In the end we gave up and use pip’s internal PackageFinder instead. This is a big problem going forward, and we are fully aware of that. The strategy we are taking at the moment is to try to limit the surface area of pip internals usage. Dan mentioned we have been building a resolver for Pipenv[1], and we took the chance to work toward centralising things interfacing with pip internals. Those are still internals, of course, but we now have a relatively good idea what we actually need from pip, and I’d be extremely happy if some parts of pip can come out as standalone with official blessing. The things I am particularly interested in (since they would be beneficial for Pipenv) are: * VcsSupport * PackageFinder * WheelBuilder (and everything that comes with it like the wheel cache, preparer, unpack_url, etc.) Sorry for the very long post, but I want to get everything out so it might be easier to paint a complete picture of the state we are currently in. [1]: https://github.com/sarugaku/passa <https://github.com/sarugaku/passa> Yours, TP -- Tzu-ping Chung (@uranusjr) uranu...@gmail.com https://uranusjr.com > On 21/8, 2018, at 00:00, distutils-sig-requ...@python.org wrote: > > Send Distutils-SIG mailing list submissions to > distutils-sig@python.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ > or, via email, send a message with subject or body 'help' to > distutils-sig-requ...@python.org > > You can reach the person managing the list at > distutils-sig-ow...@python.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Distutils-SIG digest..."Today's Topics: > > 1. Re: pipenv and pip (Dan Ryan) > 2. Re: pipenv and pip (Dan Ryan) > > From: Dan Ryan > Subject: [Distutils] Re: pipenv and pip > Date: 20 August 2018 at 22:04:11 GMT+8 > To: Chris Jerdonek > Cc: distutils sig > > > The truth is that it’s basically impossible to gauge bugs in pip vs bugs in > our patches to it which are often a lot more likely — reproductions of edge > cases can be impossible but there are specific things I know we broke (like > parsing certain kinds of extras, previously) — mostly bugs land in pips issue > tracker before we
[Distutils] Re: pipenv and pip
On Mon, Aug 20, 2018 at 2:52 AM Wes Turner wrote: > What stable API would be worth maintaining in pip for others to use? > Just to be clear, nothing in my comments was meant to suggest maintaining a stable API. There are other kinds of things pip could do to make it easier for pipenv that don’t involve that. —Chris > "[Distutils] Announcement: Pip 10 is coming, and will move all internal > APIs" > https://groups.google.com/forum/m/#!topic/pypa-dev/JVTfS6ZdAuM > > > > On Monday, August 20, 2018, Chris Jerdonek > wrote: > >> Thanks. Is the state of affairs as you described them what you're >> planning for the future as well, or do you anticipate any changes >> worthy of note? >> >> Also, are any of the bugs filed in pipenv's tracker due to bugs or >> rough spots in pip -- is there a way to find those, like by using a >> label? It would be good to be able to know about those so pip can >> improve and become more useful. It doesn't seem like any bugs have >> been filed in pip's tracker in the past year by any of pipenv's top >> contributors. That seems a bit surprising to me given pipenv's heavy >> reliance on pip (together with the fact that I know pip has its share >> of issues), or is there another way you have of communicating >> regarding things that interconnect with pip? > > > Label ideas? > - 'stable-api' > - > > Who is offering to maintain a stable API in/with/for pip and the Python > community ad infinitum? > > >> Thanks, >> --Chris >> >> >> >> On Mon, Aug 20, 2018 at 12:51 AM, Dan Ryan wrote: >> > Sure I can grab that— we patch pip because we use some internals to >> handle resolution and we have some bugs around that currently. They aren’t >> upstreamed because they aren’t actually present in pip, only in pipenv. >> Pipenv crosses back and forth across the virtualenv boundary during the >> process. Pipenv relies on piptools and vendors a patched version of pip to >> ensure consistency as well as to provide a few hacks around querying the >> index. We do have a bit of reimplementation around some kinds of logic, >> with the largest overlap being in parsing of requirements. >> > >> > As we handle some resolution, which isn’t really something pip does, >> there is no cli interface to achieve this. I maintain a library (as of last >> week) which provides compatibility shims between pip versions 8-current. It >> is a good idea to use the cli, but we already spend enough resources >> forking subprocesses into the background that it is a lot more efficient to >> use the internals, which I track quite closely. The preference toward cli >> interaction is largely to allow internal api breakage which we don’t mind. > > > What is the URL of this library of which you are speaking? > > >> > >> > For the most part, we have open channels of communication as necessary. >> We rely as heavily as we can on pip, packaging, and setuptools to connect >> the dots, retrieve package info, etc. > > > An issue label and something like a PEP would likely survive the ravages > of 10 years of tools tooling around with community packaging commitments. > > >> > >> > Dan Ryan // pipenv maintainer >> > gh: @techalchemy >> > >> >> On Aug 20, 2018, at 2:41 AM, Chris Jerdonek >> wrote: >> >> >> >> Hi, >> >> >> >> Can someone explain to me the relationship between pipenv and pip, >> >> from the perspective of pipenv's maintainers? >> >> >> >> For example, does pipenv currently reimplement anything that pip tries >> >> to do, or does it simply call out to pip through the CLI or through >> >> its internal API's? Does it have any preferences or future plans in >> >> this regard? How about upstreaming to pip fixes or things that would >> >> help pipenv? >> >> >> >> I've been contributing to pip more lately, and I had a look at >> >> pipenv's repository for the first time today. >> >> https://github.com/pypa/pipenv >> >> >> >> Given that pip's code was recently made internal, I was a bit >> >> surprised to see that pipenv vendors and patches pip: >> >> https://github.com/pypa/pipenv/tree/master/pipenv/patched/notpip >> >> Before I had always assumed that pipenv used pip's CLI (because that's >> >> what pip says you should do). >> >> >> >> I also noticed that some bugs in pipenv's tracker seem closely related >> >> to pip's behavior, but I don't recall seeing any bugs or PR's in pip's >> >> tracker reported from pipenv maintainers. >> >> >> >> Without knowing a whole lot more than what I've stated, one concern I >> >> have is around fragmentation, duplication of work, and repeating >> >> mistakes (or introducing new ones) if a lot of work is going into >> >> pipenv without coordinating with pip. Is this in any way similar to >> >> the beginning of what happened with distutils, setuptools, and >> >> distribute that we are still recovering from? >> >> >> >> --Chris >> >> -- >> >> Distutils-SIG mailing list -- distutils-sig@python.org >> >> To unsubscribe send an email to distutils-sig-le...@python.org >> >>
[Distutils] Re: pipenv and pip
The truth is that it’s basically impossible to gauge bugs in pip vs bugs in our patches to it which are often a lot more likely — reproductions of edge cases can be impossible but there are specific things I know we broke (like parsing certain kinds of extras, previously) — mostly bugs land in pips issue tracker before we report them or we will direct people there. We have like 2 active maintainers and we are maintaining like 15 pipenv related projects so we normally just point people at pip rather than file an issue. I am usually on irc as well if needed, and I often ask for clarification there Dan Ryan // pipenv maintainer gh: @techalchemy > On Aug 20, 2018, at 4:32 AM, Chris Jerdonek wrote: > > Thanks. Is the state of affairs as you described them what you're > planning for the future as well, or do you anticipate any changes > worthy of note? > > Also, are any of the bugs filed in pipenv's tracker due to bugs or > rough spots in pip -- is there a way to find those, like by using a > label? It would be good to be able to know about those so pip can > improve and become more useful. It doesn't seem like any bugs have > been filed in pip's tracker in the past year by any of pipenv's top > contributors. That seems a bit surprising to me given pipenv's heavy > reliance on pip (together with the fact that I know pip has its share > of issues), or is there another way you have of communicating > regarding things that interconnect with pip? > > Thanks, > --Chris > > > >> On Mon, Aug 20, 2018 at 12:51 AM, Dan Ryan wrote: >> Sure I can grab that— we patch pip because we use some internals to handle >> resolution and we have some bugs around that currently. They aren’t >> upstreamed because they aren’t actually present in pip, only in pipenv. >> Pipenv crosses back and forth across the virtualenv boundary during the >> process. Pipenv relies on piptools and vendors a patched version of pip to >> ensure consistency as well as to provide a few hacks around querying the >> index. We do have a bit of reimplementation around some kinds of logic, >> with the largest overlap being in parsing of requirements. >> >> As we handle some resolution, which isn’t really something pip does, there >> is no cli interface to achieve this. I maintain a library (as of last week) >> which provides compatibility shims between pip versions 8-current. It is a >> good idea to use the cli, but we already spend enough resources forking >> subprocesses into the background that it is a lot more efficient to use the >> internals, which I track quite closely. The preference toward cli >> interaction is largely to allow internal api breakage which we don’t mind. >> >> For the most part, we have open channels of communication as necessary. We >> rely as heavily as we can on pip, packaging, and setuptools to connect the >> dots, retrieve package info, etc. >> >> Dan Ryan // pipenv maintainer >> gh: @techalchemy >> >>> On Aug 20, 2018, at 2:41 AM, Chris Jerdonek >>> wrote: >>> >>> Hi, >>> >>> Can someone explain to me the relationship between pipenv and pip, >>> from the perspective of pipenv's maintainers? >>> >>> For example, does pipenv currently reimplement anything that pip tries >>> to do, or does it simply call out to pip through the CLI or through >>> its internal API's? Does it have any preferences or future plans in >>> this regard? How about upstreaming to pip fixes or things that would >>> help pipenv? >>> >>> I've been contributing to pip more lately, and I had a look at >>> pipenv's repository for the first time today. >>> https://github.com/pypa/pipenv >>> >>> Given that pip's code was recently made internal, I was a bit >>> surprised to see that pipenv vendors and patches pip: >>> https://github.com/pypa/pipenv/tree/master/pipenv/patched/notpip >>> Before I had always assumed that pipenv used pip's CLI (because that's >>> what pip says you should do). >>> >>> I also noticed that some bugs in pipenv's tracker seem closely related >>> to pip's behavior, but I don't recall seeing any bugs or PR's in pip's >>> tracker reported from pipenv maintainers. >>> >>> Without knowing a whole lot more than what I've stated, one concern I >>> have is around fragmentation, duplication of work, and repeating >>> mistakes (or introducing new ones) if a lot of work is going into >>> pipenv without coordinating with pip. Is this in any way similar to >>> the beginning of what happened with distutils, setuptools, and >>> distribute that we are still recovering from? >>> >>> --Chris >>> -- >>> Distutils-SIG mailing list -- distutils-sig@python.org >>> To unsubscribe send an email to distutils-sig-le...@python.org >>> https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ >>> Message archived at >>> https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/2QECNWSHNEW7UBB24M2K5BISYJY7GMZF/ -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email
[Distutils] Re: pipenv and pip
I actually maintain a separate library for parsing requirements which relies mainly on packaging and which provides a backing implementation for moving between requirements files and Pipfile format. It relies on some pip internals (InstallRequirement specifically) for avoiding rework. As of recently it also includes some resolution functionality. I didn’t include a link to the pip shims library because Paul et. al. aren’t happy we are using this stuff to begin with, which I get. However it is also a good representation of things we have needed in multiple projects associated with pipenv (I reimplemented them in piptools and they were picked up in at least one additional project as well). In pipenvs case specifically we wind up installing packages from user input or from a structured file, in either case we want it to succeed. It would be kind of silly to start from point 0 and pretend there is no code behind pip as we construct dependency graphs. We run into some risk of rework on the resolver front. We just finished a stateful, forward looking, backtracking resolver (we checked in with Pradyun but he is quite busy). Dan Ryan // pipenv maintainer gh: @techalchemy > On Aug 20, 2018, at 9:16 AM, Wayne Werner wrote: > > > >> On Mon, Aug 20, 2018, 7:45 AM Paul Moore wrote: >> On Mon, 20 Aug 2018 at 13:21, Wes Turner wrote: >> > >> > Something as simple as reading a requirements.txt file into JSON must >> > either reimplement or wrongly import from pip._internal. >> >> Or copy pip's code and maintain it locally... >> >> > Anyways, >> > Tool authors reimplementing in particular the requirements.txt >> > functionality shouldn't be necessary. >> >> Agreed. Maybe someone should write a package to handle >> requirements.txt reading API. > > > Or as pipenv does, abort that issue entirely in favor of a toml file because > that's the future, right? :) > > -W > -- > Distutils-SIG mailing list -- distutils-sig@python.org > To unsubscribe send an email to distutils-sig-le...@python.org > https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ > Message archived at > https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/FJFQNINEKKUNFN5TGYROYFS7LPBF6XN5/ -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/5IWQZ4HAFKPMOYWDQYEPEYN72OGHRVBR/
[Distutils] Re: pipenv and pip
On Mon, Aug 20, 2018, 7:45 AM Paul Moore wrote: > On Mon, 20 Aug 2018 at 13:21, Wes Turner wrote: > > > > Something as simple as reading a requirements.txt file into JSON must > either reimplement or wrongly import from pip._internal. > > Or copy pip's code and maintain it locally... > > > Anyways, > > Tool authors reimplementing in particular the requirements.txt > functionality shouldn't be necessary. > > Agreed. Maybe someone should write a package to handle > requirements.txt reading API. > Or as pipenv does, abort that issue entirely in favor of a toml file because that's the future, right? :) -W > -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/FJFQNINEKKUNFN5TGYROYFS7LPBF6XN5/
[Distutils] Re: pipenv and pip
On Mon, 20 Aug 2018 at 13:21, Wes Turner wrote: > > Something as simple as reading a requirements.txt file into JSON must either > reimplement or wrongly import from pip._internal. Or copy pip's code and maintain it locally... > Anyways, > Tool authors reimplementing in particular the requirements.txt functionality > shouldn't be necessary. Agreed. Maybe someone should write a package to handle requirements.txt reading API. They could copy the code from pip with our blessing, and we'd happily vendor them and use their code. As you can see, it's very easy to suggest that someone *else* should do something. The problem is finding someone who has both the interest in solving the problem, and the time to do so. The "why doesn't pip have a stable API" question is very much about the fact that we don't have the *resources* to support one, not about any dislike of programmatic APIs... Paul -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/6NDLQI4NJ2YF2HDTMAKVFFA2OTMLTBRD/
[Distutils] Re: pipenv and pip
On Mon, 20 Aug 2018 at 12:25, Wes Turner wrote: > > On Monday, August 20, 2018, Paul Moore wrote: >> I know "security by obscurity" doesn't work, but I'm happier if >> details of this library *aren't* widely known - it's not something I'd >> want to encourage people using, nor is it supported by pip, as it's >> basically a direct interface into pip's internal functions, papering >> over the name changes that we did in pip 10 specifically to dissuade >> people from doing this. > > > If someone was committing to identifying useful API methods, parameters, and > return values; > writing a ~PEP; > implementing said API; > and maintaining backwards compatible shims for some reason; > would something like `pip.api` be an appropriate namespace? > (now that we're on version 18 with a faster release cycle)? I'm not quite sure I know what you mean here. The key point is that pip 18.0 might have an internal function pip._internal.xxx, and in pip 18.1 there's no such function, and the functionality doesn't even exist any more. How would a 3rd party project maintain backwards compatible shims in the face of that? Agreed it's not likely in practice - but we're not going to guarantee it. To be honest I don't see the point of discussing pip's internal API. It's just that - internal. I'd rather discuss useful (general) packaging libraries, that tools can build on - pip can vendor those and act as (just) another consumer, rather than getting into debates about support and internal APIs. Paul -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/7CCBE53AM72JYQVEW3PH7ODVRFZV4WXA/
[Distutils] Re: pipenv and pip
On Monday, August 20, 2018, Paul Moore wrote: > On Mon, 20 Aug 2018 at 10:54, Wes Turner wrote: > > > > What stable API would be worth maintaining in pip for others to use? > > That's probably the sort of question that can only be usefully > answered by projects like pipenv identifying the functionality they > need and proposing something. Which is of course one of the reasons we > (the pip devs) advise against "just using pip's internals", because it > means we never get that information in any useful form. > > > Who is offering to maintain a stable API in/with/for pip and the Python > community ad infinitum? > > That's the crux of the problem - basically the answer is "no-one". > What we advocate is for generally useful functionality to be split out > into standalone libraries, and then pip, as well as other consumers, > can use those libraries. We already have that with the packaging > library and the script wrappers (part of distlib). The new resolver is > being developed as a standalone library (zazo) as is the PEP 517 hook > wrapper functionality (pep517). There's no reason this model couldn't > work in other areas. (But even then, the question "who's offering to > write these libraries" still applies :-() Thanks! > > >> > As we handle some resolution, which isn’t really something pip does, > there is no cli interface to achieve this. I maintain a library (as of last > week) which provides compatibility shims between pip versions 8-current. It > is a good idea to use the cli, but we already spend enough resources > forking subprocesses into the background that it is a lot more efficient to > use the internals, which I track quite closely. The preference toward cli > interaction is largely to allow internal api breakage which we don’t mind. > > > > What is the URL of this library of which you are speaking? > > I know "security by obscurity" doesn't work, but I'm happier if > details of this library *aren't* widely known - it's not something I'd > want to encourage people using, nor is it supported by pip, as it's > basically a direct interface into pip's internal functions, papering > over the name changes that we did in pip 10 specifically to dissuade > people from doing this. If someone was committing to identifying useful API methods, parameters, and return values; writing a ~PEP; implementing said API; and maintaining backwards compatible shims for some reason; would something like `pip.api` be an appropriate namespace? (now that we're on version 18 with a faster release cycle)? > > Paul > -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/HTD4EF5WIQHMOZAJU4POMA6TI3WSEUMO/
[Distutils] Re: pipenv and pip
On Mon, 20 Aug 2018 at 10:54, Wes Turner wrote: > > What stable API would be worth maintaining in pip for others to use? That's probably the sort of question that can only be usefully answered by projects like pipenv identifying the functionality they need and proposing something. Which is of course one of the reasons we (the pip devs) advise against "just using pip's internals", because it means we never get that information in any useful form. > Who is offering to maintain a stable API in/with/for pip and the Python > community ad infinitum? That's the crux of the problem - basically the answer is "no-one". What we advocate is for generally useful functionality to be split out into standalone libraries, and then pip, as well as other consumers, can use those libraries. We already have that with the packaging library and the script wrappers (part of distlib). The new resolver is being developed as a standalone library (zazo) as is the PEP 517 hook wrapper functionality (pep517). There's no reason this model couldn't work in other areas. (But even then, the question "who's offering to write these libraries" still applies :-() >> > As we handle some resolution, which isn’t really something pip does, there >> > is no cli interface to achieve this. I maintain a library (as of last >> > week) which provides compatibility shims between pip versions 8-current. >> > It is a good idea to use the cli, but we already spend enough resources >> > forking subprocesses into the background that it is a lot more efficient >> > to use the internals, which I track quite closely. The preference toward >> > cli interaction is largely to allow internal api breakage which we don’t >> > mind. > > What is the URL of this library of which you are speaking? I know "security by obscurity" doesn't work, but I'm happier if details of this library *aren't* widely known - it's not something I'd want to encourage people using, nor is it supported by pip, as it's basically a direct interface into pip's internal functions, papering over the name changes that we did in pip 10 specifically to dissuade people from doing this. Paul -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/BHI36Q4WQENSMXT2YMPEGWGMX5BXSUOB/
[Distutils] Re: pipenv and pip
What stable API would be worth maintaining in pip for others to use? "[Distutils] Announcement: Pip 10 is coming, and will move all internal APIs" https://groups.google.com/forum/m/#!topic/pypa-dev/JVTfS6ZdAuM On Monday, August 20, 2018, Chris Jerdonek wrote: > Thanks. Is the state of affairs as you described them what you're > planning for the future as well, or do you anticipate any changes > worthy of note? > > Also, are any of the bugs filed in pipenv's tracker due to bugs or > rough spots in pip -- is there a way to find those, like by using a > label? It would be good to be able to know about those so pip can > improve and become more useful. It doesn't seem like any bugs have > been filed in pip's tracker in the past year by any of pipenv's top > contributors. That seems a bit surprising to me given pipenv's heavy > reliance on pip (together with the fact that I know pip has its share > of issues), or is there another way you have of communicating > regarding things that interconnect with pip? Label ideas? - 'stable-api' - Who is offering to maintain a stable API in/with/for pip and the Python community ad infinitum? > Thanks, > --Chris > > > > On Mon, Aug 20, 2018 at 12:51 AM, Dan Ryan wrote: > > Sure I can grab that— we patch pip because we use some internals to > handle resolution and we have some bugs around that currently. They aren’t > upstreamed because they aren’t actually present in pip, only in pipenv. > Pipenv crosses back and forth across the virtualenv boundary during the > process. Pipenv relies on piptools and vendors a patched version of pip to > ensure consistency as well as to provide a few hacks around querying the > index. We do have a bit of reimplementation around some kinds of logic, > with the largest overlap being in parsing of requirements. > > > > As we handle some resolution, which isn’t really something pip does, > there is no cli interface to achieve this. I maintain a library (as of last > week) which provides compatibility shims between pip versions 8-current. It > is a good idea to use the cli, but we already spend enough resources > forking subprocesses into the background that it is a lot more efficient to > use the internals, which I track quite closely. The preference toward cli > interaction is largely to allow internal api breakage which we don’t mind. What is the URL of this library of which you are speaking? > > > > For the most part, we have open channels of communication as necessary. > We rely as heavily as we can on pip, packaging, and setuptools to connect > the dots, retrieve package info, etc. An issue label and something like a PEP would likely survive the ravages of 10 years of tools tooling around with community packaging commitments. > > > > Dan Ryan // pipenv maintainer > > gh: @techalchemy > > > >> On Aug 20, 2018, at 2:41 AM, Chris Jerdonek > wrote: > >> > >> Hi, > >> > >> Can someone explain to me the relationship between pipenv and pip, > >> from the perspective of pipenv's maintainers? > >> > >> For example, does pipenv currently reimplement anything that pip tries > >> to do, or does it simply call out to pip through the CLI or through > >> its internal API's? Does it have any preferences or future plans in > >> this regard? How about upstreaming to pip fixes or things that would > >> help pipenv? > >> > >> I've been contributing to pip more lately, and I had a look at > >> pipenv's repository for the first time today. > >> https://github.com/pypa/pipenv > >> > >> Given that pip's code was recently made internal, I was a bit > >> surprised to see that pipenv vendors and patches pip: > >> https://github.com/pypa/pipenv/tree/master/pipenv/patched/notpip > >> Before I had always assumed that pipenv used pip's CLI (because that's > >> what pip says you should do). > >> > >> I also noticed that some bugs in pipenv's tracker seem closely related > >> to pip's behavior, but I don't recall seeing any bugs or PR's in pip's > >> tracker reported from pipenv maintainers. > >> > >> Without knowing a whole lot more than what I've stated, one concern I > >> have is around fragmentation, duplication of work, and repeating > >> mistakes (or introducing new ones) if a lot of work is going into > >> pipenv without coordinating with pip. Is this in any way similar to > >> the beginning of what happened with distutils, setuptools, and > >> distribute that we are still recovering from? > >> > >> --Chris > >> -- > >> Distutils-SIG mailing list -- distutils-sig@python.org > >> To unsubscribe send an email to distutils-sig-le...@python.org > >> https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ > >> Message archived at https://mail.python.org/mm3/ > archives/list/distutils-sig@python.org/message/ > 2QECNWSHNEW7UBB24M2K5BISYJY7GMZF/ > -- > Distutils-SIG mailing list -- distutils-sig@python.org > To unsubscribe send an email to distutils-sig-le...@python.org >
[Distutils] Re: pipenv and pip
Thanks. Is the state of affairs as you described them what you're planning for the future as well, or do you anticipate any changes worthy of note? Also, are any of the bugs filed in pipenv's tracker due to bugs or rough spots in pip -- is there a way to find those, like by using a label? It would be good to be able to know about those so pip can improve and become more useful. It doesn't seem like any bugs have been filed in pip's tracker in the past year by any of pipenv's top contributors. That seems a bit surprising to me given pipenv's heavy reliance on pip (together with the fact that I know pip has its share of issues), or is there another way you have of communicating regarding things that interconnect with pip? Thanks, --Chris On Mon, Aug 20, 2018 at 12:51 AM, Dan Ryan wrote: > Sure I can grab that— we patch pip because we use some internals to handle > resolution and we have some bugs around that currently. They aren’t > upstreamed because they aren’t actually present in pip, only in pipenv. > Pipenv crosses back and forth across the virtualenv boundary during the > process. Pipenv relies on piptools and vendors a patched version of pip to > ensure consistency as well as to provide a few hacks around querying the > index. We do have a bit of reimplementation around some kinds of logic, with > the largest overlap being in parsing of requirements. > > As we handle some resolution, which isn’t really something pip does, there is > no cli interface to achieve this. I maintain a library (as of last week) > which provides compatibility shims between pip versions 8-current. It is a > good idea to use the cli, but we already spend enough resources forking > subprocesses into the background that it is a lot more efficient to use the > internals, which I track quite closely. The preference toward cli interaction > is largely to allow internal api breakage which we don’t mind. > > For the most part, we have open channels of communication as necessary. We > rely as heavily as we can on pip, packaging, and setuptools to connect the > dots, retrieve package info, etc. > > Dan Ryan // pipenv maintainer > gh: @techalchemy > >> On Aug 20, 2018, at 2:41 AM, Chris Jerdonek wrote: >> >> Hi, >> >> Can someone explain to me the relationship between pipenv and pip, >> from the perspective of pipenv's maintainers? >> >> For example, does pipenv currently reimplement anything that pip tries >> to do, or does it simply call out to pip through the CLI or through >> its internal API's? Does it have any preferences or future plans in >> this regard? How about upstreaming to pip fixes or things that would >> help pipenv? >> >> I've been contributing to pip more lately, and I had a look at >> pipenv's repository for the first time today. >> https://github.com/pypa/pipenv >> >> Given that pip's code was recently made internal, I was a bit >> surprised to see that pipenv vendors and patches pip: >> https://github.com/pypa/pipenv/tree/master/pipenv/patched/notpip >> Before I had always assumed that pipenv used pip's CLI (because that's >> what pip says you should do). >> >> I also noticed that some bugs in pipenv's tracker seem closely related >> to pip's behavior, but I don't recall seeing any bugs or PR's in pip's >> tracker reported from pipenv maintainers. >> >> Without knowing a whole lot more than what I've stated, one concern I >> have is around fragmentation, duplication of work, and repeating >> mistakes (or introducing new ones) if a lot of work is going into >> pipenv without coordinating with pip. Is this in any way similar to >> the beginning of what happened with distutils, setuptools, and >> distribute that we are still recovering from? >> >> --Chris >> -- >> Distutils-SIG mailing list -- distutils-sig@python.org >> To unsubscribe send an email to distutils-sig-le...@python.org >> https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ >> Message archived at >> https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/2QECNWSHNEW7UBB24M2K5BISYJY7GMZF/ -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/2AYIJ3KTB2QJRF3BGV446DXAJGCFVQ5R/
