Since this ties into what's being discussed, I'll mention that on pypa-dev I created an outline of where I saw holes in library support and specs in order to be able to re-constitute pip just from libraries (mostly for the wheel case): https://groups.google.com/forum/#!topic/pypa-dev/91QdZ1vxLT8 .
I'll also mention I already have a design done for PEP 425 compatibility tags that I hope to work on at the Python core dev sprints next month so I can try to get it added to 'packaging'. On Tue, 21 Aug 2018 at 05:11 Paul Moore <p.f.mo...@gmail.com> wrote: > On Tue, 21 Aug 2018 at 12:04, Tzu-ping Chung <uranu...@gmail.com> wrote: > > > > Hi, > > > > Dan and I had been doing most of the maintenance work for Pipenv > recently, and as Dan mentioned, > > we have been working on some related projects that poke into pip > internals significantly, so I feel I > > should voice some opinions. I have significantly less experience messing > with pip than Dan, and might > > be able to offer a slightly different perspective. > > Thanks, this is really useful. > > > Pipenv mainly interacts with pip for two things: > install/uninstall/upgrade packages, and to gain information > > about a package (what versions are available, what dependencies does a > particular version has, etc.). > > For the former case, we are currently using it with subprocesses, and it > is likely the intended way of > > interaction. I have to say, however, that the experience is not > flawless. pip has a significant startup time, > > and does not offer chances for interaction once it is started on > running, so we really don’t have a good > > way to, for example, provide installation progress bar for the user, > unless we parse pip’s stdout directly. > > These are not essential to Pipenv’s functionality, however, so they are > more like an annoyance rather > > than glaring problems. > > Yes, that's a good point. A programmatic API to do installs would > presumably give much better means of progress reporting, etc. > Unfortunately, it's nowhere near as simple as we'd like, because pip > messes with global state all over the place, so if you just called the > old "pip.main" (before we moved it) you got things like your logging > config, your IO streams and such messed up. It also broke if used in > the presence of threads. Using a subprocess isn't just to protect our > internal APIs, it's also to protect the caller's global state. So > there's more work there than it would seem, and it's likely to affect > the fundamental assumptions of a lot of pip's internal code, but I > agree it would help with a lot of use cases. > > The subprocess overhead is also something I can relate to. I'm heavily > running the pip test suite at the moment, and I'm royally sick of the > runtime from all the process spawning :-( > > But as you say, it's something we can live with for now. > > > The other thing Pipenv uses pip for—getting package information—is more > troubling (to me, personally). > > Pipenv has a slightly different need from pip regarding dependency > resolution. pip can (and does) freely > > drop dependencies that does not match the current environment, but > Pipenv needs to generate a lock file > > for an abstract platform that works for, say, both macOS and Windows. > This means pip’s resolver is not > > useful for us, and we need to implement our own. Our own resolver, > however, still needs to know about > > packages it gets, and we are left with two choices: a. try re-implement > the same logic, or b. use pip internals > > to cobble something together. > > > > We tried to go for a. for a while, but as you’d easily imagine, our own > implementation is buggy, cannot > > handle edge cases nearly as well, and fielded a lot of complaints along > the lines of “I can do this in pip, why > > can’t I do the same in Pipenv”. One example is how package artifacts are > discovered. At my own first > > glance, I thought to myself this wouldn’t be that hard—we have a simple > API, and the naming conventions are > > there, so as long as we specify sources in Pipfile (we do), we should be > able to discover them no problem. > > I couldn’t be more wrong. There are find_links, dependency_links, > pip.conf for the user, for the machine, all > > sorts of things, and for everything quirk in pip we don’t replicate > 100%, issues are filed urging use to fix it. > > In the end we gave up and use pip’s internal PackageFinder instead. > > This is exactly the reason a common library/API and clear spec would > be worth working on. In effect you're having to treat "how pip works" > as a de facto standard that people expect you to follow, and that's > not practical. I've hit this issue as well (luckily, only in adhoc > code) where I want to "get files like pip does" but doing anything > beyond a basic minimum is a nightmare. > > One thought on the package finder - distlib implements a finder, and > while it doesn't include a lot of the things you mention, it does > represent a competing implementation, and there's likely some mileage > in trying to have the two implementations converge on a reasonable > split between "standard finder behaviour" and "application (pip) > specific details". (For example, I think it would be entirely > reasonable for pipenv to say "we're not going to respect an > environment variable PIP_FIND_LINKS", but conversely it's a reasonable > user request to say "we'd like a standard way to specify local package > directories that all tools will respect"). > > I've said it before but it bears repeating - I'd fully support someone > pulling chunks of pip's code out and making them into supported 3rd > party libraries that we could use as a client. I doubt any of the > other pip developers would object either. But doing it properly is far > from a simple undertaking, and I've yet to see any real sign of anyone > offering to actually do that work, rather than just talking about it. > With the exception of what you guys did on pipenv, and ultimately that > ended up with you giving up and calling pip's internals... > > > This is a big problem going forward, and we are fully aware of that. The > strategy we are taking at the > > moment is to try to limit the surface area of pip internals usage. Dan > mentioned we have been building a > > resolver for Pipenv[1], > > This really ought to be co-ordinated with Pradyun's work on the pip > resolver over at https://github.com/pradyunsg/zazo... > > > and we took the chance to work toward centralising things interfacing > with pip > > internals. Those are still internals, of course, but we now have a > relatively good idea what we actually need > > from pip, and I’d be extremely happy if some parts of pip can come out > as standalone with official blessing. > > The things I am particularly interested in (since they would be > beneficial for Pipenv) are: > > > > * VcsSupport > > * PackageFinder > > * WheelBuilder (and everything that comes with it like the wheel cache, > preparer, unpack_url, etc.) > > All of those sound like reasonable things to consider. Although > "everything that comes with" the WheelBuilder is quite a lot, in > practice (you didn't explicitly mention the requirement object, and > that's a big one). > > I don't think anyone has a problem with this sort of stuff in > principle, it's just getting the resources to do it. I wonder if any > sort of sponsorship would offer an option here? Although it's not just > the initial development, we'd need to make sure we had sustainable > support (or at least, it wasn't any worse than what we have now...) > >
-- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/4NT5UYTXKYLRD4NNKJKHKXCHAZSXAUCI/
