On 20 October 2017 at 23:53, Richard Jones <r1chardj0...@gmail.com> wrote: > Hiya Paul, > > There's a bunch of tooling out there using pip's internals to extending > pip's functionality. Could you please provide a some reasoning as to why > they're all going to be broken at pip 10, and possibly some guidance on how > to get that functionality back?
Hi Richard, There was a change to the pip docs that clarified the status of pip's internal code. The PR for that is at https://github.com/pypa/pip/pull/4743 but unfortunately it appears that the docs build has been failing so it hasn't yet made it to the formal pip docs site. To summarise, pip has *never* supported the use of its internal APIs by external code. Over time, we've had a steady trickle of people raising issues when their code broke because of doing so, and it usually turned out to be because they violated assumptions made by the pip code - such as that it's running in a single-threaded application, or it has sole control over the logging subsystem, or even just that you can run your own code after calling a pip API. We've explained this position regularly on those issues, but as is typical, people don't manage to find similar issues when raising new ones, so we spent a lot of time repeating ourselves. Coming up to pip 10, there's been a *lot* of internal work going on, and it's fairly likely that a decent chunk of code using pip's internal APIs will break anyway. We don't document internals changes, so we faced the possibility of an extended period of people raising issues saying "you broke my code" and us having no better response than "you shouldn't do that", which would likely hinder adoption of pip 10, and cause problems for the ecosystem as a whole. Rather than do this, we decided to make a clean compatibility break, where we could send out a clear message - "everything's moved, if that matters to you, then you were using unsupported functionality and you should find a better way". The breakage is still there (and certainly we made it affect more people, as there are no doubt some people who would have survived the pip 10 release unscathed if we hadn't done this) but at least it's clearly defined and contained. As to alternatives, we don't have all the answers here but I can offer the following suggestions: 1. There are a number of external packages that provide functionality equivalent to what pip does - packaging, wheel, distlib, pkg_resources are the ones I'm aware of. These are designed as libraries and so *do* provide supported APIs. 2. We're making a strong push to standardise *all* of the external interfaces that pip uses, so you should be far more able to write your own code if that's necessary, without worrying that it'll work differently than pip does, or that things will suddenly change and break your code. 3. You can call pip as a subprocess - that's always been supported and will remain so. We've added some automation-friendly features there (such as json output format for "pip list") and we'd be happy to add more if people want to submit PRs. Likely the biggest problems will be for people who call into the pip resolver and build APIs, as I don't know of any alternatives out there. But they were *definitely* breaking anyway, as we've made major changes to that code (and will be making more). Also, I should note that we didn't take this decision lightly. We don't have any particular objection in principle to having a supported stable pip API, it's just that we don't have anything even close to the resources needed to define a supported API, maintain it with acceptable backward compatibility guarantees, and support users who will inevitably be using it in unexpected and creative ways (we don't even have the resources to support the limited use of pip's internals that we see today). Also, pip was never designed originally as a library, so we *would* have to design that API from scratch. As I alluded to above, the existing internals code makes some strong assumptions about how it's called - assumptions that would be unacceptable in library code, but are fine in an application's internal code. Paul PS People who want to, can of course hunt out the new equivalents of the code they were using, and just switch. It's not like we can stop them. But the new names make it clear that they shouldn't be doing this, so there's an obvious warning there. PPS Please disregard xoviat's response. This is something we've been considering for a while, and most definitely not a spur of the moment decision. It's unfortunate that he was the one most immediately affected by the change when we made it, but that was just bad timing (we didn't suddenly do this just because "someone complained").
