On Wednesday, May 23, 2018, Michael Sarahan <msara...@anaconda.com> wrote:
> Thanks for starting this discussion, Victor! This topic is something > we're very interested in at Anaconda. > > I'd like to generalize the problem statement to the question of "how can > we make pip behave well when it is sharing package management with > something else? Similarly, how can we make the something else behave well > with pip?" > > We all share the pain of trying to have two package managers effectively > manage the same space. The alternate-folder-for-pip approach is a good > idea, but ultimately has issues that you pointed out. > > After several great conversations with many people at PyCon last week, I > came to the conclusion that conda and pip probably won't ever interoperate > very well. In order for them to do so, conda must respect all of pip's > constraints during its solving of dependencies, and likewise, pip must > respect conda's constraints. We are investigating the first of those > options and having some promising initial results, but the inverse is not > something that seems feasible. It amounts to pip having a solver that is > at least as good as the best supported package manager, and pip learning > about *all* of any other package managers that it claims to be compatible > with. That doesn't seem like a viable project. > What's the status on this? AFAIU, depsolving setup.py packages is blocked because it's necessary to execute setup.py to get the conditional requirements? https://www.pypa.io/en/latest/roadmap/#pip-dependency-resolution > > Ultimately, I believe a better approach is for the PyPA to define a > minimal set of functionality and interfaces to PyPI that any package > manager claiming to manage python packages must implement. Pip can be a > reference implementation of that specification. Any distributor (Red Hat, > Canonical, Homebrew, Anaconda) could then have their own implementations > that use their solvers, but also can install software from PyPI at user's > request, or as a fall-through when a native package is unavailable. > Metadata compatibility and adapter registration could help solve for this; though there's no money and not much demand. #PEP426JSONLD > > User interface could be unified by having "pip" on distributions be a > wrapper around the native package manager, matching the exact minimal > behavior of pip. > Would `sudo pip` then be the only way? > > The same kind of approach may also be good for virtual environments, but > it seems like there's less contention there. Conda is different enough > from virtualenv that we get some friction, but I think and hope we can > smooth that out over time. > `conda install pip` and `conda export -f environment.yml` seem to work okay (for Python, R, nodejs but not yet npm,)? Solving dependencies in a container is still the correct way to avoid cruft, IMHO. > > > Best, > Michael > > On Wednesday, May 23, 2018, Victor Stinner <vstin...@redhat.com> wrote: > > Hi, > > > > pip is currently not well integrated on Linux: it conflicts with the > > system package manager like apt or rpm. When pip writes files into > > /usr, it can replace files written by the system package manager and > > so create different kind of issues. For example, if you check the > > system integry, you will likely see that some Python files have been > > modified. > > > > I would like to open a discussion to see how each Linux vendor handles > > the issue, and see if a common solution can be designed. > > > > Debian uses /usr for apt-get install and /usr/local for distutils and > > "sudo pip". > > > > Fedora decided to change pip to install files into /usr/local by > > default, instead of /usr, so "sudo pip install" doesn't replace files > > installed by dnf (Fedora package manager): > > https://fedoraproject.org/wiki/Changes/Making_sudo_pip_safe > > > > It gives you 3 main places to install Python code: /usr (managed by > > dnf), /usr/local (managed by sudo pip), $HOME/.local (managed by pip > > --user). > > > > Would it make sense to make the Fedora/Debian change upstream? At > > least, give an opt-in option for Linux vendors to use /usr/local? > > > > I propose to make the change upstream because there are still issues, > > and I don't want to be alone to have to fix them :-) It should be > > easier if we agree on a filesystem layout and an implementation, so > > we can collaborate on issues! > > > > > > Issues with the current Fedora implementation: > > > > (1) When Python is embedded in an application, there is an issue with > > the current heuristic to decide if /usr/local should be added to > > sys.path: > > > > https://bugzilla.redhat.com/show_bug.cgi?id=1532287 > > > > (2) On Fedora, "sudo pip install -U" currently removes old code from > > /usr and install the new one in /usr/local. We should leave /usr > > unchanged, since only dnf should touch /usr. > > > > https://bugzilla.redhat.com/show_bug.cgi?id=1550368#c24 > > > > The implementation is made of a single patch on the Python site module: > > > > https://src.fedoraproject.org/rpms/python3/blob/master/f/002 > 51-change-user-install-location.patch > > > > -- > > > > There are two issues related to the "sudo pip" change, but they > > already exist when pip is installed in $HOME/.local: > > > > (3) Priority issue between PATH and PYTHONPATH directories. > > > > When the user runs "pip", the pip binary may come from /usr, > > /usr/local or $HOME/.local/bin, but the Python pip module ("import > > pip") may come from a different path. Which binary and which module > > should be used? > > > > Obvisouly, users can replace these two environment variables... > > > > (4) Related to (3). Running "pip" may run pip binary of one pip > > version, but pick the "pip" Python module of another pip version. > > > > For example, pip9 binary from /usr/bin/pip, but pip10 module from > /usr/local. > > > > > > Fedora works around issue (4) with a downstream patch on pip: > > > > https://src.fedoraproject.org/rpms/python-pip/blob/master/f/ > pip9-allow-pip10-import.patch > > > > -- > > > > I don't well well how Linux distributions handle the issue with "sudo > > pip". So don't hesitate to correct me if I'm wrong :-) My goal is > > just to start a discussion about a common "upstream" solution. > > > > Victor > > -- > > Distutils-SIG mailing list > > distutils-sig@python.org > > https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ > > Message archived at https://mail.python.org/mm3/ar > chives/list/distutils-sig@python.org/message/OLGLHTSHLEPLHUT > TVNU6L5QFTMNFIB6Z/ > > >
-- Distutils-SIG mailing list distutils-sig@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/D3TQ2XPCE2C3SX4XVJYJWLLMI3WIZWSX/
