> On Dec 19, 2014, at 5:36 AM, Matthias Klose <d...@debian.org> wrote: > > On 12/19/2014 07:40 AM, Donald Stufft wrote: >> >>> On Dec 19, 2014, at 1:16 AM, Matthias Klose <d...@debian.org> wrote: >>> >>> On 12/19/2014 06:57 AM, Donald Stufft wrote: >>>> Ok, I hope I did this right. I made a sid system and updated/upgraded it >>>> then I >>>> did apt-get source python3.4 and used quilt to edit ensurepip-wheels.diff >>>> in order >>>> to fix pip inside of a virtual environment and then I added a new patch >>>> ensurepip-disabled.diff which I adapted from the one you have on Python >>>> 2.7 to >>>> disable ensurepip when running outside of a virtual environment. >>>> >>>> Output: https://bpaste.net/show/821a977c5f3c >>>> ensurepip-wheels.diff: https://bpaste.net/show/4e5852496eae >>>> ensurepip-disabled.diff: https://bpaste.net/show/5ed93f1df2fa >>>> >>>> I built the package, installed it and tested it (that’s what you see in >>>> the output >>>> link). >>> >>> this patch restores the behaviour to install the wheels twice, one time >>> into a >>> temporary directory, and one time into a directory in the venv? why is this >>> double copy needed? If you require the wheels permanently in the virtual >>> environment, why would you need the copy in the tempdir? >>> >> >> It doesn’t copy them twice, it copies the wheels named in >> /usr/share/python-wheels/%s.dependencies of which there is only currently the >> one file, pip.dependencies whose contents are: >> >> chardet >> colorama >> distlib >> html5lib >> requests >> setuptools >> six >> urllib3 >> >> Those are the dependencies that pip itself has, but importantly it’s not pip >> itself. >> >> Then it copies the projects named in the variable _PROJECTS, which is >> currently >> setuptools and pip, into a temporary directory. This isn’t strictly required >> for Debian I think, > > so they are copied twice.
The only thing copied twice is the setuptools wheel. Everything else is copied once. The setuptools wheel is copied twice because it is both desired to be installed into the virtual environment AND we want it in the private python-wheel directory. It’s two distinct lists of wheels copied into two different directories for two different purposes. The *only* overlap is the setuptools Wheel because it serves two purposes. > >> upstream ensurepip copies the bundled pip/setuptools wheels >> into a temporary directory because the stdlib might be zipped but I don’t >> think >> Debian ships a zipped stdlib. > > correct. > >> Anyways, once you have the pip and the setuptools wheels copied it then does >> the >> normal ensurepip logical which is basically >> ``pip install —no-index —find-links /tmp/dir/ pip setuptools`` to actually >> install >> pip and setuptools into the virtual environment. > > I don't see the pip wheel itself copied into the venv. the current loop only > iterates over the project's (pip and setuptools) dependency wheels, not the > pip > and setuptools wheels itself. These two wheels are only copied into the > temporary directory. Yes, setuptools is in pip's dependency list, so it is > copied. setuptools is in both lists because we want to install it into the virtual environment so that other people can use it (such as importing it into their setup.py) and we want it to be available to pip (because pip uses it) without people being able to break pip by upgrading / downgrading / uninstalling setuptools. > >> Essentially the first copy, into sys.prefix/lib/python-wheels is there to >> approximate >> the bundled dependencies that pip normally has (we don’t want to install >> these into >> the virtual environment because then ``pip uninstall requests`` inside of a >> virtual >> env would break pip without a good way to fix it which is one of the reasons >> pip >> bundles to begin with), and the second copy is just there to give pip a >> directory >> with a pip and setuptools wheels in it so it can install pip and setuptools >> into >> the virtual environment normally. > >> Does that make sense? > > I'm still unsure why you make the distinction between the temporary and the > permanent directory. as it looks like, the setuptools in python-wheels will > never be used. > The distinction is this: The permanent directory Wheels are never installed, pip has been modified by Debian to add all of the .whl files from sys.prefix/lib/python-wheels to the front of sys.path so that when pip does ``import requests`` or ``import pkg_resources`` it’ll use the copy that is inside of the .whl files sitting in sys.prefix/lib/python-wheels and it doesn’t depend what’s actually installed into the virtual environment. You can think of the permanent wheel directory as a sort of static linking for pip inside of a virtual environment, it’s Debian making pip work like pip was designed to work inside of a virtual environment even though Debian has patched it to change that in the system. The temporary directory is used by ensurepip to work around the fact that the standard library can be zipped because we *need* to have the pip and the setuptools wheels inside of a directory so that ensurepip can shell out to pip and tell it to install the pip and setuptools wheels into the virtual environment. Even though setuptools is installed here pip won’t use it because pip adds the stuff from sys.prefix/lib/python-wheels to the front of sys.path. It’s only ever used by end users for things like setup.py and such. The copy into the permanent directory is absolutely required on Debian because it’s how Debian “re-bundles” pip’s dependencies inside of a virtual environment, the copy into a temporary directory is upstream behavior to work around the fact that the standard library might be zipped (and the wheel files we need inside of it). Since Debian won’t zip the stdlib *that* copy can be removed and ensurepip adjusted to install directly from the /usr/share/python-wheels directory. However it’s not as simple as just removing the call to copy_wheels because that has additional logic that needs to happen as well and things will break if it doesn’t. I do not think it’s reasonable to break a documented feature of Python just to avoid copying two files into a temporary directory when creating a virtual environment. It does not negatively impact the long term storage of the OS (since it’s cleaned up) nor is it unsafe (since it’s a securely created temporary directory). The only downside is that ensurepip runs slightly slower. I might be able to find time to adjust the patch again to optimize that use case but I already burned time I don’t have right now to fix the patch after your adjustments broke it and I’d really rather not have to bail on my other commitments to spend a bunch of time making a tiny optimization that nobody is ever going to notice anyways. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
