TL;DR By the end of today, I expect to upload pip 8.0.2 to unstable, finally bringing us up to the latest version.
It's been a long slog, with many people helping out. I hope that all the hard
work done to get us here means it will be much, much easier to track new pip,
virtualenv, and pyvenv updates going forward. And now, details!
pip requires the use of PEP 427 wheels, which are essentially zip files of
importable, pure-Python, bilingual code, with a little bit of metadata. Put a
wheel on sys.path and Python can import from it. pip and friends use wheels
because it ensures that they can't break themselves when their dependent
packages are updated. E.g. if pip depends on requests, and you `pip install
--upgrade requests` it's possible the new version would be incompatible with
pip. Then you'd have a broken pip on your hands. Remember that you can use
pip to install pre-release packages too.
So by putting the wheels on sys.path *before* anything else, even if you pip
install an incompatible requests, it won't break pip itself.
pip "vendorizes" (i.e. bundles) the wheels of its dependencies with known good
versions, but this breaks Debian policy, so we have to unbundle these.
Several upstreams use this technique, with requests and urllib3 as probably
the most popular examples. In all these cases, we have to remove the bundled
wheels and hope that the versions of their dependencies in Debian remain
compatible. It's a chore, but that's our job as distro developers, and it's
the best option given the conflicting requirements.
Fortunately, we can create our own wheels and use them with pip and friends.
Until now, we had to build the wheels at the time that the dependent packages
where built, and we had to have a corresponding foo-whl binary package in the
archives, because we could only use setup.py's bdist_wheel command (which
comes with python{,3}-wheel) to create these. With 15 run time dependencies
as of pip 7.1.2, this has become untenable, and was a huge reason why Debian's
pip has lagged so far behind upstream. (That, and a few ITPs for new
dependencies. ;)
Along comes dirtbike. https://github.com/paulproteus/dirtbike
While it has been much discussed and attempted over the years, it would be
really nice to have a tool that turned installed Python packages into wheels.
I think we discussed such a tool for Debian at last year's (two years ago?)
Debian BOF at Pycon. Anyway, Asheesh Laroia finally took hold of the, ahem,
wheel, and wrote the initial implementation, which he called dirtbike. Since
then I've been enhancing it to work with several oddball cases in the Debian
archive, and now it's good enough to turn pip's dependencies into wheels.
Yesterday I uploaded the last of the new pip dependencies (modulo any
surprises in 8.0.2 ;), and dirtbike so now all the pieces are in place. What
pip now does during its own build process, is turn its installed dependencies
into .whl files and puts them in the policy defined location. This allows us
to eliminate all those scattered -whl binary packages, and just
Build-Depend/Built-Using in pip to install the standard Python packages,
rewheel them, and put them in a python-pip-whl package. Going forward, this
will make life so much easier, because only python-pip needs to know anything
about wheels, even if new dependencies are added.
I have everything working locally. I need to update the master branch for
pip, which involves many curse words hurled at git-dpm (and that's a post for
another day!). I also have to update my local branch to 8.0.2, but I'm
assuming there won't be any new surprises there.
Eventually, I'll be using the same techique for virtualenv and pyvenv, but not
right away.
Huge thanks also go to Donald Stufft, who always is so helpful with various
technical details, and works really hard to understand our needs and balance
them with upstream's.
Finally, we need to update Python policy to describe the new wheel policy.
Fortunately, this is much better because it removes constraints on other
packages, and reduces the number of -whl binary packages in the distro. Yes,
I will update those packages that already ship -whls, or file bugs, but only
once I'm sure everything is working correctly.
Below is the policy diff, which I'll commit to the python-defaults bzr.
As always, once everything's uploaded, please do let me know via this list or
BTS, if you notice any problems with pip.
Cheers,
-Barry
=== modified file 'debian/python-policy.sgml'
--- debian/python-policy.sgml 2016-01-24 06:01:19 +0000
+++ debian/python-policy.sgml 2016-01-29 16:49:59 +0000
@@ -458,13 +458,12 @@
<p>
<url id="http://legacy.python.org/dev/peps/pep-0427/";
name="PEP 427">
- defines a built-package format called "wheels", which is a zip
- format archive containing Python code and a "dist-info" metadata
+ defines a built-package format called "wheels", a zip
+ archive containing Python code and a "dist-info" metadata
directory, in a single file named with the .whl suffix. As zip
files, wheels containing pure-Python can be put on sys.path and
modules in the wheel can be imported directly by Python's "import"
- statement. (Importing extension modules from wheels is not yet
- supported as of Python 3.4.)
+ statement.
</p><p>
Except as described below, packages must not build or provide
wheels. They are redundant to the established way of providing
@@ -475,18 +474,20 @@
</p><p>
A very limited set of wheel packages are available in the archive,
but these support the narrow purpose of enabling
- the <prgn>pip</prgn> tool, in a Debian policy compliant way. The
- set of packages providing wheels for this purpose are (by source
- package name): chardet, distlib, html5lib, python-colorama,
- python-pip, python-setuptools, python-urllib3, requests, and six.
+ the <prgn>pip</prgn>, <prgn>virtualenv</prgn>,
+ and <prgn>pyvenv</prgn> tools in a Debian policy compliant way.
+ These packages build their own dependent wheels through the use of
+ the <prgn>dirtbike</prgn> "rewheeling" tool, which takes installed
+ Debian packages and turns them back into wheels. Only universal
+ wheels (i.e. pure-Python, Python 2 and 3 compatible packages) are
+ supported. Since only the programs that require wheels need build
+ them, only they may provide <var>-whl</var> packages,
+ e.g. <package>python-pip-whl</package>.
</p><p>
- Wheel packages supporting <prgn>pyvenv</prgn> and <prgn>pip</prgn>
- are named with the <var>python-</var> prefix, and the <var>-whl</var>
- suffix, e.g. <package>python-chardet-whl</package>. When these
- binary packages are installed, their .whl files must be placed in
- the /usr/share/python-wheels directory. Such wheels must be built
- with the <tt>--universal</tt> flag so as to generate wheels
- compatible with both Python 2 and Python 3.
+ When these binary packages are installed, .whl files must be placed
+ in the /usr/share/python-wheels directory. The location inside a
+ virtual environment will be rooted in the virtual environment,
+ instead of in /usr.
</p>
</sect>
<sect id="package_names">
pgpP1KyKVV3HQ.pgp
Description: OpenPGP digital signature
