Hi again maintainer,
As Yannick suggests, this seems to have been caused by this change to requests
back in July:
https://github.com/kennethreitz/requests/commit/47d0517d66e8cf5832768262221f0357ae134ad1
Now that requests.compat no longer imports IncompleteRead, it won't be
available for pip to use.
There are two sequential commits in pip which remove its use of
IncompleteRead:
https://github.com/pypa/pip/commit/53eab2357e4425436b51e109bfaedcb9f3772072
https://github.com/pypa/pip/commit/e77c0c573c059bb4a86b2c5c06835f57e4af6c47
These were merged as part of the PR: https://github.com/pypa/pip/pull/2024
Unfortunately, the commits themselves don't contain much context for the
change. It looks like it was perhaps made to satisfy CI warnings?
On Tue, Oct 21, 2014 at 10:14:49PM +0200, Yannick Roehlly wrote:
> A quick fix would be to use "from httplib import IncompleteRead" in the
> de-vendorize.patch (even if the IncompleteRead will not be raised by requests
> any more).
I've attached a patch that does this (with a minor change to support Python3),
and tested that it works.
I've also attached a patch which backports the two pip commits into the current
python-pip source package.
Both of these fix the current problem, and allow pip to work with both the
version of requests the Debian package, and the current version from PyPI,
which is likely to be installed by users. I'm not sure which is preferable, so
I've included both (only one should be applied, of course).
> Note that there may be other problems with pip using the upgraded requests.
This is probably also worth reiterating. I wonder how hard it would be to
change system pip to always use system versions of libraries. There's no
guarantee that a change to the internals of requests doesn't break pip again
tomorrow.
Thanks again!
Chris
On Fri, Jan 23, 2015 at 01:34:16AM -0800, Chris Kuehl wrote:
> Hi maintainer,
>
> I am able to reproduce this reliably.
>
> On a clean jessie or sid system (does not happen on wheezy):
>
> root# apt-get install python3 python3-pip
> root# pip3 install --upgrade requests
>
> After this, every invocation of pip3 fails:
>
> root# pip3
> Traceback (most recent call last):
> File "/usr/bin/pip3", line 9, in <module>
> load_entry_point('pip==1.5.6', 'console_scripts', 'pip3')()
> File "/usr/lib/python3/dist-packages/pkg_resources.py", line 356, in
> load_entry_point
> return get_distribution(dist).load_entry_point(group, name)
> File "/usr/lib/python3/dist-packages/pkg_resources.py", line 2476, in
> load_entry_point
> return ep.load()
> File "/usr/lib/python3/dist-packages/pkg_resources.py", line 2190, in
> load
> ['__name__'])
> File "/usr/lib/python3/dist-packages/pip/__init__.py", line 61, in
> <module>
> from pip.vcs import git, mercurial, subversion, bazaar # noqa
> File "/usr/lib/python3/dist-packages/pip/vcs/mercurial.py", line 9, in
> <module>
> from pip.download import path_to_url
> File "/usr/lib/python3/dist-packages/pip/download.py", line 25, in
> <module>
> from requests.compat import IncompleteRead
> ImportError: cannot import name 'IncompleteRead'
>
> The version of requests which the upgrade installs is currently 2.5.1.
>
> This renders pip unusable for many cases, and can break not just pip but
> also other Python packages. In particular, any Python package that
> depends on requests which you upgrade using system pip will permanently
> break pip.
>
> Thanks for looking into this -- I'm more than happy to help with testing
> any fixes!
>
> Happy Friday,
> Chris
From: Chris Kuehl <cku...@ocf.berkeley.edu>
Date: Fri, 23 Jan 2015 14:54:24 -0800
Subject: [PATCH] Don't raise IncompleteRead if response reads fails
This is a backport of the following commits onto the python-pip package.
https://github.com/pypa/pip/commit/53eab2357e4425436b51e109bfaedcb9f3772072
https://github.com/pypa/pip/commit/e77c0c573c059bb4a86b2c5c06835f57e4af6c47
This fixes Debian bug#744145 by allowing pip to continue to work with
upgraded versions of the requests library.
---
pip/download.py | 35 +++++++++++++++++++++++++++--------
1 file changed, 27 insertions(+), 8 deletions(-)
diff --git a/pip/download.py b/pip/download.py
index adff2ec..768e7ba 100644
--- a/pip/download.py
+++ b/pip/download.py
@@ -22,8 +22,7 @@ from pip.log import logger
import requests, six
from requests.adapters import BaseAdapter
from requests.auth import AuthBase, HTTPBasicAuth
-from requests.compat import IncompleteRead
-from requests.exceptions import InvalidURL, ChunkedEncodingError
+from requests.exceptions import InvalidURL
from requests.models import Response
from requests.structures import CaseInsensitiveDict
@@ -416,12 +415,32 @@ def _download_url(resp, link, temp_location):
def resp_read(chunk_size):
try:
# Special case for urllib3.
- try:
- for chunk in resp.raw.stream(
- chunk_size, decode_content=False):
- yield chunk
- except IncompleteRead as e:
- raise ChunkedEncodingError(e)
+ for chunk in resp.raw.stream(
+ chunk_size,
+ # We use decode_content=False here because we do
+ # want urllib3 to mess with the raw bytes we get
+ # from the server. If we decompress inside of
+ # urllib3 then we cannot verify the checksum
+ # because the checksum will be of the compressed
+ # file. This breakage will only occur if the
+ # server adds a Content-Encoding header, which
+ # depends on how the server was configured:
+ # - Some servers will notice that the file isn't a
+ # compressible file and will leave the file alone
+ # and with an empty Content-Encoding
+ # - Some servers will notice that the file is
+ # already compressed and will leave the file
+ # alone and will add a Content-Encoding: gzip
+ # header
+ # - Some servers won't notice anything at all and
+ # will take a file that's already been compressed
+ # and compress it again and set the
+ # Content-Encoding: gzip header
+ #
+ # By setting this not to decode automatically we
+ # hope to eliminate problems with the second case.
+ decode_content=False):
+ yield chunk
except AttributeError:
# Standard file-like object.
while True:
--
2.1.4
From: Chris Kuehl <cku...@ocf.berkeley.edu>
Date: Fri, 23 Jan 2015 15:36:16 -0800
Subject: [PATCH] Import IncompleteRead from httplib instead of requests
New versions of requests no longer have IncompleteRead available in
requests.compat. One simple workaround is to import it ourselves.
This fixes Debian bug#744145 by allowing pip to continue to work with
upgraded versions of the requests library.
---
debian/patches/de-vendorize.patch | 8 ++++++--
1 file changed, 6 insertions(+), 2 deletions(-)
diff --git a/debian/patches/de-vendorize.patch b/debian/patches/de-vendorize.patch
index c5daf22..2a04dd5 100644
--- a/debian/patches/de-vendorize.patch
+++ b/debian/patches/de-vendorize.patch
@@ -14,7 +14,7 @@ Forwarded: not-needed
class PrettyHelpFormatter(optparse.IndentedHelpFormatter):
--- a/pip/download.py
+++ b/pip/download.py
-@@ -19,13 +19,13 @@
+@@ -19,13 +19,17 @@
create_download_cache_folder, cache_download)
from pip.vcs import vcs
from pip.log import logger
@@ -28,10 +28,14 @@ Forwarded: not-needed
+import requests, six
+from requests.adapters import BaseAdapter
+from requests.auth import AuthBase, HTTPBasicAuth
-+from requests.compat import IncompleteRead
+from requests.exceptions import InvalidURL, ChunkedEncodingError
+from requests.models import Response
+from requests.structures import CaseInsensitiveDict
++try:
++ from httplib import IncompleteRead
++except ImportError: # python3
++ from http.client import IncompleteRead
++
__all__ = ['get_file_content',
'is_url', 'url_to_path', 'path_to_url',
--
2.1.4
