IMPALA-6863: Make pip_download.py honor redirects. As part of our continuing woes with PyPi infrastructure, we've now seen redirects. Following redirects seems like the right thing to do, so I've changed the downloader code to follow them.
I checked that this is available in Python 2.6. The build failure signature looks like: Downloading AllPairs-2.0.1.tar.gz from https://pypi.python.org/packages/04/44/cb85d029b33ddfa7b5a27ae523434c2230f11a89d1f0c73203e9007a6eb0/AllPairs-2.0.1.tar.gz ('http error', 302, 'Found', <httplib.HTTPMessage instance at 0x7fbf7819b050>) Download failed after several attempts. Warning: Unable to download Python requirements. Warning: bootstrap_virtualenv or other Python-based tooling may fail. Change-Id: Ic7551cec43a2d378df7e3cc7d521ace338b56ba2 Reviewed-on: http://gerrit.cloudera.org:8080/10083 Reviewed-by: Philip Zeyliger <[email protected]> Tested-by: Philip Zeyliger <[email protected]> Reviewed-by: Lars Volker <[email protected]> Project: http://git-wip-us.apache.org/repos/asf/impala/repo Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/eaf66172 Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/eaf66172 Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/eaf66172 Branch: refs/heads/master Commit: eaf66172df113dbf10cdb0a08a2bc51e4077ca38 Parents: ea698cd Author: Philip Zeyliger <[email protected]> Authored: Mon Apr 16 16:19:57 2018 -0700 Committer: Philip Zeyliger <[email protected]> Committed: Tue Apr 17 19:49:55 2018 +0000 ---------------------------------------------------------------------- infra/python/deps/pip_download.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/impala/blob/eaf66172/infra/python/deps/pip_download.py ---------------------------------------------------------------------- diff --git a/infra/python/deps/pip_download.py b/infra/python/deps/pip_download.py index 6fbb683..0cce9e9 100755 --- a/infra/python/deps/pip_download.py +++ b/infra/python/deps/pip_download.py @@ -28,7 +28,7 @@ import re import sys from random import randint from time import sleep -from urllib import urlopen, URLopener +from urllib import urlopen, FancyURLopener NUM_DOWNLOAD_ATTEMPTS = 8 @@ -114,7 +114,7 @@ def download_package(pkg_name, pkg_version): expected_digest): print 'File with matching digest already exists, skipping {0}'.format(file_name) return True - downloader = URLopener() + downloader = FancyURLopener() pkg_url = '{0}/packages/{1}'.format(PYPI_MIRROR, path) print 'Downloading {0} from {1}'.format(file_name, pkg_url) downloader.retrieve(pkg_url, file_name)
