Repository: impala Updated Branches: refs/heads/2.x e3bdc3e24 -> 264bd0786
IMPALA-6863: Make pip_download.py honor redirects. As part of our continuing woes with PyPi infrastructure, we've now seen redirects. Following redirects seems like the right thing to do, so I've changed the downloader code to follow them. I checked that this is available in Python 2.6. The build failure signature looks like: Downloading AllPairs-2.0.1.tar.gz from https://pypi.python.org/packages/04/44/cb85d029b33ddfa7b5a27ae523434c2230f11a89d1f0c73203e9007a6eb0/AllPairs-2.0.1.tar.gz ('http error', 302, 'Found', <httplib.HTTPMessage instance at 0x7fbf7819b050>) Download failed after several attempts. Warning: Unable to download Python requirements. Warning: bootstrap_virtualenv or other Python-based tooling may fail. Change-Id: Ic7551cec43a2d378df7e3cc7d521ace338b56ba2 Reviewed-on: http://gerrit.cloudera.org:8080/10083 Reviewed-by: Philip Zeyliger <[email protected]> Tested-by: Philip Zeyliger <[email protected]> Reviewed-by: Lars Volker <[email protected]> (cherry picked from commit eaf66172df113dbf10cdb0a08a2bc51e4077ca38) Reviewed-on: http://gerrit.cloudera.org:8080/10093 Tested-by: Impala Public Jenkins <[email protected]> Project: http://git-wip-us.apache.org/repos/asf/impala/repo Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/728b9aec Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/728b9aec Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/728b9aec Branch: refs/heads/2.x Commit: 728b9aeca79bfacbf4c990d822c168e512837cbf Parents: e3bdc3e Author: Philip Zeyliger <[email protected]> Authored: Mon Apr 16 16:19:57 2018 -0700 Committer: Impala Public Jenkins <[email protected]> Committed: Wed Apr 18 02:41:20 2018 +0000 ---------------------------------------------------------------------- infra/python/deps/pip_download.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/impala/blob/728b9aec/infra/python/deps/pip_download.py ---------------------------------------------------------------------- diff --git a/infra/python/deps/pip_download.py b/infra/python/deps/pip_download.py index 6fbb683..0cce9e9 100755 --- a/infra/python/deps/pip_download.py +++ b/infra/python/deps/pip_download.py @@ -28,7 +28,7 @@ import re import sys from random import randint from time import sleep -from urllib import urlopen, URLopener +from urllib import urlopen, FancyURLopener NUM_DOWNLOAD_ATTEMPTS = 8 @@ -114,7 +114,7 @@ def download_package(pkg_name, pkg_version): expected_digest): print 'File with matching digest already exists, skipping {0}'.format(file_name) return True - downloader = URLopener() + downloader = FancyURLopener() pkg_url = '{0}/packages/{1}'.format(PYPI_MIRROR, path) print 'Downloading {0} from {1}'.format(file_name, pkg_url) downloader.retrieve(pkg_url, file_name)
