[
https://issues.apache.org/jira/browse/IMPALA-12117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17720621#comment-17720621
]
ASF subversion and git services commented on IMPALA-12117:
----------------------------------------------------------
Commit 9fb1274867a59b467617e6292a54fd1f0bb937ad in impala's branch
refs/heads/master from Joe McDonnell
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=9fb127486 ]
IMPALA-12117: Use separate cache dirs for shell pip installs
Pip sporadically hits an error when installing impala-shell into
a virtualenv. An example symptom is this (though the issue is
not specific to thrift):
WARNING: Skipping page https://pypi.org/simple/thrift/ because the
GET request got Content-Type: Unknown. The only supported
Content-Types are application/vnd.pypi.simple.v1+json,
application/vnd.pypi.simple.v1+html, and text/html
ERROR: Could not find a version that satisfies the requirement
thrift==0.16.0 (from impala-shell) (from versions: none)
ERROR: No matching distribution found for thrift==0.16.0
It appears that this error can occur when two pip processes
are installing into virtualenvs simultaneously and share a
cache directory. This happens for our impala-shell build,
because we are doing pip install for Python 2 and Python 3
simultaneously. The impala-python/impala-python3 virtualenvs
do not use a cache directory and are not impacted.
This changes the shell's pip install to give the Python 2 and
Python 3 separate cache directories. The cache directories are
placed in ~/.cache like the regular pip cache. These do not
consume much space (a couple MB).
Testing:
- Ran all-build-options-ub2004 ten times without seeing the failure
Change-Id: I3f834b9f8c8cbc09830745ad132677a2fe17e07b
Reviewed-on: http://gerrit.cloudera.org:8080/19813
Tested-by: Impala Public Jenkins <[email protected]>
Reviewed-by: Daniel Becker <[email protected]>
Reviewed-by: Michael Smith <[email protected]>
> Intermittent pip error when building shell's venv
> -------------------------------------------------
>
> Key: IMPALA-12117
> URL: https://issues.apache.org/jira/browse/IMPALA-12117
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
> Affects Versions: Impala 4.3.0
> Reporter: Joe McDonnell
> Assignee: Joe McDonnell
> Priority: Major
> Fix For: Impala 4.3.0
>
>
> The all-build-options-ub2004 job intermittently hits this pip error when
> building the shell virtualenvs:
>
> {noformat}
> WARNING: Skipping page https://pypi.org/simple/thrift/ because the GET
> request got Content-Type: Unknown. The only supported Content-Types are
> application/vnd.pypi.simple.v1+json, application/vnd.pypi.simple.v1+html, and
> text/html
> ERROR: Could not find a version that satisfies the requirement thrift==0.16.0
> (from impala-shell) (from versions: none)
> ERROR: No matching distribution found for thrift==0.16.0
> {noformat}
> This error doesn't always apply to thrift. It can be any of the shell
> dependencies. It doesn't seem to impact the impala-python or impala-python3
> virtualenv (which don't use prebuilt binaries).
> Searching around, a theory on the forums is that this can happen when there
> are multiple pip installs running simultaneously writing to the same pip
> cache. Impala builds a python 2 virtualenv and a python 3 virtualenv
> simultaneously, so that seems like a plausible theory.
> We should try using separate pip caches for the python 2 vs python 3 shell
> virtualenvs.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
