Joe McDonnell created IMPALA-12434:
--------------------------------------
Summary: Isolate shell/pkg_resources.py to keep it off of the
PYTHONPATH
Key: IMPALA-12434
URL: https://issues.apache.org/jira/browse/IMPALA-12434
Project: IMPALA
Issue Type: Task
Components: Infrastructure
Affects Versions: Impala 4.3.0
Reporter: Joe McDonnell
In some internal builds on Redhat 8, the pip install of impala-shell into the
Python 3 virtualenv can fail with this error:
{noformat}
Exception:
Traceback (most recent call last):
File
"impala/shell/build/py3_venv/lib64/python3.6/site-packages/pip/basecommand.py",
line 215, in main
status = self.run(options, args)
File
"impala/shell/build/py3_venv/lib64/python3.6/site-packages/pip/commands/install.py",
line 357, in run
wb.build(autobuilding=True)
File
"impala/shell/build/py3_venv/lib64/python3.6/site-packages/pip/wheel.py", line
753, in build
self.requirement_set.prepare_files(self.finder)
File
"impala/shell/build/py3_venv/lib64/python3.6/site-packages/pip/req/req_set.py",
line 381, in prepare_files
ignore_dependencies=self.ignore_dependencies))
File
"impala/shell/build/py3_venv/lib64/python3.6/site-packages/pip/req/req_set.py",
line 637, in _prepare_file
abstract_dist.prep_for_dist()
File
"impala/shell/build/py3_venv/lib64/python3.6/site-packages/pip/req/req_set.py",
line 130, in prep_for_dist
self.req_to_install.run_egg_info()
File
"impala/shell/build/py3_venv/lib64/python3.6/site-packages/pip/req/req_install.py",
line 417, in run_egg_info
self.setup_py, self.link,
File
"impala/shell/build/py3_venv/lib64/python3.6/site-packages/pip/req/req_install.py",
line 387, in setup_py
import setuptools # noqa
File
"impala/shell/build/py3_venv/lib64/python3.6/site-packages/setuptools/__init__.py",
line 12, in <module>
import setuptools.version
File
"impala/shell/build/py3_venv/lib64/python3.6/site-packages/setuptools/version.py",
line 1, in <module>
import pkg_resources
File "impala/shell/pkg_resources.py", line 50
def _bypass_ensure_directory(name, mode=0777):
^
SyntaxError: invalid token
make[3]: *** [shell/CMakeFiles/shell_python3_install.dir/build.make:70:
shell/CMakeFiles/shell_python3_install] Error 2
{noformat}
What seems to happen is shell/pkg_resources.py is on the PYTHONPATH and our
pkg_resources.py doesn't work with Python 3. If I move pkg_resources.py to a
different directory (like a legacy subdirectory) or if I add a
WORKING_DIRECTORY for the CMake command, it works fine. In general, keeping
pkg_resources.py away from our PYTHONPATH (which includes $IMPALA_HOME/shell)
seems like a good idea.
This problem doesn't reproduce on the regular upstream Impala build or on my
local Ubuntu 20 workstation build, but it reliably reproduces on an internal
build.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
