[
https://issues.apache.org/jira/browse/IMPALA-10606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17312007#comment-17312007
]
ASF subversion and git services commented on IMPALA-10606:
----------------------------------------------------------
Commit 1142c7b58eeb26cf727327c0e42329184cfd52bf in impala's branch
refs/heads/master from Joe McDonnell
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=1142c7b ]
IMPALA-10606: Simplify impala-python virtualenv bootstrapping
Bootstrapping the impala-python virtualenv requires multiple
rounds of pip installs with different sets of requirements.
This consolidates the requirements.txt, stage2-requirements.txt,
and compiled-requirements.txt into a single requirements.txt.
This will make it easier to upgrade python packages.
This also splits out setuptools into its own
setuptools-requirements.txt. Setuptools is used during the
pip install for several of the dependencies. Recent versions
of setuptools do not support Python 2, but some of the install
tools (like easy_install) don't know how to pick a version
of setuptools that works with Python 2. Splitting it out to its
own requirements file lets us pin the version.
To make review easier, this does not change any of the versions
of the dependencies. It also leaves the stage2-requirements.txt
and compiled-requirements.txt split out in separate sections
of requirements.txt. These will later be turned into a single
alphabetical list.
Testing:
- Tested impala-python locally
- Ran GVO
Change-Id: I8e920e5a257f1e0613065685078624a50d59bf2e
Reviewed-on: http://gerrit.cloudera.org:8080/17226
Reviewed-by: Joe McDonnell <[email protected]>
Tested-by: Joe McDonnell <[email protected]>
> Simplify impala-python virtualenv requirements files
> ----------------------------------------------------
>
> Key: IMPALA-10606
> URL: https://issues.apache.org/jira/browse/IMPALA-10606
> Project: IMPALA
> Issue Type: Improvement
> Components: Infrastructure
> Affects Versions: Impala 4.0
> Reporter: Joe McDonnell
> Assignee: Joe McDonnell
> Priority: Major
>
> The impala-python virtualenv currently has complicated logic that can do
> multiple rounds of pip installs depending on whether the toolchain has been
> bootstrapped. For example, the packages in compile-requirements.txt are only
> installed if the toolchain GCC has been installed. The Kudu python client is
> only installed if Kudu has been downloaded. This was a workaround because
> bootstrap_toolchain.py required the impala-python virtualenv. The different
> stages allowed the basics to be installed, then bootstrap_toolchain.py can
> run, then the rest can be installed.
> The bootstrap_toolchain.py script no longer requires the impala-python
> virtualenv, so there is no need for such a complicated setup. The things
> bootstrapping the impala-python virtualenv can assume that the toolchain
> compiler is present. This would allow the requirements to be consolidated
> into a main requirements file that includes both compiled and non-compiled
> packages. A consolidated file makes it easier to update dependency versions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
