David Knupp has posted comments on this change. ( http://gerrit.cloudera.org:8080/9798 )
Change subject: IMPALA-6731: Use private index in bootstrap_virtualenv ...................................................................... Patch Set 4: I've always been a little perplexed by the complexity of the way we manage our python environments. This review seems like a good context to bring this up. >From past experience, a familiar pattern to me would be set up the environment >once, use "pip freeze" to generate a requirements.txt, and then use that >requirements.txt to set up future environments as needed. As an experiment, I took a centos69 machine with python 2.6.6 installed. I had to make sure that python-devel and cyrus-sasl-devel were installed, and that the Kudu repo was added to /etc/yum.repos.d/. Then I went to machine which has an Impala dev environment set up, and pulled this list: $ pip freeze You are using pip version 7.1.0, however version 9.0.3 is available. You should consider upgrading via the 'pip install --upgrade pip' command. AllPairs==2.0.1 apipkg==1.4 argparse==1.4.0 bitarray==0.8.1 boto3==1.2.3 botocore==1.3.30 cm-api==10.0.0 Cython==0.23.4 docopt==0.6.2 docutils==0.12 ecdsa==0.13 execnet==1.4.0 Fabric==1.10.2 Flask==0.10.1 futures==3.0.5 hdfs==2.0.2 impyla==0.14.0 ipython==1.2.1 itsdangerous==0.24 Jinja2==2.8 jmespath==0.9.0 kazoo==2.2.1 kudu-python==1.2.0 MarkupSafe==0.23 numpy==1.10.4 ordereddict==1.1 paramiko==1.15.2 pbr==1.8.1 pexpect==3.3 pg8000==1.10.2 prettytable==0.7.2 psutil==0.7.1 py==1.4.32 pycrypto==2.6.1 pyparsing==2.0.3 pytest==2.9.2 pytest-random==0.2 pytest-xdist==1.15.0 python-dateutil==2.5.2 python-magic==0.4.11 pytz==2018.3 pywebhdfs==0.3.2 requests==2.7.0 sasl==0.1.3 setuptools-scm==1.15.4 sh==1.11 simplejson==3.3.0 six==1.9.0 sqlparse==0.1.15 texttable==0.8.3 thrift==0.9.0 thrift-sasl==0.1.0 virtualenv==13.1.0 Werkzeug==0.11.3 I saved this list (minus the warning about the outdated pip version) to a req.txt file on my test centos6.9 box. Then I created a virtualenv, and installed all of the packages into it with one command: $ pip install -i https://pypi.infra.cloudera.com/api/pypi/pypi-public/simple -r req.txt This seems successful, which I checked by running "pip freeze" in the new environment: (pip_test5) [systest@impala-centos6-client ~]$ pip freeze DEPRECATION: Python 2.6 is no longer supported by the Python core team, please upgrade your Python. A future version of pip will drop support for Python 2.6 AllPairs==2.0.1 apipkg==1.4 argparse==1.4.0 bitarray==0.8.1 boto3==1.2.3 botocore==1.3.30 cm-api==10.0.0 Cython==0.23.4 docopt==0.6.2 docutils==0.12 ecdsa==0.13 execnet==1.4.0 Fabric==1.10.2 Flask==0.10.1 futures==3.0.5 hdfs==2.0.2 impyla==0.14.0 ipython==1.2.1 itsdangerous==0.24 Jinja2==2.8 jmespath==0.9.0 kazoo==2.2.1 kudu-python==1.2.0 MarkupSafe==0.23 numpy==1.10.4 ordereddict==1.1 paramiko==1.15.2 pbr==1.8.1 pexpect==3.3 pg8000==1.10.2 prettytable==0.7.2 psutil==0.7.1 py==1.4.32 pycrypto==2.6.1 pyparsing==2.0.3 pytest==2.9.2 pytest-random==0.2 pytest-xdist==1.15.0 python-dateutil==2.5.2 python-magic==0.4.11 pytz==2018.3 pywebhdfs==0.3.2 requests==2.7.0 sasl==0.1.3 setuptools-scm==1.15.4 sh==1.11 simplejson==3.3.0 six==1.9.0 sqlparse==0.1.15 texttable==0.8.3 thrift==0.9.0 thrift-sasl==0.1.0 virtualenv==13.1.0 Werkzeug==0.11.3 Would something like this work for us? -- To view, visit http://gerrit.cloudera.org:8080/9798 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I85f75f1f1a305f3043e0910ab88a880eeb30f00b Gerrit-Change-Number: 9798 Gerrit-PatchSet: 4 Gerrit-Owner: Lars Volker <[email protected]> Gerrit-Reviewer: David Knupp <[email protected]> Gerrit-Reviewer: Lars Volker <[email protected]> Gerrit-Reviewer: Philip Zeyliger <[email protected]> Gerrit-Comment-Date: Mon, 26 Mar 2018 20:19:25 +0000 Gerrit-HasComments: No
