[
https://issues.apache.org/jira/browse/BEAM-4256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Łukasz Gajowy updated BEAM-4256:
--------------------------------
Description:
AFAIK the setupVirtualenv task should run "pip install" command in the virtual
environment. This works on my local machine, yet on Jenkins, when we run:
{code:java}
gradlew clean install -xcheck --stacktrace{code}
from within PerfkitBenchmarker tool, we get the following stacktrace:
{code:java}
> Task :beam-sdks-python:setupVirtualenv
Collecting tox==3.0.0
Using cached
https://files.pythonhosted.org/packages/e6/41/4dcfd713282bf3213b0384320fa8841e4db032ddcb80bc08a540159d42a8/tox-3.0.0-py2.py3-none-any.whl
Collecting grpcio-tools==1.3.5
Using cached
https://files.pythonhosted.org/packages/05/f6/0296e29b1bac6f85d2a8556d48adf825307f73109a3c2c17fb734292db0a/grpcio_tools-1.3.5-cp27-cp27mu-manylinux1_x86_64.whl
Collecting pluggy<1.0,>=0.3.0 (from tox==3.0.0)
Using cached
https://files.pythonhosted.org/packages/82/05/43e3947125a2137cba4746135c75934ceed1863f27e050fc560052104a71/pluggy-0.6.0-py2-none-any.whl
Requirement not upgraded as not directly required: six in
/usr/local/lib/python2.7/dist-packages (from tox==3.0.0) (1.11.0)
Requirement not upgraded as not directly required: virtualenv>=1.11.2 in
/usr/lib/python2.7/dist-packages (from tox==3.0.0) (15.0.1)
Collecting py>=1.4.17 (from tox==3.0.0)
Using cached
https://files.pythonhosted.org/packages/67/a5/f77982214dd4c8fd104b066f249adea2c49e25e8703d284382eb5e9ab35a/py-1.5.3-py2.py3-none-any.whl
Collecting grpcio>=1.3.5 (from grpcio-tools==1.3.5)
Using cached
https://files.pythonhosted.org/packages/0d/54/b647a6323be6526be27b2c90bb042769f1a7a6e59bd1a5f2eeb795bfece4/grpcio-1.11.0-cp27-cp27mu-manylinux1_x86_64.whl
Collecting protobuf>=3.2.0 (from grpcio-tools==1.3.5)
Using cached
https://files.pythonhosted.org/packages/9d/61/54c3a9cfde6ffe0ca6a1786ddb8874263f4ca32e7693ad383bd8cf935015/protobuf-3.5.2.post1-cp27-cp27mu-manylinux1_x86_64.whl
Requirement not upgraded as not directly required: enum34>=1.0.4 in
/usr/local/lib/python2.7/dist-packages (from
grpcio>=1.3.5->grpcio-tools==1.3.5) (1.1.6)
Collecting futures>=2.2.0 (from grpcio>=1.3.5->grpcio-tools==1.3.5)
Using cached
https://files.pythonhosted.org/packages/2d/99/b2c4e9d5a30f6471e410a146232b4118e697fa3ffc06d6a65efde84debd0/futures-3.2.0-py2-none-any.whl
Requirement not upgraded as not directly required: setuptools in
/usr/local/lib/python2.7/dist-packages (from
protobuf>=3.2.0->grpcio-tools==1.3.5) (39.0.1)
Installing collected packages: pluggy, py, tox, protobuf, futures, grpcio,
grpcio-tools
Could not install packages due to an EnvironmentError: [Errno 13] Permission
denied: '/usr/local/lib/python2.7/dist-packages/pluggy-0.6.0.dist-info'
Consider using the `--user` option or check the permissions.
> Task :beam-model-job-management:shadowJar
> Task :beam-sdks-python:setupVirtualenv FAILED
> Task :beam-model-job-management:jar
> Task :beam-model-fn-execution:shadowJar
> Task :beam-sdks-java-core:shadowJar
Deprecated Gradle features were used in this build, making it incompatible with
Gradle 5.0.
See
https://docs.gradle.org/4.7/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 44s
133 actionable tasks: 126 executed, 5 from cache, 2 up-to-date{code}
It looks as if the virtualenv is not used while installing the needed
dependencies (why is it trying to install pluggy in /usr/local/lib?). Therefore
it throws "PermissionDenied" exception.
This affects all Performance tests on Beam's Jenkins. Currently none of the
Jenkins jobs can build the whole repository code before the benchmark is
started.
was:
AFAIK the setupVirtualenv task should run "pip install" command in the virtual
environment. This works on my local machine, yet on Jenkins, when we run:
gradlew clean install -xcheck --stacktrace
from within PerfkitBenchmarker tool, we get the following stacktrace:
{code:java}
> Task :beam-sdks-python:setupVirtualenv
Collecting tox==3.0.0
Using cached
https://files.pythonhosted.org/packages/e6/41/4dcfd713282bf3213b0384320fa8841e4db032ddcb80bc08a540159d42a8/tox-3.0.0-py2.py3-none-any.whl
Collecting grpcio-tools==1.3.5
Using cached
https://files.pythonhosted.org/packages/05/f6/0296e29b1bac6f85d2a8556d48adf825307f73109a3c2c17fb734292db0a/grpcio_tools-1.3.5-cp27-cp27mu-manylinux1_x86_64.whl
Collecting pluggy<1.0,>=0.3.0 (from tox==3.0.0)
Using cached
https://files.pythonhosted.org/packages/82/05/43e3947125a2137cba4746135c75934ceed1863f27e050fc560052104a71/pluggy-0.6.0-py2-none-any.whl
Requirement not upgraded as not directly required: six in
/usr/local/lib/python2.7/dist-packages (from tox==3.0.0) (1.11.0)
Requirement not upgraded as not directly required: virtualenv>=1.11.2 in
/usr/lib/python2.7/dist-packages (from tox==3.0.0) (15.0.1)
Collecting py>=1.4.17 (from tox==3.0.0)
Using cached
https://files.pythonhosted.org/packages/67/a5/f77982214dd4c8fd104b066f249adea2c49e25e8703d284382eb5e9ab35a/py-1.5.3-py2.py3-none-any.whl
Collecting grpcio>=1.3.5 (from grpcio-tools==1.3.5)
Using cached
https://files.pythonhosted.org/packages/0d/54/b647a6323be6526be27b2c90bb042769f1a7a6e59bd1a5f2eeb795bfece4/grpcio-1.11.0-cp27-cp27mu-manylinux1_x86_64.whl
Collecting protobuf>=3.2.0 (from grpcio-tools==1.3.5)
Using cached
https://files.pythonhosted.org/packages/9d/61/54c3a9cfde6ffe0ca6a1786ddb8874263f4ca32e7693ad383bd8cf935015/protobuf-3.5.2.post1-cp27-cp27mu-manylinux1_x86_64.whl
Requirement not upgraded as not directly required: enum34>=1.0.4 in
/usr/local/lib/python2.7/dist-packages (from
grpcio>=1.3.5->grpcio-tools==1.3.5) (1.1.6)
Collecting futures>=2.2.0 (from grpcio>=1.3.5->grpcio-tools==1.3.5)
Using cached
https://files.pythonhosted.org/packages/2d/99/b2c4e9d5a30f6471e410a146232b4118e697fa3ffc06d6a65efde84debd0/futures-3.2.0-py2-none-any.whl
Requirement not upgraded as not directly required: setuptools in
/usr/local/lib/python2.7/dist-packages (from
protobuf>=3.2.0->grpcio-tools==1.3.5) (39.0.1)
Installing collected packages: pluggy, py, tox, protobuf, futures, grpcio,
grpcio-tools
Could not install packages due to an EnvironmentError: [Errno 13] Permission
denied: '/usr/local/lib/python2.7/dist-packages/pluggy-0.6.0.dist-info'
Consider using the `--user` option or check the permissions.
> Task :beam-model-job-management:shadowJar
> Task :beam-sdks-python:setupVirtualenv FAILED
> Task :beam-model-job-management:jar
> Task :beam-model-fn-execution:shadowJar
> Task :beam-sdks-java-core:shadowJar
Deprecated Gradle features were used in this build, making it incompatible with
Gradle 5.0.
See
https://docs.gradle.org/4.7/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 44s
133 actionable tasks: 126 executed, 5 from cache, 2 up-to-date{code}
It looks as if the virtualenv is not used while installing the needed
dependencies (why is it trying to install pluggy in /usr/local/lib?). Therefore
it throws "PermissionDenied" exception.
This affects all Performance tests on Beam's Jenkins. Currently none of the
Jenkins jobs can build the whole repository code before the benchmark is
started.
> Gradle setupVirtualenv task fails on Jenkins
> --------------------------------------------
>
> Key: BEAM-4256
> URL: https://issues.apache.org/jira/browse/BEAM-4256
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
> Reporter: Łukasz Gajowy
> Assignee: Ahmet Altay
> Priority: Critical
>
> AFAIK the setupVirtualenv task should run "pip install" command in the
> virtual environment. This works on my local machine, yet on Jenkins, when we
> run:
> {code:java}
> gradlew clean install -xcheck --stacktrace{code}
> from within PerfkitBenchmarker tool, we get the following stacktrace:
> {code:java}
> > Task :beam-sdks-python:setupVirtualenv
> Collecting tox==3.0.0
> Using cached
> https://files.pythonhosted.org/packages/e6/41/4dcfd713282bf3213b0384320fa8841e4db032ddcb80bc08a540159d42a8/tox-3.0.0-py2.py3-none-any.whl
> Collecting grpcio-tools==1.3.5
> Using cached
> https://files.pythonhosted.org/packages/05/f6/0296e29b1bac6f85d2a8556d48adf825307f73109a3c2c17fb734292db0a/grpcio_tools-1.3.5-cp27-cp27mu-manylinux1_x86_64.whl
> Collecting pluggy<1.0,>=0.3.0 (from tox==3.0.0)
> Using cached
> https://files.pythonhosted.org/packages/82/05/43e3947125a2137cba4746135c75934ceed1863f27e050fc560052104a71/pluggy-0.6.0-py2-none-any.whl
> Requirement not upgraded as not directly required: six in
> /usr/local/lib/python2.7/dist-packages (from tox==3.0.0) (1.11.0)
> Requirement not upgraded as not directly required: virtualenv>=1.11.2 in
> /usr/lib/python2.7/dist-packages (from tox==3.0.0) (15.0.1)
> Collecting py>=1.4.17 (from tox==3.0.0)
> Using cached
> https://files.pythonhosted.org/packages/67/a5/f77982214dd4c8fd104b066f249adea2c49e25e8703d284382eb5e9ab35a/py-1.5.3-py2.py3-none-any.whl
> Collecting grpcio>=1.3.5 (from grpcio-tools==1.3.5)
> Using cached
> https://files.pythonhosted.org/packages/0d/54/b647a6323be6526be27b2c90bb042769f1a7a6e59bd1a5f2eeb795bfece4/grpcio-1.11.0-cp27-cp27mu-manylinux1_x86_64.whl
> Collecting protobuf>=3.2.0 (from grpcio-tools==1.3.5)
> Using cached
> https://files.pythonhosted.org/packages/9d/61/54c3a9cfde6ffe0ca6a1786ddb8874263f4ca32e7693ad383bd8cf935015/protobuf-3.5.2.post1-cp27-cp27mu-manylinux1_x86_64.whl
> Requirement not upgraded as not directly required: enum34>=1.0.4 in
> /usr/local/lib/python2.7/dist-packages (from
> grpcio>=1.3.5->grpcio-tools==1.3.5) (1.1.6)
> Collecting futures>=2.2.0 (from grpcio>=1.3.5->grpcio-tools==1.3.5)
> Using cached
> https://files.pythonhosted.org/packages/2d/99/b2c4e9d5a30f6471e410a146232b4118e697fa3ffc06d6a65efde84debd0/futures-3.2.0-py2-none-any.whl
> Requirement not upgraded as not directly required: setuptools in
> /usr/local/lib/python2.7/dist-packages (from
> protobuf>=3.2.0->grpcio-tools==1.3.5) (39.0.1)
> Installing collected packages: pluggy, py, tox, protobuf, futures, grpcio,
> grpcio-tools
> Could not install packages due to an EnvironmentError: [Errno 13] Permission
> denied: '/usr/local/lib/python2.7/dist-packages/pluggy-0.6.0.dist-info'
> Consider using the `--user` option or check the permissions.
> > Task :beam-model-job-management:shadowJar
> > Task :beam-sdks-python:setupVirtualenv FAILED
> > Task :beam-model-job-management:jar
> > Task :beam-model-fn-execution:shadowJar
> > Task :beam-sdks-java-core:shadowJar
> Deprecated Gradle features were used in this build, making it incompatible
> with Gradle 5.0.
> See
> https://docs.gradle.org/4.7/userguide/command_line_interface.html#sec:command_line_warnings
> BUILD FAILED in 44s
> 133 actionable tasks: 126 executed, 5 from cache, 2 up-to-date{code}
> It looks as if the virtualenv is not used while installing the needed
> dependencies (why is it trying to install pluggy in /usr/local/lib?).
> Therefore it throws "PermissionDenied" exception.
> This affects all Performance tests on Beam's Jenkins. Currently none of the
> Jenkins jobs can build the whole repository code before the benchmark is
> started.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
