[
https://issues.apache.org/jira/browse/BEAM-5873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16677597#comment-16677597
]
Valentyn Tymofieiev commented on BEAM-5873:
-------------------------------------------
[https://github.com/pypa/pip/issues/5599] describes the problem that happens
when one is trying to update pip, while also having an older distribution of
pip already installed.
When we boot the worker, we do not install pip, as it is already preinstalled
in the container, so I suspect the rootcause may be different, especially since
we use the same worker harness container on all workers, but:
- only some workers hit this problem.
- the problem is permanent if the worker hits it.
To gain more insight, we can try to reproduce the problem and inspect the
contents of the worker container, while worker VM is running.
The error keeps happening:
[https://pantheon.corp.google.com/logs/viewer?project=apache-beam-testing&resource=dataflow_step%2Fjob_id%2F2018-11-04_13_47_29-13067618618283531682]
Also, another strange thing from Job logs is that we stage multiple versions of
setuptools with the pipeline:
{noformat}
I Downloading:
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-1026134618-198861.1540561578.198985/setuptools-40.0.0.zip
to /var/opt/google/tmp/download.4.129098878/file.0 (size: 849 Kb, MD5:
JgYwrhpkuvo53MU0BNY4KQ==)
I Downloading:
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-1026134618-198861.1540561578.198985/setuptools-39.2.0.zip
to /var/opt/google/tmp/download.2.330461292/file.0 (size: 851 Kb, MD5:
3U4/qDohv3v5xRAm3IpOWQ==)
D Debug: downloading 20:setuptools-40.4.2.zip [#8]
I Downloading:
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-1026134618-198861.1540561578.198985/setuptools-40.3.0.zip
to /var/opt/google/tmp/download.9.090587849/file.0 (size: 853 Kb, MD5:
+3zyufmqkyd3RnoZsOfxmA==)
{noformat}
> Python test failure: "ImportError: No module named pip._internal"
> -----------------------------------------------------------------
>
> Key: BEAM-5873
> URL: https://issues.apache.org/jira/browse/BEAM-5873
> Project: Beam
> Issue Type: Bug
> Components: test-failures
> Reporter: Henning Rohde
> Assignee: Valentyn Tymofieiev
> Priority: Major
>
> https://scans.gradle.com/s/r55ln7mdibu2w/console-log?task=:beam-sdks-python:postCommitITTests#L163
> Logs:
> https://pantheon.corp.google.com/logs/viewer?resource=dataflow_step%2Fjob_id%2F2018-10-26_06_46_26-13501822612780835073&logName=projects%2Fapache-beam-testing%2Flogs%2Fdataflow.googleapis.com%252Fworker-startup&interval=NO_LIMIT&project=apache-beam-testing&minLogLevel=0&expandAll=false×tamp=2018-10-26T20:01:54.773000000Z&customFacets=&limitCustomFacetWidth=true&scrollTimestamp=2018-10-26T13:49:18.405228000Z
> Executing: /usr/local/bin/pip install
> /var/opt/google/dataflow/dataflow_python_sdk.tar[gcp]
> Debug: delayed tasks complete
> Debug: download complete
> Traceback (most recent call last):
> File "/usr/local/bin/pip", line 7, in <module>
> from pip._internal import main
> ImportError
> :
> No module named pip._internal
> /usr/local/bin/pip failed with exit status 1
> Maybe a flake?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
