[
https://issues.apache.org/jira/browse/BEAM-5873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16665775#comment-16665775
]
Ahmet Altay edited comment on BEAM-5873 at 10/26/18 11:59 PM:
--------------------------------------------------------------
[~tvalentyn] could you take a look at this.
This is what I found so far:
# As Henning mentioned single bad worker will cause job failure. This is
because the same thing is retried 4 times and fails the same exact way.
# Apparently this become a common problem after pip 10 and usually problem
with the installation of pip ([https://github.com/pypa/pip/issues/5599).] I
suspect the following part of the container needs to change:
# Update to latest pip and setuptools. Note that the Debian apt repository#
contains an out-of-date version of Python's setuptools.RUN wget
https://bootstrap.pypa.io/get-pip.py -O - | python && \ # Remove pip cache.
rm -rf /root/.cache/pip
I am not sure what causes this to happen sometimes and does not make it
deterministic.
was (Author: altay):
[~tvalentyn] could you take a look at this.
This is what I found so far:
# As Henning mentioned single bad worker will cause job failure. This is
because the same thing is retried 4 times and fails the same exact way.
# Apparently this become a common problem after pip 10 and usually problem
with the installation of pip ([https://github.com/pypa/pip/issues/5599).] I
suspect the following part of the container needs to change:
# Update to latest pip and setuptools. Note that the Debian apt repository#
contains an out-of-date version of Python's setuptools.RUN wget
https://bootstrap.pypa.io/get-pip.py -O - | python && \ # Remove pip cache.
rm -rf /root/.cache/pip
Finally, I do not know what causes this to happen in some worker and not in the
others.
> Python test failure: "ImportError: No module named pip._internal"
> -----------------------------------------------------------------
>
> Key: BEAM-5873
> URL: https://issues.apache.org/jira/browse/BEAM-5873
> Project: Beam
> Issue Type: Bug
> Components: test-failures
> Reporter: Henning Rohde
> Assignee: Valentyn Tymofieiev
> Priority: Major
>
> https://scans.gradle.com/s/r55ln7mdibu2w/console-log?task=:beam-sdks-python:postCommitITTests#L163
> Logs:
> https://pantheon.corp.google.com/logs/viewer?resource=dataflow_step%2Fjob_id%2F2018-10-26_06_46_26-13501822612780835073&logName=projects%2Fapache-beam-testing%2Flogs%2Fdataflow.googleapis.com%252Fworker-startup&interval=NO_LIMIT&project=apache-beam-testing&minLogLevel=0&expandAll=false×tamp=2018-10-26T20:01:54.773000000Z&customFacets=&limitCustomFacetWidth=true&scrollTimestamp=2018-10-26T13:49:18.405228000Z
> Executing: /usr/local/bin/pip install
> /var/opt/google/dataflow/dataflow_python_sdk.tar[gcp]
> Debug: delayed tasks complete
> Debug: download complete
> Traceback (most recent call last):
> File "/usr/local/bin/pip", line 7, in <module>
> from pip._internal import main
> ImportError
> :
> No module named pip._internal
> /usr/local/bin/pip failed with exit status 1
> Maybe a flake?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
