See
<https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/1418/display/redirect>
Changes:
------------------------------------------
[...truncated 738.72 KB...]
createTime: '2021-11-09T06:27:40.658965Z'
...021-11-09T06:27:40.658965Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)> at 0x7f8d6e862d68>
duration = None
def wait_until_finish(self, duration=None):
if not self.is_in_terminal_state():
if not self.has_job:
raise IOError('Failed to get the Dataflow job id.')
thread = threading.Thread(
target=DataflowRunner.poll_for_job_completion,
args=(self._runner, self, duration))
# Mark the thread as a daemon thread so a keyboard interrupt on the main
# thread will terminate everything. This is also the reason we will not
# use thread.join() to wait for the polling thread.
thread.daemon = True
thread.start()
while thread.is_alive():
time.sleep(5.0)
# TODO: Merge the termination code in poll_for_job_completion and
# is_in_terminal_state.
terminated = self.is_in_terminal_state()
assert duration or terminated, (
'Job did not reach to a terminal state after waiting indefinitely.')
if terminated and self.state != PipelineState.DONE:
# TODO(BEAM-1290): Consider converting this to an error log based on
# theresolution of the issue.
raise DataflowRuntimeException(
'Dataflow pipeline failed. State: %s, Error:\n%s' %
(self.state, getattr(self._runner, 'last_error_msg', None)),
> self)
E
apache_beam.runners.dataflow.dataflow_runner.DataflowRuntimeException: Dataflow
pipeline failed. State: FAILED, Error:
E Workflow failed. Causes: Job appears to be stuck. Several workers
have failed to start up in a row, and no worker has successfully started up for
this job. Last error reported: Unable to pull container image due to error:
image pull request failed with error: Error response from daemon: manifest for
us.gcr.io/apache-beam-testing/java-postcommit-it/java:20211109060114 not found:
manifest unknown: Failed to fetch "20211109060114" from request
"/v2/apache-beam-testing/java-postcommit-it/java/manifests/20211109060114"..
This is likely due to an invalid SDK container image URL. Please verify any
provided SDK container image is valid and that Dataflow workers have
permissions to pull image..
apache_beam/runners/dataflow/dataflow_runner.py:1643: DataflowRuntimeException
------------------------------ Captured log call -------------------------------
INFO apache_beam.runners.portability.stager:stager.py:303 Copying Beam SDK
"<https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/ws/src/sdks/python/build/apache-beam.tar.gz";>
to staging location.
WARNING root:environments.py:374 Make sure that locally built Python SDK
docker image has Python 3.6 interpreter.
INFO root:environments.py:380 Default Python SDK image for environment is
apache/beam_python3.6_sdk:2.35.0.dev
INFO root:environments.py:296 Using provided Python SDK container image:
gcr.io/cloud-dataflow/v1beta3/python36-fnapi:beam-master-20211015
INFO root:environments.py:304 Python SDK container image set to
"gcr.io/cloud-dataflow/v1beta3/python36-fnapi:beam-master-20211015" for Docker
environment
INFO
apache_beam.runners.portability.fn_api_runner.translations:translations.py:678
==================== <function pack_combiners at 0x7f8d70cd2378>
====================
INFO
apache_beam.runners.portability.fn_api_runner.translations:translations.py:678
==================== <function sort_stages at 0x7f8d70cd2ae8>
====================
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:454
Defaulting to the temp_location as staging_location:
gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:648
Starting GCS upload to
gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-1109062734-298501.1636439254.298983/icedtea-sound-Bdoi2wYa757-fzq5vconCy4SSQ22ZaOq7yuC98fKPs8.jar...
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:667
Completed GCS upload to
gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-1109062734-298501.1636439254.298983/icedtea-sound-Bdoi2wYa757-fzq5vconCy4SSQ22ZaOq7yuC98fKPs8.jar
in 0 seconds.
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:648
Starting GCS upload to
gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-1109062734-298501.1636439254.298983/jaccess-CMbK-IOdQPLKHEqCuDnE-yBk-VpbtVT-hgjbHRUGO78.jar...
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:667
Completed GCS upload to
gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-1109062734-298501.1636439254.298983/jaccess-CMbK-IOdQPLKHEqCuDnE-yBk-VpbtVT-hgjbHRUGO78.jar
in 0 seconds.
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:648
Starting GCS upload to
gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-1109062734-298501.1636439254.298983/localedata-ae5Z0L6ak4922fztWeWy7ajiWXdG3ubNrwerJRFoFj0.jar...
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:667
Completed GCS upload to
gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-1109062734-298501.1636439254.298983/localedata-ae5Z0L6ak4922fztWeWy7ajiWXdG3ubNrwerJRFoFj0.jar
in 0 seconds.
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:648
Starting GCS upload to
gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-1109062734-298501.1636439254.298983/nashorn-XHtz_UehGpYcLTOrATrTnMNVUgEVa_ttoWkPxnVfqTo.jar...
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:667
Completed GCS upload to
gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-1109062734-298501.1636439254.298983/nashorn-XHtz_UehGpYcLTOrATrTnMNVUgEVa_ttoWkPxnVfqTo.jar
in 0 seconds.
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:648
Starting GCS upload to
gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-1109062734-298501.1636439254.298983/cldrdata-k07I6K9W3X5KTQbcDIEsqM0LXyM18f0eR6IaJw-P_xk.jar...
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:667
Completed GCS upload to
gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-1109062734-298501.1636439254.298983/cldrdata-k07I6K9W3X5KTQbcDIEsqM0LXyM18f0eR6IaJw-P_xk.jar
in 0 seconds.
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:648
Starting GCS upload to
gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-1109062734-298501.1636439254.298983/dnsns-RGhCDg3GVOQVC2r6ka2N0hmI4eqQH6VobuoAnQ74MnE.jar...
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:667
Completed GCS upload to
gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-1109062734-298501.1636439254.298983/dnsns-RGhCDg3GVOQVC2r6ka2N0hmI4eqQH6VobuoAnQ74MnE.jar
in 0 seconds.
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:648
Starting GCS upload to
gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-1109062734-298501.1636439254.298983/beam-sdks-java-testing-expansion-service-testExpansionService-2.35.0-SNAPSHOT-UROKmaGNH684BTk9cZMtLqcho6glylYpdf6EKMYw3pA.jar...
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:667
Completed GCS upload to
gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-1109062734-298501.1636439254.298983/beam-sdks-java-testing-expansion-service-testExpansionService-2.35.0-SNAPSHOT-UROKmaGNH684BTk9cZMtLqcho6glylYpdf6EKMYw3pA.jar
in 4 seconds.
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:648
Starting GCS upload to
gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-1109062734-298501.1636439254.298983/dataflow_python_sdk.tar...
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:667
Completed GCS upload to
gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-1109062734-298501.1636439254.298983/dataflow_python_sdk.tar
in 0 seconds.
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:648
Starting GCS upload to
gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-1109062734-298501.1636439254.298983/pipeline.pb...
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:667
Completed GCS upload to
gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-1109062734-298501.1636439254.298983/pipeline.pb
in 0 seconds.
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:828
Create job: <Job
clientRequestId: '20211109062734299887-3474'
createTime: '2021-11-09T06:27:40.658965Z'
currentStateTime: '1970-01-01T00:00:00Z'
id:
'2021-11-08_22_27_39-3268535017721424543'
location: 'us-central1'
name: 'beamapp-jenkins-1109062734-298501'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2021-11-09T06:27:40.658965Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:830
Created job with id: [2021-11-08_22_27_39-3268535017721424543]
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:831
Submitted job: 2021-11-08_22_27_39-3268535017721424543
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:837 To
access the Dataflow monitoring console, please navigate to
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-11-08_22_27_39-3268535017721424543?project=apache-beam-testing
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:191
Job 2021-11-08_22_27_39-3268535017721424543 is in state JOB_STATE_RUNNING
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:43.427Z: JOB_MESSAGE_DETAILED: Autoscaling is enabled for job
2021-11-08_22_27_39-3268535017721424543. The number of workers will be between
1 and 1000.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:43.579Z: JOB_MESSAGE_DETAILED: Autoscaling was automatically
enabled for job 2021-11-08_22_27_39-3268535017721424543.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:49.646Z: JOB_MESSAGE_BASIC: Worker configuration:
e2-standard-2 in us-central1-b.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:50.174Z: JOB_MESSAGE_DETAILED: Expanding SplittableParDo
operations into optimizable parts.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:50.229Z: JOB_MESSAGE_DETAILED: Expanding CollectionToSingleton
operations into optimizable parts.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:50.317Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey
operations into optimizable parts.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:50.356Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step
check_odd/Group/CoGroupByKeyImpl/GroupByKey: GroupByKey not followed by a
combiner.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:50.422Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step
check_even/Group/CoGroupByKeyImpl/GroupByKey: GroupByKey not followed by a
combiner.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:50.459Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step
Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey: GroupByKey not
followed by a combiner.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:50.512Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations
into optimizable parts.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:50.551Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner
information.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:50.603Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read,
Write, and Flatten operations
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:50.637Z: JOB_MESSAGE_DEBUG: Inserted coder converter before
flatten ref_AppliedPTransform_check_even-Group-CoGroupByKeyImpl-Flatten_27
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:50.669Z: JOB_MESSAGE_DEBUG: Inserted coder converter before
flatten ref_AppliedPTransform_check_even-Group-CoGroupByKeyImpl-Flatten_27
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:50.712Z: JOB_MESSAGE_DEBUG: Inserted coder converter before
flatten ref_AppliedPTransform_check_odd-Group-CoGroupByKeyImpl-Flatten_45
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:50.750Z: JOB_MESSAGE_DEBUG: Inserted coder converter before
flatten ref_AppliedPTransform_check_odd-Group-CoGroupByKeyImpl-Flatten_45
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:50.786Z: JOB_MESSAGE_DETAILED: Unzipping flatten
ref_AppliedPTransform_check_even-Group-CoGroupByKeyImpl-Flatten_27 for input
ref_AppliedPTransform_check_even-Group-CoGroupByKeyImpl-Tag-0-_25.None-post19
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:50.827Z: JOB_MESSAGE_DETAILED: Fusing unzipped copy of
check_even/Group/CoGroupByKeyImpl/GroupByKey/Write, through flatten
check_even/Group/CoGroupByKeyImpl/Flatten, into producer
check_even/Group/CoGroupByKeyImpl/Flatten/InputIdentity
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:50.865Z: JOB_MESSAGE_DETAILED: Fusing consumer
check_even/Group/CoGroupByKeyImpl/MapTuple(collect_values) into
check_even/Group/CoGroupByKeyImpl/GroupByKey/Read
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:50.895Z: JOB_MESSAGE_DETAILED: Fusing consumer
check_even/Group/RestoreTags into
check_even/Group/CoGroupByKeyImpl/MapTuple(collect_values)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:50.928Z: JOB_MESSAGE_DETAILED: Fusing consumer
check_even/Unkey into check_even/Group/RestoreTags
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:50.960Z: JOB_MESSAGE_DETAILED: Fusing consumer
check_even/Match into check_even/Unkey
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:50.998Z: JOB_MESSAGE_DETAILED: Unzipping flatten
ref_AppliedPTransform_check_odd-Group-CoGroupByKeyImpl-Flatten_45 for input
ref_AppliedPTransform_check_odd-Group-CoGroupByKeyImpl-Tag-0-_43.None-post25
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:51.054Z: JOB_MESSAGE_DETAILED: Fusing unzipped copy of
check_odd/Group/CoGroupByKeyImpl/GroupByKey/Write, through flatten
check_odd/Group/CoGroupByKeyImpl/Flatten, into producer
check_odd/Group/CoGroupByKeyImpl/Flatten/InputIdentity
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:51.084Z: JOB_MESSAGE_DETAILED: Fusing consumer
check_odd/Group/CoGroupByKeyImpl/MapTuple(collect_values) into
check_odd/Group/CoGroupByKeyImpl/GroupByKey/Read
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:51.123Z: JOB_MESSAGE_DETAILED: Fusing consumer
check_odd/Group/RestoreTags into
check_odd/Group/CoGroupByKeyImpl/MapTuple(collect_values)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:51.233Z: JOB_MESSAGE_DETAILED: Fusing consumer check_odd/Unkey
into check_odd/Group/RestoreTags
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:51.277Z: JOB_MESSAGE_DETAILED: Fusing consumer check_odd/Match
into check_odd/Unkey
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:51.309Z: JOB_MESSAGE_DETAILED: Fusing consumer
check_even/Group/CoGroupByKeyImpl/GroupByKey/Write into
check_even/Group/CoGroupByKeyImpl/Flatten/InputIdentity
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:51.345Z: JOB_MESSAGE_DETAILED: Fusing consumer
check_odd/Group/CoGroupByKeyImpl/GroupByKey/Write into
check_odd/Group/CoGroupByKeyImpl/Flatten/InputIdentity
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:51.381Z: JOB_MESSAGE_DETAILED: Fusing consumer
Create/FlatMap(<lambda at core.py:3222>) into Create/Impulse
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:51.417Z: JOB_MESSAGE_DETAILED: Fusing consumer
Create/MaybeReshuffle/Reshuffle/AddRandomKeys into Create/FlatMap(<lambda at
core.py:3222>)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:51.453Z: JOB_MESSAGE_DETAILED: Fusing consumer
Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/Map(reify_timestamps) into
Create/MaybeReshuffle/Reshuffle/AddRandomKeys
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:51.484Z: JOB_MESSAGE_DETAILED: Fusing consumer
Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Reify into
Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/Map(reify_timestamps)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:51.518Z: JOB_MESSAGE_DETAILED: Fusing consumer
Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Write into
Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Reify
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:51.552Z: JOB_MESSAGE_DETAILED: Fusing consumer
Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/GroupByWindow into
Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Read
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:51.578Z: JOB_MESSAGE_DETAILED: Fusing consumer
Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps)
into Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/GroupByWindow
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:51.627Z: JOB_MESSAGE_DETAILED: Fusing consumer
Create/MaybeReshuffle/Reshuffle/RemoveRandomKeys into
Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:51.658Z: JOB_MESSAGE_DETAILED: Fusing consumer
Create/Map(decode) into Create/MaybeReshuffle/Reshuffle/RemoveRandomKeys
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:51.691Z: JOB_MESSAGE_DETAILED: Fusing consumer
ExternalTransform(beam:transforms:xlang:test:partition)/ParMultiDo(Partition)
into Create/Map(decode)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:51.725Z: JOB_MESSAGE_DETAILED: Fusing consumer
check_even/WindowInto(WindowIntoFn) into
ExternalTransform(beam:transforms:xlang:test:partition)/ParMultiDo(Partition)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:51.767Z: JOB_MESSAGE_DETAILED: Fusing consumer
check_odd/WindowInto(WindowIntoFn) into
ExternalTransform(beam:transforms:xlang:test:partition)/ParMultiDo(Partition)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:51.789Z: JOB_MESSAGE_DETAILED: Fusing consumer
check_even/Create/FlatMap(<lambda at core.py:3222>) into
check_even/Create/Impulse
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:51.831Z: JOB_MESSAGE_DETAILED: Fusing consumer
check_even/Create/Map(decode) into check_even/Create/FlatMap(<lambda at
core.py:3222>)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:51.861Z: JOB_MESSAGE_DETAILED: Fusing consumer
check_even/Group/CoGroupByKeyImpl/Tag[0] into check_even/Create/Map(decode)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:51.895Z: JOB_MESSAGE_DETAILED: Fusing consumer
check_even/Group/CoGroupByKeyImpl/Flatten/InputIdentity into
check_even/Group/CoGroupByKeyImpl/Tag[0]
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:51.939Z: JOB_MESSAGE_DETAILED: Fusing consumer
check_even/ToVoidKey into check_even/WindowInto(WindowIntoFn)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:51.979Z: JOB_MESSAGE_DETAILED: Fusing consumer
check_even/Group/CoGroupByKeyImpl/Tag[1] into check_even/ToVoidKey
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:52.019Z: JOB_MESSAGE_DETAILED: Fusing consumer
check_even/Group/CoGroupByKeyImpl/Flatten/InputIdentity into
check_even/Group/CoGroupByKeyImpl/Tag[1]
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:52.055Z: JOB_MESSAGE_DETAILED: Fusing consumer
check_odd/Create/FlatMap(<lambda at core.py:3222>) into check_odd/Create/Impulse
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:52.100Z: JOB_MESSAGE_DETAILED: Fusing consumer
check_odd/Create/Map(decode) into check_odd/Create/FlatMap(<lambda at
core.py:3222>)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:52.136Z: JOB_MESSAGE_DETAILED: Fusing consumer
check_odd/Group/CoGroupByKeyImpl/Tag[0] into check_odd/Create/Map(decode)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:52.172Z: JOB_MESSAGE_DETAILED: Fusing consumer
check_odd/Group/CoGroupByKeyImpl/Flatten/InputIdentity into
check_odd/Group/CoGroupByKeyImpl/Tag[0]
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:52.227Z: JOB_MESSAGE_DETAILED: Fusing consumer
check_odd/ToVoidKey into check_odd/WindowInto(WindowIntoFn)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:52.276Z: JOB_MESSAGE_DETAILED: Fusing consumer
check_odd/Group/CoGroupByKeyImpl/Tag[1] into check_odd/ToVoidKey
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:52.312Z: JOB_MESSAGE_DETAILED: Fusing consumer
check_odd/Group/CoGroupByKeyImpl/Flatten/InputIdentity into
check_odd/Group/CoGroupByKeyImpl/Tag[1]
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:52.349Z: JOB_MESSAGE_DEBUG: Workflow config is missing a
default resource spec.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:52.380Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and
teardown to workflow graph.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:52.412Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop
steps.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:52.458Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:52.633Z: JOB_MESSAGE_DEBUG: Executing wait step start39
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:52.698Z: JOB_MESSAGE_BASIC: Executing operation
Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Create
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:52.738Z: JOB_MESSAGE_BASIC: Executing operation
check_even/Group/CoGroupByKeyImpl/GroupByKey/Create
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:52.743Z: JOB_MESSAGE_DEBUG: Starting worker pool setup.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:52.773Z: JOB_MESSAGE_BASIC: Starting 1 workers in
us-central1-b...
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:52.774Z: JOB_MESSAGE_BASIC: Executing operation
check_odd/Group/CoGroupByKeyImpl/GroupByKey/Create
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:52.819Z: JOB_MESSAGE_BASIC: Finished operation
Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Create
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:52.837Z: JOB_MESSAGE_BASIC: Finished operation
check_even/Group/CoGroupByKeyImpl/GroupByKey/Create
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:52.837Z: JOB_MESSAGE_BASIC: Finished operation
check_odd/Group/CoGroupByKeyImpl/GroupByKey/Create
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:52.900Z: JOB_MESSAGE_DEBUG: Value
"Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Session"
materialized.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:52.923Z: JOB_MESSAGE_DEBUG: Value
"check_odd/Group/CoGroupByKeyImpl/GroupByKey/Session" materialized.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:52.950Z: JOB_MESSAGE_DEBUG: Value
"check_even/Group/CoGroupByKeyImpl/GroupByKey/Session" materialized.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:52.993Z: JOB_MESSAGE_BASIC: Executing operation
Create/Impulse+Create/FlatMap(<lambda at
core.py:3222>)+Create/MaybeReshuffle/Reshuffle/AddRandomKeys+Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/Map(reify_timestamps)+Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Reify+Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Write
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:53.033Z: JOB_MESSAGE_BASIC: Executing operation
check_odd/Create/Impulse+check_odd/Create/FlatMap(<lambda at
core.py:3222>)+check_odd/Create/Map(decode)+check_odd/Group/CoGroupByKeyImpl/Tag[0]+check_odd/Group/CoGroupByKeyImpl/Flatten/InputIdentity+check_odd/Group/CoGroupByKeyImpl/GroupByKey/Write
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:27:53.062Z: JOB_MESSAGE_BASIC: Executing operation
check_even/Create/Impulse+check_even/Create/FlatMap(<lambda at
core.py:3222>)+check_even/Create/Map(decode)+check_even/Group/CoGroupByKeyImpl/Tag[0]+check_even/Group/CoGroupByKeyImpl/Flatten/InputIdentity+check_even/Group/CoGroupByKeyImpl/GroupByKey/Write
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:28:08.737Z: JOB_MESSAGE_BASIC: Your project already contains 100
Dataflow-created metric descriptors, so new user metrics of the form
custom.googleapis.com/* will not be created. However, all user metrics are also
available in the metric dataflow.googleapis.com/job/user_counter. If you rely
on the custom metrics, you can delete old / unused metric descriptors. See
https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list
and
https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:28:33.037Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number
of workers to 1 based on the rate of progress in the currently running stage(s).
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:29:15.267Z: JOB_MESSAGE_WARNING: A worker was unable to start up.
Error: Unable to pull container image due to error: image pull request failed
with error: Error response from daemon: manifest for
us.gcr.io/apache-beam-testing/java-postcommit-it/java:20211109060114 not found:
manifest unknown: Failed to fetch "20211109060114" from request
"/v2/apache-beam-testing/java-postcommit-it/java/manifests/20211109060114"..
This is likely due to an invalid SDK container image URL. Please verify any
provided SDK container image is valid and that Dataflow workers have
permissions to pull image.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:29:40.996Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number
of workers to 1 based on the rate of progress in the currently running stage(s).
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:30:26.384Z: JOB_MESSAGE_WARNING: A worker was unable to start up.
Error: Unable to pull container image due to error: image pull request failed
with error: Error response from daemon: manifest for
us.gcr.io/apache-beam-testing/java-postcommit-it/java:20211109060114 not found:
manifest unknown: Failed to fetch "20211109060114" from request
"/v2/apache-beam-testing/java-postcommit-it/java/manifests/20211109060114"..
This is likely due to an invalid SDK container image URL. Please verify any
provided SDK container image is valid and that Dataflow workers have
permissions to pull image.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:30:54.539Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number
of workers to 1 based on the rate of progress in the currently running stage(s).
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:31:37.751Z: JOB_MESSAGE_WARNING: A worker was unable to start up.
Error: Unable to pull container image due to error: image pull request failed
with error: Error response from daemon: manifest for
us.gcr.io/apache-beam-testing/java-postcommit-it/java:20211109060114 not found:
manifest unknown: Failed to fetch "20211109060114" from request
"/v2/apache-beam-testing/java-postcommit-it/java/manifests/20211109060114"..
This is likely due to an invalid SDK container image URL. Please verify any
provided SDK container image is valid and that Dataflow workers have
permissions to pull image.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:32:02.052Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number
of workers to 1 based on the rate of progress in the currently running stage(s).
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:32:43.978Z: JOB_MESSAGE_WARNING: A worker was unable to start up.
Error: Unable to pull container image due to error: image pull request failed
with error: Error response from daemon: manifest for
us.gcr.io/apache-beam-testing/java-postcommit-it/java:20211109060114 not found:
manifest unknown: Failed to fetch "20211109060114" from request
"/v2/apache-beam-testing/java-postcommit-it/java/manifests/20211109060114"..
This is likely due to an invalid SDK container image URL. Please verify any
provided SDK container image is valid and that Dataflow workers have
permissions to pull image.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:33:10.250Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number
of workers to 1 based on the rate of progress in the currently running stage(s).
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:33:54.372Z: JOB_MESSAGE_WARNING: A worker was unable to start up.
Error: Unable to pull container image due to error: image pull request failed
with error: Error response from daemon: manifest for
us.gcr.io/apache-beam-testing/java-postcommit-it/java:20211109060114 not found:
manifest unknown: Failed to fetch "20211109060114" from request
"/v2/apache-beam-testing/java-postcommit-it/java/manifests/20211109060114"..
This is likely due to an invalid SDK container image URL. Please verify any
provided SDK container image is valid and that Dataflow workers have
permissions to pull image.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:33:54.419Z: JOB_MESSAGE_ERROR: Workflow failed. Causes: Job
appears to be stuck. Several workers have failed to start up in a row, and no
worker has successfully started up for this job. Last error reported: Unable to
pull container image due to error: image pull request failed with error: Error
response from daemon: manifest for
us.gcr.io/apache-beam-testing/java-postcommit-it/java:20211109060114 not found:
manifest unknown: Failed to fetch "20211109060114" from request
"/v2/apache-beam-testing/java-postcommit-it/java/manifests/20211109060114"..
This is likely due to an invalid SDK container image URL. Please verify any
provided SDK container image is valid and that Dataflow workers have
permissions to pull image..
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:33:54.503Z: JOB_MESSAGE_BASIC: Finished operation
Create/Impulse+Create/FlatMap(<lambda at
core.py:3222>)+Create/MaybeReshuffle/Reshuffle/AddRandomKeys+Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/Map(reify_timestamps)+Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Reify+Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Write
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:33:54.503Z: JOB_MESSAGE_BASIC: Finished operation
check_odd/Create/Impulse+check_odd/Create/FlatMap(<lambda at
core.py:3222>)+check_odd/Create/Map(decode)+check_odd/Group/CoGroupByKeyImpl/Tag[0]+check_odd/Group/CoGroupByKeyImpl/Flatten/InputIdentity+check_odd/Group/CoGroupByKeyImpl/GroupByKey/Write
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:33:54.503Z: JOB_MESSAGE_BASIC: Finished operation
check_even/Create/Impulse+check_even/Create/FlatMap(<lambda at
core.py:3222>)+check_even/Create/Map(decode)+check_even/Group/CoGroupByKeyImpl/Tag[0]+check_even/Group/CoGroupByKeyImpl/Flatten/InputIdentity+check_even/Group/CoGroupByKeyImpl/GroupByKey/Write
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:33:54.618Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:33:54.726Z: JOB_MESSAGE_DEBUG: Starting worker pool teardown.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:33:54.775Z: JOB_MESSAGE_BASIC: Stopping worker pool...
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:33:55.068Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized worker
pool from 1 to 0.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:34:41.835Z: JOB_MESSAGE_BASIC: Worker pool stopped.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-11-09T06:34:41.908Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:191
Job 2021-11-08_22_27_39-3268535017721424543 is in state JOB_STATE_FAILED
=============================== warnings summary ===============================
apache_beam/io/filesystems_test.py:54
apache_beam/io/filesystems_test.py:54
apache_beam/io/filesystems_test.py:54
<https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/ws/src/sdks/python/apache_beam/io/filesystems_test.py>:54:
DeprecationWarning: invalid escape sequence \c
self.assertIsNone(FileSystems.get_scheme('c:\\abc\cdf')) # pylint:
disable=anomalous-backslash-in-string
apache_beam/io/filesystems_test.py:62
apache_beam/io/filesystems_test.py:62
apache_beam/io/filesystems_test.py:62
<https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/ws/src/sdks/python/apache_beam/io/filesystems_test.py>:62:
DeprecationWarning: invalid escape sequence \d
self.assertTrue(isinstance(FileSystems.get_filesystem('c:\\abc\def'), #
pylint: disable=anomalous-backslash-in-string
-- Docs: https://docs.pytest.org/en/latest/warnings.html
- generated xml file:
<https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/ws/src/sdks/python/pytest_xlangValidateRunner.xml>
-
============== 9 failed, 1 passed, 6 warnings in 1009.00 seconds ===============
> Task
> :runners:google-cloud-dataflow-java:validatesCrossLanguageRunnerPythonUsingJava
> FAILED
> Task :runners:google-cloud-dataflow-java:validatesCrossLanguageRunnerCleanup
Stopping expansion service pid: 532.
Stopping expansion service pid: 536.
> Task :runners:google-cloud-dataflow-java:cleanUpDockerImages FAILED
Untagged: us.gcr.io/apache-beam-testing/java-postcommit-it/java:20211109060114
Untagged:
us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:38c40d4bf1bfc1042077fc3bc7fbe4b89094826879139aac3ce17d5b1ed7167a
ERROR: (gcloud.container.images.untag) Image could not be found:
[us.gcr.io/apache-beam-testing/java-postcommit-it/java:20211109060114]
FAILURE: Build completed with 2 failures.
1: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task
':runners:google-cloud-dataflow-java:validatesCrossLanguageRunnerPythonUsingJava'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug
option to get more log output. Run with --scan to get full insights.
==============================================================================
2: Task failed with an exception.
-----------
* Where:
Build file
'<https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/ws/src/runners/google-cloud-dataflow-java/build.gradle'>
line: 278
* What went wrong:
Execution failed for task
':runners:google-cloud-dataflow-java:cleanUpDockerImages'.
> Process 'command 'gcloud'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug
option to get more log output. Run with --scan to get full insights.
==============================================================================
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with
Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See
https://docs.gradle.org/6.9.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 34m 53s
132 actionable tasks: 94 executed, 34 from cache, 4 up-to-date
Publishing build scan...
https://gradle.com/s/v3j2pqdhb3jjq
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
