See
<https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/1366/display/redirect?page=changes>
Changes:
[heejong] [BEAM-12978] Customizable dependency for Java external transform
[heejong] add ConfigT param to getDependencies method
------------------------------------------
[...truncated 716.99 KB...]
thread = threading.Thread(
target=DataflowRunner.poll_for_job_completion,
args=(self._runner, self, duration))
# Mark the thread as a daemon thread so a keyboard interrupt on the main
# thread will terminate everything. This is also the reason we will not
# use thread.join() to wait for the polling thread.
thread.daemon = True
thread.start()
while thread.is_alive():
time.sleep(5.0)
# TODO: Merge the termination code in poll_for_job_completion and
# is_in_terminal_state.
terminated = self.is_in_terminal_state()
assert duration or terminated, (
'Job did not reach to a terminal state after waiting indefinitely.')
if terminated and self.state != PipelineState.DONE:
# TODO(BEAM-1290): Consider converting this to an error log based on
# theresolution of the issue.
raise DataflowRuntimeException(
'Dataflow pipeline failed. State: %s, Error:\n%s' %
(self.state, getattr(self._runner, 'last_error_msg', None)),
> self)
E
apache_beam.runners.dataflow.dataflow_runner.DataflowRuntimeException: Dataflow
pipeline failed. State: FAILED, Error:
E Workflow failed. Causes: Job appears to be stuck. Several workers
have failed to start up in a row, and no worker has successfully started up for
this job. Last error reported: Unable to pull container image due to error:
image pull request failed with error: Error response from daemon: manifest for
us.gcr.io/apache-beam-testing/java-postcommit-it/java:20211027060111 not found:
manifest unknown: Failed to fetch "20211027060111" from request
"/v2/apache-beam-testing/java-postcommit-it/java/manifests/20211027060111"..
This is likely due to an invalid SDK container image URL. Please verify any
provided SDK container image is valid and that Dataflow workers have
permissions to pull image..
apache_beam/runners/dataflow/dataflow_runner.py:1643: DataflowRuntimeException
------------------------------ Captured log call -------------------------------
INFO apache_beam.runners.portability.stager:stager.py:303 Copying Beam SDK
"<https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/ws/src/sdks/python/build/apache-beam.tar.gz";>
to staging location.
WARNING root:environments.py:374 Make sure that locally built Python SDK
docker image has Python 3.6 interpreter.
INFO root:environments.py:380 Default Python SDK image for environment is
apache/beam_python3.6_sdk:2.35.0.dev
INFO root:environments.py:296 Using provided Python SDK container image:
gcr.io/cloud-dataflow/v1beta3/python36-fnapi:beam-master-20211015
INFO root:environments.py:304 Python SDK container image set to
"gcr.io/cloud-dataflow/v1beta3/python36-fnapi:beam-master-20211015" for Docker
environment
INFO
apache_beam.runners.portability.fn_api_runner.translations:translations.py:678
==================== <function pack_combiners at 0x7fd5a426c6a8>
====================
INFO
apache_beam.runners.portability.fn_api_runner.translations:translations.py:678
==================== <function sort_stages at 0x7fd5a426ce18>
====================
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:454
Defaulting to the temp_location as staging_location:
gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:638
Starting GCS upload to
gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-1027063230-770302.1635316350.770851/icedtea-sound-Bdoi2wYa757-fzq5vconCy4SSQ22ZaOq7yuC98fKPs8.jar...
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:657
Completed GCS upload to
gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-1027063230-770302.1635316350.770851/icedtea-sound-Bdoi2wYa757-fzq5vconCy4SSQ22ZaOq7yuC98fKPs8.jar
in 0 seconds.
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:638
Starting GCS upload to
gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-1027063230-770302.1635316350.770851/jaccess-CMbK-IOdQPLKHEqCuDnE-yBk-VpbtVT-hgjbHRUGO78.jar...
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:657
Completed GCS upload to
gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-1027063230-770302.1635316350.770851/jaccess-CMbK-IOdQPLKHEqCuDnE-yBk-VpbtVT-hgjbHRUGO78.jar
in 0 seconds.
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:638
Starting GCS upload to
gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-1027063230-770302.1635316350.770851/localedata-ae5Z0L6ak4922fztWeWy7ajiWXdG3ubNrwerJRFoFj0.jar...
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:657
Completed GCS upload to
gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-1027063230-770302.1635316350.770851/localedata-ae5Z0L6ak4922fztWeWy7ajiWXdG3ubNrwerJRFoFj0.jar
in 0 seconds.
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:638
Starting GCS upload to
gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-1027063230-770302.1635316350.770851/nashorn-XHtz_UehGpYcLTOrATrTnMNVUgEVa_ttoWkPxnVfqTo.jar...
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:657
Completed GCS upload to
gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-1027063230-770302.1635316350.770851/nashorn-XHtz_UehGpYcLTOrATrTnMNVUgEVa_ttoWkPxnVfqTo.jar
in 0 seconds.
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:638
Starting GCS upload to
gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-1027063230-770302.1635316350.770851/cldrdata-k07I6K9W3X5KTQbcDIEsqM0LXyM18f0eR6IaJw-P_xk.jar...
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:657
Completed GCS upload to
gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-1027063230-770302.1635316350.770851/cldrdata-k07I6K9W3X5KTQbcDIEsqM0LXyM18f0eR6IaJw-P_xk.jar
in 0 seconds.
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:638
Starting GCS upload to
gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-1027063230-770302.1635316350.770851/dnsns-RGhCDg3GVOQVC2r6ka2N0hmI4eqQH6VobuoAnQ74MnE.jar...
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:657
Completed GCS upload to
gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-1027063230-770302.1635316350.770851/dnsns-RGhCDg3GVOQVC2r6ka2N0hmI4eqQH6VobuoAnQ74MnE.jar
in 0 seconds.
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:638
Starting GCS upload to
gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-1027063230-770302.1635316350.770851/beam-sdks-java-testing-expansion-service-testExpansionService-2.35.0-SNAPSHOT-O43zcCUfdR3oTjmYYqWM94AknVWsxLmVJ8HCKDFiMlY.jar...
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:657
Completed GCS upload to
gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-1027063230-770302.1635316350.770851/beam-sdks-java-testing-expansion-service-testExpansionService-2.35.0-SNAPSHOT-O43zcCUfdR3oTjmYYqWM94AknVWsxLmVJ8HCKDFiMlY.jar
in 3 seconds.
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:638
Starting GCS upload to
gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-1027063230-770302.1635316350.770851/dataflow_python_sdk.tar...
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:657
Completed GCS upload to
gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-1027063230-770302.1635316350.770851/dataflow_python_sdk.tar
in 0 seconds.
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:638
Starting GCS upload to
gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-1027063230-770302.1635316350.770851/pipeline.pb...
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:657
Completed GCS upload to
gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-1027063230-770302.1635316350.770851/pipeline.pb
in 0 seconds.
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:818
Create job: <Job
clientRequestId: '20211027063230771790-3474'
createTime: '2021-10-27T06:32:36.246307Z'
currentStateTime: '1970-01-01T00:00:00Z'
id:
'2021-10-26_23_32_35-17138742594254128411'
location: 'us-central1'
name: 'beamapp-jenkins-1027063230-770302'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2021-10-27T06:32:36.246307Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:820
Created job with id: [2021-10-26_23_32_35-17138742594254128411]
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:821
Submitted job: 2021-10-26_23_32_35-17138742594254128411
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:827 To
access the Dataflow monitoring console, please navigate to
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-10-26_23_32_35-17138742594254128411?project=apache-beam-testing
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:191
Job 2021-10-26_23_32_35-17138742594254128411 is in state JOB_STATE_RUNNING
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:32:38.902Z: JOB_MESSAGE_DETAILED: Autoscaling is enabled for job
2021-10-26_23_32_35-17138742594254128411. The number of workers will be between
1 and 1000.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:32:39.099Z: JOB_MESSAGE_DETAILED: Autoscaling was automatically
enabled for job 2021-10-26_23_32_35-17138742594254128411.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:32:40.875Z: JOB_MESSAGE_BASIC: Worker configuration:
e2-standard-2 in us-central1-b.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:32:41.634Z: JOB_MESSAGE_DETAILED: Expanding SplittableParDo
operations into optimizable parts.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:32:41.665Z: JOB_MESSAGE_DETAILED: Expanding CollectionToSingleton
operations into optimizable parts.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:32:41.730Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey
operations into optimizable parts.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:32:41.767Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step
assert_that/Group/CoGroupByKeyImpl/GroupByKey: GroupByKey not followed by a
combiner.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:32:41.797Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step
Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey: GroupByKey not
followed by a combiner.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:32:41.833Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations
into optimizable parts.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:32:41.863Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner
information.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:32:41.902Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read,
Write, and Flatten operations
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:32:41.928Z: JOB_MESSAGE_DEBUG: Inserted coder converter before
flatten ref_AppliedPTransform_assert_that-Group-CoGroupByKeyImpl-Flatten_27
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:32:41.962Z: JOB_MESSAGE_DEBUG: Inserted coder converter before
flatten ref_AppliedPTransform_assert_that-Group-CoGroupByKeyImpl-Flatten_27
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:32:41.995Z: JOB_MESSAGE_DETAILED: Unzipping flatten
ref_AppliedPTransform_assert_that-Group-CoGroupByKeyImpl-Flatten_27 for input
ref_AppliedPTransform_assert_that-Group-CoGroupByKeyImpl-Tag-0-_25.None-post16
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:32:42.028Z: JOB_MESSAGE_DETAILED: Fusing unzipped copy of
assert_that/Group/CoGroupByKeyImpl/GroupByKey/Write, through flatten
assert_that/Group/CoGroupByKeyImpl/Flatten, into producer
assert_that/Group/CoGroupByKeyImpl/Flatten/InputIdentity
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:32:42.064Z: JOB_MESSAGE_DETAILED: Fusing consumer
assert_that/Group/CoGroupByKeyImpl/MapTuple(collect_values) into
assert_that/Group/CoGroupByKeyImpl/GroupByKey/Read
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:32:42.086Z: JOB_MESSAGE_DETAILED: Fusing consumer
assert_that/Group/RestoreTags into
assert_that/Group/CoGroupByKeyImpl/MapTuple(collect_values)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:32:42.119Z: JOB_MESSAGE_DETAILED: Fusing consumer
assert_that/Unkey into assert_that/Group/RestoreTags
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:32:42.151Z: JOB_MESSAGE_DETAILED: Fusing consumer
assert_that/Match into assert_that/Unkey
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:32:42.184Z: JOB_MESSAGE_DETAILED: Fusing consumer
assert_that/Group/CoGroupByKeyImpl/GroupByKey/Write into
assert_that/Group/CoGroupByKeyImpl/Flatten/InputIdentity
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:32:42.218Z: JOB_MESSAGE_DETAILED: Fusing consumer
Create/FlatMap(<lambda at core.py:3222>) into Create/Impulse
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:32:42.273Z: JOB_MESSAGE_DETAILED: Fusing consumer
Create/MaybeReshuffle/Reshuffle/AddRandomKeys into Create/FlatMap(<lambda at
core.py:3222>)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:32:42.307Z: JOB_MESSAGE_DETAILED: Fusing consumer
Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/Map(reify_timestamps) into
Create/MaybeReshuffle/Reshuffle/AddRandomKeys
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:32:42.345Z: JOB_MESSAGE_DETAILED: Fusing consumer
Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Reify into
Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/Map(reify_timestamps)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:32:42.369Z: JOB_MESSAGE_DETAILED: Fusing consumer
Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Write into
Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Reify
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:32:42.396Z: JOB_MESSAGE_DETAILED: Fusing consumer
Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/GroupByWindow into
Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Read
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:32:42.431Z: JOB_MESSAGE_DETAILED: Fusing consumer
Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps)
into Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/GroupByWindow
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:32:42.464Z: JOB_MESSAGE_DETAILED: Fusing consumer
Create/MaybeReshuffle/Reshuffle/RemoveRandomKeys into
Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:32:42.500Z: JOB_MESSAGE_DETAILED: Fusing consumer
Create/Map(decode) into Create/MaybeReshuffle/Reshuffle/RemoveRandomKeys
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:32:42.531Z: JOB_MESSAGE_DETAILED: Fusing consumer
ExternalTransform(beam:transforms:xlang:test:prefix)/Map/ParMultiDo(Anonymous)
into Create/Map(decode)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:32:42.568Z: JOB_MESSAGE_DETAILED: Fusing consumer
assert_that/WindowInto(WindowIntoFn) into
ExternalTransform(beam:transforms:xlang:test:prefix)/Map/ParMultiDo(Anonymous)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:32:42.601Z: JOB_MESSAGE_DETAILED: Fusing consumer
assert_that/Create/FlatMap(<lambda at core.py:3222>) into
assert_that/Create/Impulse
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:32:42.634Z: JOB_MESSAGE_DETAILED: Fusing consumer
assert_that/Create/Map(decode) into assert_that/Create/FlatMap(<lambda at
core.py:3222>)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:32:42.665Z: JOB_MESSAGE_DETAILED: Fusing consumer
assert_that/Group/CoGroupByKeyImpl/Tag[0] into assert_that/Create/Map(decode)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:32:42.691Z: JOB_MESSAGE_DETAILED: Fusing consumer
assert_that/Group/CoGroupByKeyImpl/Flatten/InputIdentity into
assert_that/Group/CoGroupByKeyImpl/Tag[0]
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:32:42.711Z: JOB_MESSAGE_DETAILED: Fusing consumer
assert_that/ToVoidKey into assert_that/WindowInto(WindowIntoFn)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:32:42.744Z: JOB_MESSAGE_DETAILED: Fusing consumer
assert_that/Group/CoGroupByKeyImpl/Tag[1] into assert_that/ToVoidKey
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:32:42.768Z: JOB_MESSAGE_DETAILED: Fusing consumer
assert_that/Group/CoGroupByKeyImpl/Flatten/InputIdentity into
assert_that/Group/CoGroupByKeyImpl/Tag[1]
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:32:42.805Z: JOB_MESSAGE_DEBUG: Workflow config is missing a
default resource spec.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:32:42.823Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and
teardown to workflow graph.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:32:42.847Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop
steps.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:32:42.868Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:32:43.012Z: JOB_MESSAGE_DEBUG: Executing wait step start26
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:32:43.082Z: JOB_MESSAGE_BASIC: Executing operation
Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Create
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:32:43.117Z: JOB_MESSAGE_BASIC: Executing operation
assert_that/Group/CoGroupByKeyImpl/GroupByKey/Create
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:32:43.130Z: JOB_MESSAGE_DEBUG: Starting worker pool setup.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:32:43.163Z: JOB_MESSAGE_BASIC: Starting 1 workers in
us-central1-b...
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:32:43.216Z: JOB_MESSAGE_BASIC: Finished operation
Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Create
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:32:43.229Z: JOB_MESSAGE_BASIC: Finished operation
assert_that/Group/CoGroupByKeyImpl/GroupByKey/Create
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:32:43.297Z: JOB_MESSAGE_DEBUG: Value
"Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Session"
materialized.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:32:43.332Z: JOB_MESSAGE_DEBUG: Value
"assert_that/Group/CoGroupByKeyImpl/GroupByKey/Session" materialized.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:32:43.364Z: JOB_MESSAGE_BASIC: Executing operation
Create/Impulse+Create/FlatMap(<lambda at
core.py:3222>)+Create/MaybeReshuffle/Reshuffle/AddRandomKeys+Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/Map(reify_timestamps)+Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Reify+Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Write
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:32:43.401Z: JOB_MESSAGE_BASIC: Executing operation
assert_that/Create/Impulse+assert_that/Create/FlatMap(<lambda at
core.py:3222>)+assert_that/Create/Map(decode)+assert_that/Group/CoGroupByKeyImpl/Tag[0]+assert_that/Group/CoGroupByKeyImpl/Flatten/InputIdentity+assert_that/Group/CoGroupByKeyImpl/GroupByKey/Write
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:32:52.560Z: JOB_MESSAGE_BASIC: Your project already contains 100
Dataflow-created metric descriptors, so new user metrics of the form
custom.googleapis.com/* will not be created. However, all user metrics are also
available in the metric dataflow.googleapis.com/job/user_counter. If you rely
on the custom metrics, you can delete old / unused metric descriptors. See
https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list
and
https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:33:23.730Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number
of workers to 1 based on the rate of progress in the currently running stage(s).
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:34:07.994Z: JOB_MESSAGE_WARNING: A worker was unable to start up.
Error: Unable to pull container image due to error: image pull request failed
with error: Error response from daemon: manifest for
us.gcr.io/apache-beam-testing/java-postcommit-it/java:20211027060111 not found:
manifest unknown: Failed to fetch "20211027060111" from request
"/v2/apache-beam-testing/java-postcommit-it/java/manifests/20211027060111"..
This is likely due to an invalid SDK container image URL. Please verify any
provided SDK container image is valid and that Dataflow workers have
permissions to pull image.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:34:33.673Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number
of workers to 1 based on the rate of progress in the currently running stage(s).
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:35:15.756Z: JOB_MESSAGE_WARNING: A worker was unable to start up.
Error: Unable to pull container image due to error: image pull request failed
with error: Error response from daemon: manifest for
us.gcr.io/apache-beam-testing/java-postcommit-it/java:20211027060111 not found:
manifest unknown: Failed to fetch "20211027060111" from request
"/v2/apache-beam-testing/java-postcommit-it/java/manifests/20211027060111"..
This is likely due to an invalid SDK container image URL. Please verify any
provided SDK container image is valid and that Dataflow workers have
permissions to pull image.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:35:42.108Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number
of workers to 1 based on the rate of progress in the currently running stage(s).
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:36:26.879Z: JOB_MESSAGE_WARNING: A worker was unable to start up.
Error: Unable to pull container image due to error: image pull request failed
with error: Error response from daemon: manifest for
us.gcr.io/apache-beam-testing/java-postcommit-it/java:20211027060111 not found:
manifest unknown: Failed to fetch "20211027060111" from request
"/v2/apache-beam-testing/java-postcommit-it/java/manifests/20211027060111"..
This is likely due to an invalid SDK container image URL. Please verify any
provided SDK container image is valid and that Dataflow workers have
permissions to pull image.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:36:50.146Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number
of workers to 1 based on the rate of progress in the currently running stage(s).
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:37:36.899Z: JOB_MESSAGE_WARNING: A worker was unable to start up.
Error: Unable to pull container image due to error: image pull request failed
with error: Error response from daemon: manifest for
us.gcr.io/apache-beam-testing/java-postcommit-it/java:20211027060111 not found:
manifest unknown: Failed to fetch "20211027060111" from request
"/v2/apache-beam-testing/java-postcommit-it/java/manifests/20211027060111"..
This is likely due to an invalid SDK container image URL. Please verify any
provided SDK container image is valid and that Dataflow workers have
permissions to pull image.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:38:03.587Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number
of workers to 1 based on the rate of progress in the currently running stage(s).
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:38:48.611Z: JOB_MESSAGE_WARNING: A worker was unable to start up.
Error: Unable to pull container image due to error: image pull request failed
with error: Error response from daemon: manifest for
us.gcr.io/apache-beam-testing/java-postcommit-it/java:20211027060111 not found:
manifest unknown: Failed to fetch "20211027060111" from request
"/v2/apache-beam-testing/java-postcommit-it/java/manifests/20211027060111"..
This is likely due to an invalid SDK container image URL. Please verify any
provided SDK container image is valid and that Dataflow workers have
permissions to pull image.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:38:48.647Z: JOB_MESSAGE_ERROR: Workflow failed. Causes: Job
appears to be stuck. Several workers have failed to start up in a row, and no
worker has successfully started up for this job. Last error reported: Unable to
pull container image due to error: image pull request failed with error: Error
response from daemon: manifest for
us.gcr.io/apache-beam-testing/java-postcommit-it/java:20211027060111 not found:
manifest unknown: Failed to fetch "20211027060111" from request
"/v2/apache-beam-testing/java-postcommit-it/java/manifests/20211027060111"..
This is likely due to an invalid SDK container image URL. Please verify any
provided SDK container image is valid and that Dataflow workers have
permissions to pull image..
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:38:48.695Z: JOB_MESSAGE_BASIC: Finished operation
assert_that/Create/Impulse+assert_that/Create/FlatMap(<lambda at
core.py:3222>)+assert_that/Create/Map(decode)+assert_that/Group/CoGroupByKeyImpl/Tag[0]+assert_that/Group/CoGroupByKeyImpl/Flatten/InputIdentity+assert_that/Group/CoGroupByKeyImpl/GroupByKey/Write
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:38:48.695Z: JOB_MESSAGE_BASIC: Finished operation
Create/Impulse+Create/FlatMap(<lambda at
core.py:3222>)+Create/MaybeReshuffle/Reshuffle/AddRandomKeys+Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/Map(reify_timestamps)+Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Reify+Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Write
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:38:48.764Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:38:48.829Z: JOB_MESSAGE_DEBUG: Starting worker pool teardown.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:38:48.847Z: JOB_MESSAGE_BASIC: Stopping worker pool...
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:38:51.047Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized worker
pool from 1 to 0.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:41:50.131Z: JOB_MESSAGE_BASIC: Worker pool stopped.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236
2021-10-27T06:41:50.168Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:191
Job 2021-10-26_23_32_35-17138742594254128411 is in state JOB_STATE_FAILED
=============================== warnings summary ===============================
apache_beam/io/filesystems_test.py:54
apache_beam/io/filesystems_test.py:54
apache_beam/io/filesystems_test.py:54
apache_beam/io/filesystems_test.py:54
apache_beam/io/filesystems_test.py:54
<https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/ws/src/sdks/python/apache_beam/io/filesystems_test.py>:54:
DeprecationWarning: invalid escape sequence \c
self.assertIsNone(FileSystems.get_scheme('c:\\abc\cdf')) # pylint:
disable=anomalous-backslash-in-string
apache_beam/io/filesystems_test.py:62
apache_beam/io/filesystems_test.py:62
apache_beam/io/filesystems_test.py:62
apache_beam/io/filesystems_test.py:62
apache_beam/io/filesystems_test.py:62
<https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/ws/src/sdks/python/apache_beam/io/filesystems_test.py>:62:
DeprecationWarning: invalid escape sequence \d
self.assertTrue(isinstance(FileSystems.get_filesystem('c:\\abc\def'), #
pylint: disable=anomalous-backslash-in-string
-- Docs: https://docs.pytest.org/en/latest/warnings.html
- generated xml file:
<https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/ws/src/sdks/python/pytest_xlangValidateRunner.xml>
-
============== 9 failed, 1 passed, 10 warnings in 1132.05 seconds ==============
> Task
> :runners:google-cloud-dataflow-java:validatesCrossLanguageRunnerPythonUsingJava
> FAILED
> Task :runners:google-cloud-dataflow-java:validatesCrossLanguageRunnerCleanup
Stopping expansion service pid: 20063.
Stopping expansion service pid: 20067.
> Task :runners:google-cloud-dataflow-java:cleanUpDockerImages
Untagged: us.gcr.io/apache-beam-testing/java-postcommit-it/java:20211027060111
Untagged:
us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:ddb8a6c8b5fb6217ec0477770f96270bd5ff604a22a0feafd492e942acc556a6
Deleted: sha256:1400ae5ed00208bee11abca50eba277df25e7502c4f0f6d66243993fb0e662ba
Deleted: sha256:d3e12eaf8fda5591173f84a26f8b1bf64ad25281e997ae224952c28f61abf5c4
Deleted: sha256:1c6944777a8394ec1022f14219aabfe8e971381aabe834ccc6208e554d09c217
Deleted: sha256:b3fece3ebd6be67a502a0e5d3a4665a24b0d107bf26fd0fa36d565c446dbd419
Deleted: sha256:c0da1ab06cb23794d29bd40c452bc71faf1b0e2ded24a38935d3062edececdb2
Deleted: sha256:919ffe08323c4ee86d9c788f1f4625fda068fd394358ba2f886849f2f23bb0ad
Deleted: sha256:04b653bb4afb04e496d672f266c07a80da80748cbbd579e2e12acad266a41893
Deleted: sha256:3abb5cfaec79897c5e0445894632e7cdd3964cf177c5994ef50190df4c7bf71e
Deleted: sha256:85cb8eaa25ee971b66ae2151f5702521fe979391e29d5b92f25e299cece40706
Deleted: sha256:44e265edc283dfbef1e99f9eb98957bca43ee29977b39e33f381996e11926a38
Deleted: sha256:7867857c2499ca3bc5ec67700c5c25d3958e19002164080fa6c319b4d2bc8e5c
Deleted: sha256:8c815ecc824b1d7a910abbc22106396bf6b4cd3574592b93b4ac96b4c0a15f5a
Deleted: sha256:6ac75c14e1abe171f6e5e054916432e307f4757e899949cae74c9b8e8cd86b93
Deleted: sha256:3cb53c9476b82e7048d4f330067874760d30cbe3227afd611f55f0161cd64d0f
Deleted: sha256:3c67db24f69a88e14a56242fc054503a42c59500cc974a91a3bf5e3aff6d218d
Deleted: sha256:f26c424cd345d6b43ec8df23fc7bfb4367837d7c9be1ef486cc024c542e6a604
Deleted: sha256:a331e3c624a85b86f55f54d99c749a3aca254e9d94b9438e395f1e86a2eee3ce
Deleted: sha256:cf92f384aff5fcd85227a08d3c6483cfc79ce399040494aa00600c340e8ebd5f
Deleted: sha256:2987b5ca1094d3a2d0e2bf56340b0fd9061b0127550c84f63c70d8b6429eb22e
Deleted: sha256:b22acc822e460bd308d6ce0f7329e50eac0a0a4e878a207b57f29543a9f143f9
Deleted: sha256:d7cc592a3cfb827d821dc06cfa4b1d8f2e7ea810bd0d6f357c54d08496decdfd
Deleted: sha256:85e90e1b2a8d6d9636379534d5d1112cfa7dc3a536e61349b7951291b3c9e11b
Deleted: sha256:045b120b9fc13fb16c1eb8980595bcf503966d88734d99e17753967e663df10b
Deleted: sha256:87afac0c8a3bac9cb5c107acd32ff5cfdc0a7de050a81be221070e6dd2b1c299
Deleted: sha256:ce19ea10602cba8e1f28807fbfb464a97d4e1b3e7b20e99e5b49e79a9a1d1dce
Deleted: sha256:aaa5e31e44ccea9e28f8b4457cb9bda7282694570fe3e73b47a9d366a7f5a1e1
Deleted: sha256:bbae8e73e142256ba954d3c358d5a5caad1478581b08d9d8c21afc172425974b
Deleted: sha256:51abac25b6f70ce3ede052a83bddce51aeab6172eb1cd520843bd979b1e0ed75
ERROR: (gcloud.container.images.untag) Image could not be found:
[us.gcr.io/apache-beam-testing/java-postcommit-it/java:20211027060111]
> Task :runners:google-cloud-dataflow-java:cleanUpDockerImages FAILED
FAILURE: Build completed with 2 failures.
1: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task
':runners:google-cloud-dataflow-java:validatesCrossLanguageRunnerPythonUsingJava'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug
option to get more log output. Run with --scan to get full insights.
==============================================================================
2: Task failed with an exception.
-----------
* Where:
Build file
'<https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/ws/src/runners/google-cloud-dataflow-java/build.gradle'>
line: 278
* What went wrong:
Execution failed for task
':runners:google-cloud-dataflow-java:cleanUpDockerImages'.
> Process 'command 'gcloud'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug
option to get more log output. Run with --scan to get full insights.
==============================================================================
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with
Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See
https://docs.gradle.org/6.9.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 42m 3s
132 actionable tasks: 112 executed, 16 from cache, 4 up-to-date
Publishing build scan...
https://gradle.com/s/xd4dzm3vpv4f6
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
