See
<https://ci-beam.apache.org/job/beam_PostCommit_Python38/1326/display/redirect>
Changes:
------------------------------------------
[...truncated 46.85 MB...]
"pipeline_proto_coder_id":
"ref_Coder_FastPrimitivesCoder_7"
},
{
"@type":
"FastPrimitivesCoder$eNprYJmaxgABPZKJBYnJGanxSamJuXrJ+SmpRcVQakqPsFticUlAUWZuZklmWWqxM0h4yuQpmo1Taqf08MenJebkJCUmZ8eD1U/JYOjhDshMzs5JRVGYVJykBwDOUCqY",
"component_encodings": [],
"pipeline_proto_coder_id":
"ref_Coder_FastPrimitivesCoder_7"
}
],
"is_pair_like": true,
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_7"
},
{
"@type": "kind:global_window"
}
],
"is_wrapper": true
},
"output_name": "None",
"user_name": "assert_that/Match.out"
}
],
"parallel_input": {
"@type": "OutputReference",
"output_name": "None",
"step_name": "s28"
},
"serialized_fn": "ref_AppliedPTransform_assert_that-Match_43",
"user_name": "assert_that/Match"
}
}
],
"type": "JOB_TYPE_BATCH"
}
apache_beam.runners.dataflow.internal.apiclient: INFO: Create job: <Job
createTime: '2021-06-16T12:38:09.194806Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2021-06-16_05_38_08-11637755170750681811'
location: 'us-central1'
name: 'beamapp-jenkins-0616123759-082055'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2021-06-16T12:38:09.194806Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
apache_beam.runners.dataflow.internal.apiclient: INFO: Created job with id:
[2021-06-16_05_38_08-11637755170750681811]
apache_beam.runners.dataflow.internal.apiclient: INFO: Submitted job:
2021-06-16_05_38_08-11637755170750681811
apache_beam.runners.dataflow.internal.apiclient: INFO: To access the Dataflow
monitoring console, please navigate to
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-16_05_38_08-11637755170750681811?project=apache-beam-testing
apache_beam.runners.dataflow.dataflow_runner: INFO: Job
2021-06-16_05_38_08-11637755170750681811 is in state JOB_STATE_RUNNING
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:38:11.799Z:
JOB_MESSAGE_DETAILED: Autoscaling is enabled for job
2021-06-16_05_38_08-11637755170750681811. The number of workers will be between
1 and 1000.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:38:12.011Z:
JOB_MESSAGE_DETAILED: Autoscaling was automatically enabled for job
2021-06-16_05_38_08-11637755170750681811.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:38:13.659Z:
JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in us-central1-f.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:38:14.264Z:
JOB_MESSAGE_DETAILED: Expanding SplittableParDo operations into optimizable
parts.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:38:14.297Z:
JOB_MESSAGE_DETAILED: Expanding CollectionToSingleton operations into
optimizable parts.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:38:14.358Z:
JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:38:14.392Z:
JOB_MESSAGE_DEBUG: Combiner lifting skipped for step
assert_that/Group/GroupByKey: GroupByKey not followed by a combiner.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:38:14.415Z:
JOB_MESSAGE_DEBUG: Combiner lifting skipped for step
Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey: GroupByKey not
followed by a combiner.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:38:14.452Z:
JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:38:14.488Z:
JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:38:14.533Z:
JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:38:14.563Z:
JOB_MESSAGE_DEBUG: Inserted coder converter after flatten
ref_AppliedPTransform_assert_that-Group-Flatten_39
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:38:14.598Z:
JOB_MESSAGE_DETAILED: Unzipping flatten
ref_AppliedPTransform_assert_that-Group-Flatten_39 for input
ref_AppliedPTransform_assert_that-Group-pair_with_0_37.None
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:38:14.632Z:
JOB_MESSAGE_DETAILED: Fusing unzipped copy of
assert_that/Group/Flatten/OutputIdentity, through flatten
assert_that/Group/Flatten, into producer assert_that/Group/pair_with_0
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:38:14.667Z:
JOB_MESSAGE_DETAILED: Fusing consumer
assert_that/Group/Map(_merge_tagged_vals_under_key) into
assert_that/Group/GroupByKey/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:38:14.698Z:
JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Unkey into
assert_that/Group/Map(_merge_tagged_vals_under_key)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:38:14.718Z:
JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Match into assert_that/Unkey
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:38:14.741Z:
JOB_MESSAGE_DETAILED: Unzipping flatten
ref_AppliedPTransform_assert_that-Group-Flatten_39-u26 for input
ref_AppliedPTransform_assert_that-Group-Flatten_39.None-c24
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:38:14.765Z:
JOB_MESSAGE_DETAILED: Fusing unzipped copy of
assert_that/Group/GroupByKey/Write, through flatten
assert_that/Group/Flatten/Unzipped-1, into producer
assert_that/Group/Flatten/OutputIdentity
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:38:14.791Z:
JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Group/Flatten/OutputIdentity
into assert_that/Group/pair_with_1
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:38:14.815Z:
JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Group/GroupByKey/Write into
assert_that/Group/Flatten/OutputIdentity
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:38:14.845Z:
JOB_MESSAGE_DETAILED: Fusing consumer Create/FlatMap(<lambda at core.py:2962>)
into Create/Impulse
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:38:14.873Z:
JOB_MESSAGE_DETAILED: Fusing consumer
Create/MaybeReshuffle/Reshuffle/AddRandomKeys into Create/FlatMap(<lambda at
core.py:2962>)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:38:14.895Z:
JOB_MESSAGE_DETAILED: Fusing consumer
Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/Map(reify_timestamps) into
Create/MaybeReshuffle/Reshuffle/AddRandomKeys
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:38:14.918Z:
JOB_MESSAGE_DETAILED: Fusing consumer
Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Reify into
Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/Map(reify_timestamps)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:38:14.950Z:
JOB_MESSAGE_DETAILED: Fusing consumer
Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Write into
Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Reify
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:38:14.985Z:
JOB_MESSAGE_DETAILED: Fusing consumer
Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/GroupByWindow into
Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:38:15.034Z:
JOB_MESSAGE_DETAILED: Fusing consumer
Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps)
into Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/GroupByWindow
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:38:15.086Z:
JOB_MESSAGE_DETAILED: Fusing consumer
Create/MaybeReshuffle/Reshuffle/RemoveRandomKeys into
Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:38:15.119Z:
JOB_MESSAGE_DETAILED: Fusing consumer Create/Map(decode) into
Create/MaybeReshuffle/Reshuffle/RemoveRandomKeys
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:38:15.154Z:
JOB_MESSAGE_DETAILED: Fusing consumer
ReadAllFromBigQuery/ParDo(_BigQueryReadSplit)/ParDo(_BigQueryReadSplit) into
Create/Map(decode)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:38:15.196Z:
JOB_MESSAGE_DETAILED: Fusing consumer
ref_AppliedPTransform_ReadAllFromBigQuery-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_18/PairWithRestriction
into ReadAllFromBigQuery/ParDo(_BigQueryReadSplit)/ParDo(_BigQueryReadSplit)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:38:15.232Z:
JOB_MESSAGE_DETAILED: Fusing consumer
ref_AppliedPTransform_ReadAllFromBigQuery-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_18/SplitWithSizing
into
ref_AppliedPTransform_ReadAllFromBigQuery-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_18/PairWithRestriction
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:38:15.266Z:
JOB_MESSAGE_DETAILED: Fusing consumer
ReadAllFromBigQuery/_PassThroughThenCleanup/ParDo(PassThrough)/ParDo(PassThrough)
into
ref_AppliedPTransform_ReadAllFromBigQuery-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_18/ProcessElementAndRestrictionWithSizing
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:38:15.303Z:
JOB_MESSAGE_DETAILED: Fusing consumer
ReadAllFromBigQuery/_PassThroughThenCleanup/Create/FlatMap(<lambda at
core.py:2962>) into ReadAllFromBigQuery/_PassThroughThenCleanup/Create/Impulse
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:38:15.333Z:
JOB_MESSAGE_DETAILED: Fusing consumer
ReadAllFromBigQuery/_PassThroughThenCleanup/Create/Map(decode) into
ReadAllFromBigQuery/_PassThroughThenCleanup/Create/FlatMap(<lambda at
core.py:2962>)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:38:15.365Z:
JOB_MESSAGE_DETAILED: Fusing consumer
ReadAllFromBigQuery/_PassThroughThenCleanup/ParDo(RemoveExtractedFiles) into
ReadAllFromBigQuery/_PassThroughThenCleanup/Create/Map(decode)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:38:15.396Z:
JOB_MESSAGE_DETAILED: Fusing consumer assert_that/WindowInto(WindowIntoFn) into
ReadAllFromBigQuery/_PassThroughThenCleanup/ParDo(PassThrough)/ParDo(PassThrough)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:38:15.420Z:
JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Create/FlatMap(<lambda at
core.py:2962>) into assert_that/Create/Impulse
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:38:15.443Z:
JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Create/Map(decode) into
assert_that/Create/FlatMap(<lambda at core.py:2962>)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:38:15.490Z:
JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Group/pair_with_0 into
assert_that/Create/Map(decode)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:38:15.525Z:
JOB_MESSAGE_DETAILED: Fusing consumer assert_that/ToVoidKey into
assert_that/WindowInto(WindowIntoFn)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:38:15.558Z:
JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Group/pair_with_1 into
assert_that/ToVoidKey
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:38:15.597Z:
JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:38:15.625Z:
JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:38:15.659Z:
JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:38:15.679Z:
JOB_MESSAGE_DEBUG: Assigning stage ids.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:38:15.954Z:
JOB_MESSAGE_DEBUG: Executing wait step start37
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:38:16.020Z:
JOB_MESSAGE_BASIC: Executing operation
Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:38:16.056Z:
JOB_MESSAGE_DEBUG: Starting worker pool setup.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:38:16.086Z:
JOB_MESSAGE_BASIC: Starting 1 workers in us-central1-f...
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:38:16.468Z:
JOB_MESSAGE_BASIC: Finished operation
Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:38:16.520Z:
JOB_MESSAGE_DEBUG: Value
"Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Session"
materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:38:16.586Z:
JOB_MESSAGE_BASIC: Executing operation Create/Impulse+Create/FlatMap(<lambda at
core.py:2962>)+Create/MaybeReshuffle/Reshuffle/AddRandomKeys+Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/Map(reify_timestamps)+Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Reify+Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:38:41.534Z:
JOB_MESSAGE_BASIC: Your project already contains 100 Dataflow-created metric
descriptors, so new user metrics of the form custom.googleapis.com/* will not
be created. However, all user metrics are also available in the metric
dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics,
you can delete old / unused metric descriptors. See
https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list
and
https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:39:06.944Z:
JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of workers to 1 based on
the rate of progress in the currently running stage(s).
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:39:45.208Z:
JOB_MESSAGE_DETAILED: Workers have started successfully.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:39:45.235Z:
JOB_MESSAGE_DETAILED: Workers have started successfully.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:45:04.709Z:
JOB_MESSAGE_BASIC: Finished operation Create/Impulse+Create/FlatMap(<lambda at
core.py:2962>)+Create/MaybeReshuffle/Reshuffle/AddRandomKeys+Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/Map(reify_timestamps)+Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Reify+Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:45:04.777Z:
JOB_MESSAGE_BASIC: Executing operation
Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:45:05.184Z:
JOB_MESSAGE_BASIC: Finished operation
Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:45:05.301Z:
JOB_MESSAGE_BASIC: Executing operation
Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Read+Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/GroupByWindow+Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps)+Create/MaybeReshuffle/Reshuffle/RemoveRandomKeys+Create/Map(decode)+ReadAllFromBigQuery/ParDo(_BigQueryReadSplit)/ParDo(_BigQueryReadSplit)+ref_AppliedPTransform_ReadAllFromBigQuery-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_18/PairWithRestriction+ref_AppliedPTransform_ReadAllFromBigQuery-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_18/SplitWithSizing
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:45:45.612Z:
JOB_MESSAGE_BASIC: Finished operation
Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Read+Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/GroupByWindow+Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps)+Create/MaybeReshuffle/Reshuffle/RemoveRandomKeys+Create/Map(decode)+ReadAllFromBigQuery/ParDo(_BigQueryReadSplit)/ParDo(_BigQueryReadSplit)+ref_AppliedPTransform_ReadAllFromBigQuery-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_18/PairWithRestriction+ref_AppliedPTransform_ReadAllFromBigQuery-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_18/SplitWithSizing
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:45:45.675Z:
JOB_MESSAGE_DEBUG: Value
"ReadAllFromBigQuery/ParDo(_BigQueryReadSplit)/ParDo(_BigQueryReadSplit).location_to_cleanup"
materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:45:45.711Z:
JOB_MESSAGE_DEBUG: Value
"ref_AppliedPTransform_ReadAllFromBigQuery-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_18-split-with-sizing-out3"
materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:45:45.735Z:
JOB_MESSAGE_BASIC: Executing operation
ReadAllFromBigQuery/_PassThroughThenCleanup/ParDo(RemoveExtractedFiles)/View-python_side_input1-ReadAllFromBigQuery/_PassThroughThenCleanup/ParDo(RemoveExtractedFiles)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:45:45.780Z:
JOB_MESSAGE_BASIC: Executing operation assert_that/Group/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:45:45.785Z:
JOB_MESSAGE_BASIC: Finished operation
ReadAllFromBigQuery/_PassThroughThenCleanup/ParDo(RemoveExtractedFiles)/View-python_side_input1-ReadAllFromBigQuery/_PassThroughThenCleanup/ParDo(RemoveExtractedFiles)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:45:45.848Z:
JOB_MESSAGE_DEBUG: Value
"ReadAllFromBigQuery/_PassThroughThenCleanup/ParDo(RemoveExtractedFiles)/View-python_side_input1-ReadAllFromBigQuery/_PassThroughThenCleanup/ParDo(RemoveExtractedFiles).out"
materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:45:48.513Z:
JOB_MESSAGE_BASIC: Finished operation assert_that/Group/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:45:48.581Z:
JOB_MESSAGE_DEBUG: Value "assert_that/Group/GroupByKey/Session" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:45:48.650Z:
JOB_MESSAGE_BASIC: Executing operation
ref_AppliedPTransform_ReadAllFromBigQuery-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_18/ProcessElementAndRestrictionWithSizing+ReadAllFromBigQuery/_PassThroughThenCleanup/ParDo(PassThrough)/ParDo(PassThrough)+assert_that/WindowInto(WindowIntoFn)+assert_that/ToVoidKey+assert_that/Group/pair_with_1+assert_that/Group/Flatten/OutputIdentity+assert_that/Group/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:45:48.683Z:
JOB_MESSAGE_BASIC: Executing operation
assert_that/Create/Impulse+assert_that/Create/FlatMap(<lambda at
core.py:2962>)+assert_that/Create/Map(decode)+assert_that/Group/pair_with_0+assert_that/Group/Flatten/OutputIdentity+assert_that/Group/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:46:09.200Z:
JOB_MESSAGE_BASIC: Finished operation
assert_that/Create/Impulse+assert_that/Create/FlatMap(<lambda at
core.py:2962>)+assert_that/Create/Map(decode)+assert_that/Group/pair_with_0+assert_that/Group/Flatten/OutputIdentity+assert_that/Group/GroupByKey/Write
oauth2client.transport: INFO: Refreshing due to a 401 (attempt 1/2)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:51:53.491Z:
JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of workers to 1 based on
the rate of progress in the currently running stage(s).
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T12:57:49.635Z:
JOB_MESSAGE_ERROR: SDK harness sdk-0-0 disconnected.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T13:03:46.458Z:
JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of workers to 1 based on
the rate of progress in the currently running stage(s).
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T13:10:22.942Z:
JOB_MESSAGE_ERROR: SDK harness sdk-0-0 disconnected.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T13:10:24.464Z:
JOB_MESSAGE_BASIC: Finished operation
ref_AppliedPTransform_ReadAllFromBigQuery-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_18/ProcessElementAndRestrictionWithSizing+ReadAllFromBigQuery/_PassThroughThenCleanup/ParDo(PassThrough)/ParDo(PassThrough)+assert_that/WindowInto(WindowIntoFn)+assert_that/ToVoidKey+assert_that/Group/pair_with_1+assert_that/Group/Flatten/OutputIdentity+assert_that/Group/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T13:10:24.572Z:
JOB_MESSAGE_DEBUG: Executing failure step failure36
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T13:10:24.592Z:
JOB_MESSAGE_ERROR: Workflow failed. Causes:
S07:ref_AppliedPTransform_ReadAllFromBigQuery-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_18/ProcessElementAndRestrictionWithSizing+ReadAllFromBigQuery/_PassThroughThenCleanup/ParDo(PassThrough)/ParDo(PassThrough)+assert_that/WindowInto(WindowIntoFn)+assert_that/ToVoidKey+assert_that/Group/pair_with_1+assert_that/Group/Flatten/OutputIdentity+assert_that/Group/GroupByKey/Write
failed., The job failed because a work item has failed 4 times. Look in
previous log entries for the cause of each one of the 4 failures. For more
information, see https://cloud.google.com/dataflow/docs/guides/common-errors.
The work item was attempted on these workers:
beamapp-jenkins-061612375-06160538-69jo-harness-k1sd
Root cause: The worker lost contact with the service.,
beamapp-jenkins-061612375-06160538-69jo-harness-v75m
Root cause: SDK harness sdk-0-0 disconnected.,
beamapp-jenkins-061612375-06160538-69jo-harness-v75m
Root cause: The worker lost contact with the service.,
beamapp-jenkins-061612375-06160538-69jo-harness-37n1
Root cause: SDK harness sdk-0-0 disconnected.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T13:10:24.950Z:
JOB_MESSAGE_DETAILED: Cleaning up.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T13:10:25.168Z:
JOB_MESSAGE_DEBUG: Starting worker pool teardown.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T13:10:25.187Z:
JOB_MESSAGE_BASIC: Stopping worker pool...
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T13:11:14.701Z:
JOB_MESSAGE_DETAILED: Autoscaling: Resized worker pool from 1 to 0.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T13:11:14.733Z:
JOB_MESSAGE_BASIC: Worker pool stopped.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-16T13:11:14.761Z:
JOB_MESSAGE_DEBUG: Tearing down pending resources...
apache_beam.runners.dataflow.dataflow_runner: INFO: Job
2021-06-16_05_38_08-11637755170750681811 is in state JOB_STATE_FAILED
--------------------- >> end captured logging << ---------------------
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-16_05_05_44-12644158585290160600?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-16_05_19_57-2854619695763801108?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-16_05_29_17-8841915674416286005?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-16_05_40_50-10255511779361229255?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-16_05_50_39-5264359172423319749?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-16_06_00_15-1621711853174729378?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-16_06_17_28-750881181187396750?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-16_05_05_39-4616899157800494698?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-16_05_27_07-14996569118231706420?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-16_05_36_33-13555806035454091395?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-16_05_44_28-9313230770795404989?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-16_05_52_33-9020064789116195362?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-16_06_02_18-5074363769419742480?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-16_06_11_55-7674572451796604692?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-16_06_21_38-12875914136613229239?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-16_06_31_44-9986732439982469036?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-16_05_05_41-13311570795316969711?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-16_05_17_22-8238719772439689102?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-16_05_26_40-11511603732602169628?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-16_05_34_50-587025256840877081?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-16_05_45_03-13430517847415889065?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-16_05_53_43-17486302865397368528?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-16_06_02_25-16373448846852979891?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-16_06_12_14-9665638634866294728?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-16_06_20_13-4760489256654662272?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-16_05_05_35-9828907894856521363?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-16_05_29_15-13839919694877082596?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-16_05_37_54-14415632887924283877?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-16_05_47_10-2878631213497934275?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-16_06_02_50-5650583467351590057?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-16_06_11_32-7782918264643582003?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-16_06_21_16-16291470774778667133?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-16_05_07_43-4944476938194561955?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-16_05_18_00-15527058983688969074?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-16_05_28_34-6832845748572001961?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-16_05_38_37-4413875145835097281?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-16_05_48_28-6750398546516749025?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-16_05_59_34-12974073232662429606?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-16_06_09_25-16539204558046614940?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-16_06_20_52-2784556784647203266?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-16_05_05_38-8523713032702637033?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-16_05_14_37-8849963160662857101?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-16_05_23_31-9028581994587209187?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-16_05_31_54-13262349882890510672?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-16_05_40_13-14509770625820341833?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-16_05_49_52-14991275383120456469?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-16_05_57_50-9453331530867813764?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-16_06_06_42-246249657891730443?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-16_06_15_57-5203469920811228926?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-16_06_25_14-5017787222115800603?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-16_05_05_40-699293665881026806?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-16_05_15_42-13859658414001795208?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-16_05_26_46-16718105553502013960?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-16_05_38_08-11637755170750681811?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-16_06_11_35-10742361608235897258?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-16_06_19_30-2878858659053242184?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-16_05_05_41-17537153868263984227?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-16_05_15_23-14397980627096334455?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-16_05_31_38-16648436272603734388?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-16_05_57_36-3681509160827773679?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-16_06_06_35-7613565650233052590?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-16_06_14_17-16478705143807322959?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-16_06_22_35-2552276728518388170?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-16_06_30_45-15551592811275904600?project=apache-beam-testing
----------------------------------------------------------------------
XML: nosetests-postCommitIT-df-py38.xml
----------------------------------------------------------------------
XML:
<https://ci-beam.apache.org/job/beam_PostCommit_Python38/ws/src/sdks/python/nosetests.xml>
----------------------------------------------------------------------
Ran 71 tests in 5743.843s
FAILED (SKIP=8, errors=1)
> Task :sdks:python:test-suites:dataflow:py38:postCommitIT FAILED
FAILURE: Build failed with an exception.
* Where:
Script
'<https://ci-beam.apache.org/job/beam_PostCommit_Python38/ws/src/sdks/python/test-suites/dataflow/common.gradle'>
line: 126
* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py38:postCommitIT'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug
option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with
Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See
https://docs.gradle.org/6.8.3/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 1h 40m 9s
214 actionable tasks: 152 executed, 58 from cache, 4 up-to-date
Publishing build scan...
https://gradle.com/s/yj3pq5qqy4uxs
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
