See
<https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/799/display/redirect?page=changes>
Changes:
[Brian Hulette] Add DataFrame API updates to CHANGES.md
------------------------------------------
[...truncated 2.15 MB...]
"value": "_equal"
}
],
"non_parallel_inputs": {},
"output_info": [
{
"encoding": {
"@type": "kind:windowed_value",
"component_encodings": [
{
"@type":
"FastPrimitivesCoder$eNprYE5OLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqYIfgYGhvi0xJycpMTk7HiwlkJ8pgVkJmfnpEJNYQGawlpbyJZUnKQHACYlLgM=",
"component_encodings": [
{
"@type":
"FastPrimitivesCoder$eNprYE5OLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqYIfgYGhvi0xJycpMTk7HiwlkJ8pgVkJmfnpEJNYQGawlpbyJZUnKQHACYlLgM=",
"component_encodings": [],
"pipeline_proto_coder_id":
"ref_Coder_FastPrimitivesCoder_7"
},
{
"@type":
"FastPrimitivesCoder$eNprYE5OLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqYIfgYGhvi0xJycpMTk7HiwlkJ8pgVkJmfnpEJNYQGawlpbyJZUnKQHACYlLgM=",
"component_encodings": [],
"pipeline_proto_coder_id":
"ref_Coder_FastPrimitivesCoder_7"
}
],
"is_pair_like": true,
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_7"
},
{
"@type": "kind:global_window"
}
],
"is_wrapper": true
},
"output_name": "None",
"user_name": "assert_that/Match.out"
}
],
"parallel_input": {
"@type": "OutputReference",
"output_name": "None",
"step_name": "s20"
},
"serialized_fn": "ref_AppliedPTransform_assert_that-Match_30",
"user_name": "assert_that/Match"
}
}
],
"type": "JOB_TYPE_BATCH"
}
apache_beam.runners.dataflow.internal.apiclient: INFO: Create job: <Job
createTime: '2021-06-02T06:15:58.134667Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2021-06-01_23_15_57-18389643972782428164'
location: 'us-central1'
name: 'beamapp-jenkins-0602061550-897099'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2021-06-02T06:15:58.134667Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
apache_beam.runners.dataflow.internal.apiclient: INFO: Created job with id:
[2021-06-01_23_15_57-18389643972782428164]
apache_beam.runners.dataflow.internal.apiclient: INFO: Submitted job:
2021-06-01_23_15_57-18389643972782428164
apache_beam.runners.dataflow.internal.apiclient: INFO: To access the Dataflow
monitoring console, please navigate to
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-01_23_15_57-18389643972782428164?project=apache-beam-testing
apache_beam.runners.dataflow.dataflow_runner: INFO: Job
2021-06-01_23_15_57-18389643972782428164 is in state JOB_STATE_RUNNING
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:01.619Z:
JOB_MESSAGE_DETAILED: Autoscaling is enabled for job
2021-06-01_23_15_57-18389643972782428164. The number of workers will be between
1 and 1000.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:01.694Z:
JOB_MESSAGE_DETAILED: Autoscaling was automatically enabled for job
2021-06-01_23_15_57-18389643972782428164.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:04.153Z:
JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in us-central1-f.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:04.989Z:
JOB_MESSAGE_DETAILED: Expanding SplittableParDo operations into optimizable
parts.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:05.043Z:
JOB_MESSAGE_DETAILED: Expanding CollectionToSingleton operations into
optimizable parts.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:05.481Z:
JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:05.772Z:
JOB_MESSAGE_DEBUG: Combiner lifting skipped for step
assert_that/Group/GroupByKey: GroupByKey not followed by a combiner.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:06.076Z:
JOB_MESSAGE_DEBUG: Combiner lifting skipped for step
Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey: GroupByKey not
followed by a combiner.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:06.247Z:
JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:06.360Z:
JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:06.762Z:
JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:06.870Z:
JOB_MESSAGE_DEBUG: Inserted coder converter after flatten
ref_AppliedPTransform_assert_that-Group-Flatten_26
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:06.961Z:
JOB_MESSAGE_DETAILED: Unzipping flatten
ref_AppliedPTransform_assert_that-Group-Flatten_26 for input
ref_AppliedPTransform_assert_that-Group-pair_with_0_24.None
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:07.053Z:
JOB_MESSAGE_DETAILED: Fusing unzipped copy of
assert_that/Group/Flatten/OutputIdentity, through flatten
assert_that/Group/Flatten, into producer assert_that/Group/pair_with_0
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:07.097Z:
JOB_MESSAGE_DETAILED: Fusing consumer
assert_that/Group/Map(_merge_tagged_vals_under_key) into
assert_that/Group/GroupByKey/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:07.150Z:
JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Unkey into
assert_that/Group/Map(_merge_tagged_vals_under_key)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:07.182Z:
JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Match into assert_that/Unkey
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:07.204Z:
JOB_MESSAGE_DETAILED: Unzipping flatten
ref_AppliedPTransform_assert_that-Group-Flatten_26-u40 for input
ref_AppliedPTransform_assert_that-Group-Flatten_26.None-c38
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:07.237Z:
JOB_MESSAGE_DETAILED: Fusing unzipped copy of
assert_that/Group/GroupByKey/Write, through flatten
assert_that/Group/Flatten/Unzipped-1, into producer
assert_that/Group/Flatten/OutputIdentity
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:07.268Z:
JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Group/Flatten/OutputIdentity
into assert_that/Group/pair_with_1
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:07.305Z:
JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Group/GroupByKey/Write into
assert_that/Group/Flatten/OutputIdentity
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:07.338Z:
JOB_MESSAGE_DETAILED: Unzipping flatten
external_1ExternalTransform-beam-transforms-xlang-test-comgl--Flatten-PCollections
for input
external_1ExternalTransform-beam-transforms-xlang-test-comgl--ProduceDefault-ParMultiDo-Anonymous-.output
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:07.392Z:
JOB_MESSAGE_DETAILED: Fusing unzipped copy of
assert_that/WindowInto(WindowIntoFn), through flatten
ExternalTransform(beam:transforms:xlang:test:comgl)/Flatten.PCollections, into
producer
ExternalTransform(beam:transforms:xlang:test:comgl)/ProduceDefault/ParMultiDo(Anonymous)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:07.415Z:
JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Group/pair_with_1 into
assert_that/ToVoidKey
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:07.447Z:
JOB_MESSAGE_DETAILED: Unzipping flatten
external_1ExternalTransform-beam-transforms-xlang-test-u47 for input
ref_AppliedPTransform_assert_that-WindowInto-WindowIntoFn-_21.None-c45
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:07.479Z:
JOB_MESSAGE_DETAILED: Fusing unzipped copy of assert_that/ToVoidKey, through
flatten
ExternalTransform(beam:transforms:xlang:test:comgl)/Flatten.PCollections/Unzipped-1,
into producer assert_that/WindowInto(WindowIntoFn)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:07.500Z:
JOB_MESSAGE_DETAILED: Fusing consumer
ExternalTransform(beam:transforms:xlang:test:comgl)/View.AsIterable/MapElements/Map/ParMultiDo(Anonymous)
into
ExternalTransform(beam:transforms:xlang:test:comgl)/Values/Values/Map/ParMultiDo(Anonymous)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:07.534Z:
JOB_MESSAGE_DETAILED: Fusing consumer assert_that/WindowInto(WindowIntoFn) into
ExternalTransform(beam:transforms:xlang:test:comgl)/Values/Values/Map/ParMultiDo(Anonymous)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:07.574Z:
JOB_MESSAGE_DETAILED: Fusing consumer assert_that/ToVoidKey into
assert_that/WindowInto(WindowIntoFn)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:07.605Z:
JOB_MESSAGE_DETAILED: Fusing consumer Create/FlatMap(<lambda at core.py:2962>)
into Create/Impulse
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:07.637Z:
JOB_MESSAGE_DETAILED: Fusing consumer
Create/MaybeReshuffle/Reshuffle/AddRandomKeys into Create/FlatMap(<lambda at
core.py:2962>)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:07.670Z:
JOB_MESSAGE_DETAILED: Fusing consumer
Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/Map(reify_timestamps) into
Create/MaybeReshuffle/Reshuffle/AddRandomKeys
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:07.724Z:
JOB_MESSAGE_DETAILED: Fusing consumer
Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Reify into
Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/Map(reify_timestamps)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:07.756Z:
JOB_MESSAGE_DETAILED: Fusing consumer
Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Write into
Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Reify
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:07.789Z:
JOB_MESSAGE_DETAILED: Fusing consumer
Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/GroupByWindow into
Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:07.813Z:
JOB_MESSAGE_DETAILED: Fusing consumer
Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps)
into Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/GroupByWindow
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:07.834Z:
JOB_MESSAGE_DETAILED: Fusing consumer
Create/MaybeReshuffle/Reshuffle/RemoveRandomKeys into
Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:07.865Z:
JOB_MESSAGE_DETAILED: Fusing consumer Create/Map(decode) into
Create/MaybeReshuffle/Reshuffle/RemoveRandomKeys
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:07.888Z:
JOB_MESSAGE_DETAILED: Fusing consumer
ExternalTransform(beam:transforms:xlang:test:comgl)/WithKeys/AddKeys/Map/ParMultiDo(Anonymous)
into Create/Map(decode)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:07.917Z:
JOB_MESSAGE_DETAILED: Fusing consumer
ExternalTransform(beam:transforms:xlang:test:comgl)/Combine.perKey(SumLong)/GroupByKey+ExternalTransform(beam:transforms:xlang:test:comgl)/Combine.perKey(SumLong)/Combine.GroupedValues/Partial
into
ExternalTransform(beam:transforms:xlang:test:comgl)/WithKeys/AddKeys/Map/ParMultiDo(Anonymous)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:07.939Z:
JOB_MESSAGE_DETAILED: Fusing consumer
ExternalTransform(beam:transforms:xlang:test:comgl)/Combine.perKey(SumLong)/GroupByKey/Write
into
ExternalTransform(beam:transforms:xlang:test:comgl)/Combine.perKey(SumLong)/GroupByKey+ExternalTransform(beam:transforms:xlang:test:comgl)/Combine.perKey(SumLong)/Combine.GroupedValues/Partial
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:07.967Z:
JOB_MESSAGE_DETAILED: Fusing consumer
ExternalTransform(beam:transforms:xlang:test:comgl)/Combine.perKey(SumLong)/Combine.GroupedValues
into
ExternalTransform(beam:transforms:xlang:test:comgl)/Combine.perKey(SumLong)/GroupByKey/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:08.043Z:
JOB_MESSAGE_DETAILED: Fusing consumer
ExternalTransform(beam:transforms:xlang:test:comgl)/Combine.perKey(SumLong)/Combine.GroupedValues/Extract
into
ExternalTransform(beam:transforms:xlang:test:comgl)/Combine.perKey(SumLong)/Combine.GroupedValues
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:08.074Z:
JOB_MESSAGE_DETAILED: Fusing consumer
ExternalTransform(beam:transforms:xlang:test:comgl)/Values/Values/Map/ParMultiDo(Anonymous)
into
ExternalTransform(beam:transforms:xlang:test:comgl)/Combine.perKey(SumLong)/Combine.GroupedValues/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:08.106Z:
JOB_MESSAGE_DETAILED: Fusing consumer
ExternalTransform(beam:transforms:xlang:test:comgl)/CreateVoid/Read(CreateSource)/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource)
into
ExternalTransform(beam:transforms:xlang:test:comgl)/CreateVoid/Read(CreateSource)/Impulse
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:08.141Z:
JOB_MESSAGE_DETAILED: Fusing consumer
external_1ExternalTransform-beam-transforms-xlang-test-comgl--CreateVoid-Read-CreateSource--ParDo-BoundedSourc/PairWithRestriction
into
ExternalTransform(beam:transforms:xlang:test:comgl)/CreateVoid/Read(CreateSource)/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:08.175Z:
JOB_MESSAGE_DETAILED: Fusing consumer
external_1ExternalTransform-beam-transforms-xlang-test-comgl--CreateVoid-Read-CreateSource--ParDo-BoundedSourc/SplitWithSizing
into
external_1ExternalTransform-beam-transforms-xlang-test-comgl--CreateVoid-Read-CreateSource--ParDo-BoundedSourc/PairWithRestriction
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:08.197Z:
JOB_MESSAGE_DETAILED: Fusing consumer
ExternalTransform(beam:transforms:xlang:test:comgl)/ProduceDefault/ParMultiDo(Anonymous)
into
external_1ExternalTransform-beam-transforms-xlang-test-comgl--CreateVoid-Read-CreateSource--ParDo-BoundedSourc/ProcessElementAndRestrictionWithSizing
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:08.228Z:
JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Create/FlatMap(<lambda at
core.py:2962>) into assert_that/Create/Impulse
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:08.262Z:
JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Create/Map(decode) into
assert_that/Create/FlatMap(<lambda at core.py:2962>)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:08.306Z:
JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Group/pair_with_0 into
assert_that/Create/Map(decode)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:08.347Z:
JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:08.393Z:
JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:08.427Z:
JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:08.461Z:
JOB_MESSAGE_DEBUG: Assigning stage ids.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:08.607Z:
JOB_MESSAGE_DEBUG: Executing wait step start65
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:08.672Z:
JOB_MESSAGE_BASIC: Executing operation
ExternalTransform(beam:transforms:xlang:test:comgl)/CreateVoid/Read(CreateSource)/Impulse+ExternalTransform(beam:transforms:xlang:test:comgl)/CreateVoid/Read(CreateSource)/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource)+external_1ExternalTransform-beam-transforms-xlang-test-comgl--CreateVoid-Read-CreateSource--ParDo-BoundedSourc/PairWithRestriction+external_1ExternalTransform-beam-transforms-xlang-test-comgl--CreateVoid-Read-CreateSource--ParDo-BoundedSourc/SplitWithSizing
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:08.709Z:
JOB_MESSAGE_DEBUG: Starting worker pool setup.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:08.725Z:
JOB_MESSAGE_BASIC: Executing operation
ExternalTransform(beam:transforms:xlang:test:comgl)/Combine.perKey(SumLong)/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:08.735Z:
JOB_MESSAGE_BASIC: Starting 1 workers in us-central1-f...
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:08.757Z:
JOB_MESSAGE_BASIC: Executing operation
Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:08.779Z:
JOB_MESSAGE_BASIC: Executing operation assert_that/Group/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:08.792Z:
JOB_MESSAGE_BASIC: Finished operation
ExternalTransform(beam:transforms:xlang:test:comgl)/Combine.perKey(SumLong)/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:08.809Z:
JOB_MESSAGE_BASIC: Finished operation
Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:08.827Z:
JOB_MESSAGE_BASIC: Finished operation assert_that/Group/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:08.853Z:
JOB_MESSAGE_DEBUG: Value
"ExternalTransform(beam:transforms:xlang:test:comgl)/Combine.perKey(SumLong)/GroupByKey/Session"
materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:08.874Z:
JOB_MESSAGE_DEBUG: Value
"Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Session"
materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:08.930Z:
JOB_MESSAGE_DEBUG: Value "assert_that/Group/GroupByKey/Session" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:09.083Z:
JOB_MESSAGE_BASIC: Executing operation Create/Impulse+Create/FlatMap(<lambda at
core.py:2962>)+Create/MaybeReshuffle/Reshuffle/AddRandomKeys+Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/Map(reify_timestamps)+Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Reify+Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:09.363Z:
JOB_MESSAGE_BASIC: Executing operation
assert_that/Create/Impulse+assert_that/Create/FlatMap(<lambda at
core.py:2962>)+assert_that/Create/Map(decode)+assert_that/Group/pair_with_0+assert_that/Group/Flatten/OutputIdentity+assert_that/Group/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:16:27.558Z:
JOB_MESSAGE_BASIC: Your project already contains 100 Dataflow-created metric
descriptors, so new user metrics of the form custom.googleapis.com/* will not
be created. However, all user metrics are also available in the metric
dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics,
you can delete old / unused metric descriptors. See
https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list
and
https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:17:04.567Z:
JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of workers to 1 based on
the rate of progress in the currently running stage(s).
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:17:29.345Z:
JOB_MESSAGE_DETAILED: Workers have started successfully.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T06:17:29.483Z:
JOB_MESSAGE_DETAILED: Workers have started successfully.
oauth2client.transport: INFO: Refreshing due to a 401 (attempt 1/2)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T07:16:09.860Z:
JOB_MESSAGE_ERROR: Workflow failed. Causes: The Dataflow job appears to be
stuck because no worker activity has been seen in the last 1h. Please check the
worker logs in Stackdriver Logging. You can also get help with Cloud Dataflow
at https://cloud.google.com/dataflow/support.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T07:16:10.371Z:
JOB_MESSAGE_BASIC: Cancel request is committed for workflow job:
2021-06-01_23_15_57-18389643972782428164.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T07:16:10.551Z:
JOB_MESSAGE_BASIC: Finished operation Create/Impulse+Create/FlatMap(<lambda at
core.py:2962>)+Create/MaybeReshuffle/Reshuffle/AddRandomKeys+Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/Map(reify_timestamps)+Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Reify+Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T07:16:10.554Z:
JOB_MESSAGE_BASIC: Finished operation
assert_that/Create/Impulse+assert_that/Create/FlatMap(<lambda at
core.py:2962>)+assert_that/Create/Map(decode)+assert_that/Group/pair_with_0+assert_that/Group/Flatten/OutputIdentity+assert_that/Group/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T07:16:11.001Z:
JOB_MESSAGE_WARNING:
S11:ExternalTransform(beam:transforms:xlang:test:comgl)/CreateVoid/Read(CreateSource)/Impulse+ExternalTransform(beam:transforms:xlang:test:comgl)/CreateVoid/Read(CreateSource)/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource)+external_1ExternalTransform-beam-transforms-xlang-test-comgl--CreateVoid-Read-CreateSource--ParDo-BoundedSourc/PairWithRestriction+external_1ExternalTransform-beam-transforms-xlang-test-comgl--CreateVoid-Read-CreateSource--ParDo-BoundedSourc/SplitWithSizing
failed.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T07:16:11.026Z:
JOB_MESSAGE_BASIC: Finished operation
ExternalTransform(beam:transforms:xlang:test:comgl)/CreateVoid/Read(CreateSource)/Impulse+ExternalTransform(beam:transforms:xlang:test:comgl)/CreateVoid/Read(CreateSource)/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource)+external_1ExternalTransform-beam-transforms-xlang-test-comgl--CreateVoid-Read-CreateSource--ParDo-BoundedSourc/PairWithRestriction+external_1ExternalTransform-beam-transforms-xlang-test-comgl--CreateVoid-Read-CreateSource--ParDo-BoundedSourc/SplitWithSizing
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T07:16:11.511Z:
JOB_MESSAGE_DETAILED: Cleaning up.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T07:16:11.618Z:
JOB_MESSAGE_DEBUG: Starting worker pool teardown.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T07:16:11.653Z:
JOB_MESSAGE_BASIC: Stopping worker pool...
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T07:17:02.887Z:
JOB_MESSAGE_DETAILED: Autoscaling: Resized worker pool from 1 to 0.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T07:17:03.177Z:
JOB_MESSAGE_BASIC: Worker pool stopped.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-06-02T07:17:03.476Z:
JOB_MESSAGE_DEBUG: Tearing down pending resources...
apache_beam.runners.dataflow.dataflow_runner: INFO: Job
2021-06-01_23_15_57-18389643972782428164 is in state JOB_STATE_FAILED
--------------------- >> end captured logging << ---------------------
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-01_23_15_57-10476815187715795526?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-01_23_15_57-6793507679098853754?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-01_23_15_57-18389643972782428164?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-01_23_15_57-4140263778924904262?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-01_23_15_56-3190825879025656663?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-01_23_15_57-6975065003332386104?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-01_23_15_57-302562077605949379?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-01_23_15_57-7216457024713743095?project=apache-beam-testing
----------------------------------------------------------------------
XML: nosetests-xlangValidateRunner.xml
----------------------------------------------------------------------
XML:
<https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/ws/src/sdks/python/nosetests.xml>
----------------------------------------------------------------------
Ran 8 tests in 3700.921s
FAILED (errors=7)
> Task
> :runners:google-cloud-dataflow-java:validatesCrossLanguageRunnerPythonUsingJava
> FAILED
> Task :runners:google-cloud-dataflow-java:validatesCrossLanguageRunnerCleanup
Stopping expansion service pid: 16285.
Stopping expansion service pid: 16288.
> Task :runners:google-cloud-dataflow-java:cleanUpDockerImages FAILED
Untagged: us.gcr.io/apache-beam-testing/java-postcommit-it/java:20210602060105
Untagged:
us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:127cc1e4e4e9418aad8174d2fdb24de134acadbe08e26abc0af4952d15c9987f
Deleted: sha256:7dfbe1bdb944107e0d6aa7aff86f91a60973407b64d9c54b6cf73887654b713c
Deleted: sha256:a2133970553de141826d270ba5f0e3bd8d0e0fb6ee8300f38d1e3417e1ea13eb
Deleted: sha256:1bc8f1d2a3d6bc702f7984e0063e5714fdaa3fdac25eab680f718dc85e7649fd
Deleted: sha256:99c2811fc8978fd5aef86d9b2a3d5f1a409736d8d806b0134f585d7fc41fc454
Deleted: sha256:8ca6ae24eb8848daba931a61adb835a84acdb3f5a8b4cb6cacb8f1f09f5ccd17
Deleted: sha256:f35f84f16f232036bafb06cf3471ccb32a07c7f1acf9d7fc8810ec74e7206ffc
Deleted: sha256:cc31e100d3ef60ce6a5bcecb602021fefee8fb848cbb65e079921be21de1c332
Deleted: sha256:746b9259193b69f2ee11e84475f8197395861c808d7bfa77026aa0b476e65317
Deleted: sha256:334b9f7369b34c1bee43e52c45bc2f8b0141b408a48869ef472e4f9f18a7c121
Deleted: sha256:639b85e2a0e3f729d8349ade4936844e7c1449025911a9a1b2af8b1a9f3a8782
Deleted: sha256:35e7f0205c6f8b204d8f220fee1cd9bfdf7283657c662633e8801e3d82ca43b1
Deleted: sha256:211c3e0fe9384fdae5e9f813055d1a03d7f47abd158fa65c79ea3f8ea2cb38d3
Deleted: sha256:ebc57d696735b2554191b3bbb86f60408df82abbc62578f7a2e19e13f799ada1
Deleted: sha256:d8d5f8da662f53718f79b57c065b2f424ac06c4e8e0864f9ecf5ca83bab756fc
Deleted: sha256:eb22ff6421501801827184e460f4d66c6d8f77917b849feee9588122d5a2124a
Deleted: sha256:4217575a27661f9000ea15b42767b0156e40ede29a7356c4b282a15b67fe3553
Deleted: sha256:2b312e862501668d177bf89b9af83abe09388d87d9cc157d0f100d0e3a71bfa2
Deleted: sha256:eb14effdc1e671044e5a1c632435c44fbd2fe8101bce6cb7d9973512f86e7d9a
Deleted: sha256:dc0264dfafc0cd8a699dc4c631256df5fe315d1cbfca95c2e3b075543d11fa39
Deleted: sha256:d3795a4f7e7a7730b8571c81c4d867fcfb48bedf13b2464611e2c27af8e4efb9
Deleted: sha256:de8e8580d00ff834b5c557cb9e1545c7d6a9be2d8a4c56e7508c747a28620f07
Deleted: sha256:d1e86a26a1f97f39ac993d7042a94d747b8cbe1081d9b5922d939d973eaf7b8e
Deleted: sha256:175bbe07c9fdfe3b44ce77e3014befbd17b22aa8440052bc79109d274b4e3332
Deleted: sha256:d27ccc22c9ff07c5807efe3e0fe547abfc183f11bbce3b27bfd968c49c41a450
Deleted: sha256:6dafb22175529806cf9c5c32c994c4952348340866f83eab4eed7241850fa2ff
Deleted: sha256:dc09281bebf173033f1fad2fe038f654fd32253158218d2007cd77270a9126da
Deleted: sha256:8ac62231ec7fb35bc9b7f67f461a443ca61cd863575577e4f72fc9142a44db53
Deleted: sha256:adbf84e6ef7219082dfcd7326934ef5026c01a2cbb3bf6a78d9588c0ac4c0ecf
ERROR: (gcloud.container.images.delete)
[us.gcr.io/apache-beam-testing/java-postcommit-it/java:20210602060105] is not a
valid name. Expected tag in the form "base:tag" or "tag" or digest in the form
"sha256:<digest>"
FAILURE: Build completed with 2 failures.
1: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task
':runners:google-cloud-dataflow-java:validatesCrossLanguageRunnerPythonUsingJava'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug
option to get more log output. Run with --scan to get full insights.
==============================================================================
2: Task failed with an exception.
-----------
* Where:
Build file
'<https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/ws/src/runners/google-cloud-dataflow-java/build.gradle'>
line: 279
* What went wrong:
Execution failed for task
':runners:google-cloud-dataflow-java:cleanUpDockerImages'.
> Process 'command 'gcloud'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug
option to get more log output. Run with --scan to get full insights.
==============================================================================
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with
Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See
https://docs.gradle.org/6.8.3/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 1h 16m 55s
113 actionable tasks: 84 executed, 25 from cache, 4 up-to-date
Publishing build scan...
https://gradle.com/s/ux7orwllmw2l4
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
