See
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/7891/display/redirect?page=changes>
Changes:
[iemejia] [BEAM-5775] Implement a custom class to lazily encode values for
[iemejia] [BEAM-5775] Rename BeamSparkRunnerRegistrator to
[iemejia] [BEAM-5775] Rename avoidRddSerialization to canAvoidRddSerialization
[iemejia] [BEAM-5775] Update spark runner to use non-deprecated Coder API
methods
[iemejia] [BEAM-5775] Make TranslationUtils iterators Java 8 style
------------------------------------------
[...truncated 540.66 KB...]
root: INFO: 2019-04-12T13:37:42.917Z: JOB_MESSAGE_DETAILED: Fusing consumer
read/Read/Reshuffle/ReshufflePerKey/Map(reify_timestamps) into
read/Read/Reshuffle/AddRandomKeys
root: INFO: 2019-04-12T13:37:42.952Z: JOB_MESSAGE_DETAILED: Fusing consumer
write/Write/WriteImpl/WindowInto(WindowIntoFn) into write/Write/WriteImpl/Pair
root: INFO: 2019-04-12T13:37:42.994Z: JOB_MESSAGE_DETAILED: Fusing consumer
write/Write/WriteImpl/Pair into write/Write/WriteImpl/WriteBundles/WriteBundles
root: INFO: 2019-04-12T13:37:43.041Z: JOB_MESSAGE_DETAILED: Fusing consumer
read/Read/Reshuffle/AddRandomKeys into read/Read/Split
root: INFO: 2019-04-12T13:37:43.085Z: JOB_MESSAGE_DETAILED: Fusing consumer
write/Write/WriteImpl/WriteBundles/WriteBundles into format
root: INFO: 2019-04-12T13:37:43.124Z: JOB_MESSAGE_DETAILED: Fusing consumer
format into count
root: INFO: 2019-04-12T13:37:43.160Z: JOB_MESSAGE_DETAILED: Fusing consumer
write/Write/WriteImpl/FinalizeWrite/_DataflowIterableSideInput(MapToVoidKey2.out.0)/ToIsmRecordForMultimap
into
write/Write/WriteImpl/FinalizeWrite/_DataflowIterableSideInput(MapToVoidKey2.out.0)/GroupByKeyHashAndSortByKeyAndWindow/Read
root: INFO: 2019-04-12T13:37:43.200Z: JOB_MESSAGE_DETAILED: Fusing consumer
write/Write/WriteImpl/FinalizeWrite/_DataflowIterableSideInput(MapToVoidKey2.out.0)/GroupByKeyHashAndSortByKeyAndWindow/Write
into
write/Write/WriteImpl/FinalizeWrite/_DataflowIterableSideInput(MapToVoidKey2.out.0)/CreateIsmShardKeyAndSortKey
root: INFO: 2019-04-12T13:37:43.239Z: JOB_MESSAGE_DETAILED: Fusing consumer
write/Write/WriteImpl/FinalizeWrite/_DataflowIterableSideInput(MapToVoidKey2.out.0)/CreateIsmShardKeyAndSortKey
into write/Write/WriteImpl/FinalizeWrite/MapToVoidKey2
root: INFO: 2019-04-12T13:37:43.263Z: JOB_MESSAGE_DETAILED: Fusing consumer
group/Reify into pair_with_one
root: INFO: 2019-04-12T13:37:43.300Z: JOB_MESSAGE_DETAILED: Fusing consumer
write/Write/WriteImpl/FinalizeWrite/_DataflowIterableSideInput(MapToVoidKey1.out.0)/GroupByKeyHashAndSortByKeyAndWindow/Write
into
write/Write/WriteImpl/FinalizeWrite/_DataflowIterableSideInput(MapToVoidKey1.out.0)/CreateIsmShardKeyAndSortKey
root: INFO: 2019-04-12T13:37:43.349Z: JOB_MESSAGE_DETAILED: Fusing consumer
write/Write/WriteImpl/FinalizeWrite/_DataflowIterableSideInput(MapToVoidKey1.out.0)/CreateIsmShardKeyAndSortKey
into write/Write/WriteImpl/FinalizeWrite/MapToVoidKey1
root: INFO: 2019-04-12T13:37:43.387Z: JOB_MESSAGE_DETAILED: Fusing consumer
write/Write/WriteImpl/FinalizeWrite/_DataflowIterableSideInput(MapToVoidKey0.out.0)/ToIsmRecordForMultimap
into
write/Write/WriteImpl/FinalizeWrite/_DataflowIterableSideInput(MapToVoidKey0.out.0)/GroupByKeyHashAndSortByKeyAndWindow/Read
root: INFO: 2019-04-12T13:37:43.419Z: JOB_MESSAGE_DETAILED: Fusing consumer
write/Write/WriteImpl/FinalizeWrite/_DataflowIterableSideInput(MapToVoidKey0.out.0)/GroupByKeyHashAndSortByKeyAndWindow/Write
into
write/Write/WriteImpl/FinalizeWrite/_DataflowIterableSideInput(MapToVoidKey0.out.0)/CreateIsmShardKeyAndSortKey
root: INFO: 2019-04-12T13:37:43.455Z: JOB_MESSAGE_DETAILED: Fusing consumer
write/Write/WriteImpl/FinalizeWrite/_DataflowIterableSideInput(MapToVoidKey0.out.0)/CreateIsmShardKeyAndSortKey
into write/Write/WriteImpl/FinalizeWrite/MapToVoidKey0
root: INFO: 2019-04-12T13:37:43.487Z: JOB_MESSAGE_DETAILED: Fusing consumer
write/Write/WriteImpl/DoOnce/FlatMap(<lambda at core.py:2172>) into
write/Write/WriteImpl/DoOnce/Impulse
root: INFO: 2019-04-12T13:37:43.529Z: JOB_MESSAGE_DETAILED: Fusing consumer
read/Read/Split into read/Read/Impulse
root: INFO: 2019-04-12T13:37:43.567Z: JOB_MESSAGE_DETAILED: Fusing consumer
count into group/GroupByWindow
root: INFO: 2019-04-12T13:37:43.617Z: JOB_MESSAGE_DETAILED: Fusing consumer
write/Write/WriteImpl/PreFinalize/_DataflowIterableSideInput(MapToVoidKey1.out.0)/GroupByKeyHashAndSortByKeyAndWindow/Write
into
write/Write/WriteImpl/PreFinalize/_DataflowIterableSideInput(MapToVoidKey1.out.0)/CreateIsmShardKeyAndSortKey
root: INFO: 2019-04-12T13:37:43.656Z: JOB_MESSAGE_DETAILED: Fusing consumer
write/Write/WriteImpl/PreFinalize/_DataflowIterableSideInput(MapToVoidKey1.out.0)/CreateIsmShardKeyAndSortKey
into write/Write/WriteImpl/PreFinalize/MapToVoidKey1
root: INFO: 2019-04-12T13:37:43.690Z: JOB_MESSAGE_DETAILED: Fusing consumer
read/Read/Reshuffle/ReshufflePerKey/GroupByKey/GroupByWindow into
read/Read/Reshuffle/ReshufflePerKey/GroupByKey/Read
root: INFO: 2019-04-12T13:37:43.737Z: JOB_MESSAGE_DETAILED: Fusing consumer
write/Write/WriteImpl/Extract into
write/Write/WriteImpl/GroupByKey/GroupByWindow
root: INFO: 2019-04-12T13:37:43.799Z: JOB_MESSAGE_DETAILED: Fusing consumer
write/Write/WriteImpl/GroupByKey/GroupByWindow into
write/Write/WriteImpl/GroupByKey/Read
root: INFO: 2019-04-12T13:37:43.831Z: JOB_MESSAGE_DETAILED: Fusing consumer
write/Write/WriteImpl/GroupByKey/Write into
write/Write/WriteImpl/GroupByKey/Reify
root: INFO: 2019-04-12T13:37:43.871Z: JOB_MESSAGE_DETAILED: Fusing consumer
write/Write/WriteImpl/GroupByKey/Reify into
write/Write/WriteImpl/WindowInto(WindowIntoFn)
root: INFO: 2019-04-12T13:37:43.909Z: JOB_MESSAGE_DETAILED: Fusing consumer
group/GroupByWindow into group/Read
root: INFO: 2019-04-12T13:37:43.955Z: JOB_MESSAGE_DETAILED: Fusing consumer
write/Write/WriteImpl/WriteBundles/_DataflowIterableSideInput(MapToVoidKey0.out.0)/GroupByKeyHashAndSortByKeyAndWindow/Write
into
write/Write/WriteImpl/WriteBundles/_DataflowIterableSideInput(MapToVoidKey0.out.0)/CreateIsmShardKeyAndSortKey
root: INFO: 2019-04-12T13:37:43.993Z: JOB_MESSAGE_DETAILED: Fusing consumer
group/Write into group/Reify
root: INFO: 2019-04-12T13:37:44.043Z: JOB_MESSAGE_DETAILED: Fusing consumer
write/Write/WriteImpl/WriteBundles/_DataflowIterableSideInput(MapToVoidKey0.out.0)/CreateIsmShardKeyAndSortKey
into write/Write/WriteImpl/WriteBundles/MapToVoidKey0
root: INFO: 2019-04-12T13:37:44.089Z: JOB_MESSAGE_DETAILED: Fusing consumer
write/Write/WriteImpl/PreFinalize/_DataflowIterableSideInput(MapToVoidKey0.out.0)/ToIsmRecordForMultimap
into
write/Write/WriteImpl/PreFinalize/_DataflowIterableSideInput(MapToVoidKey0.out.0)/GroupByKeyHashAndSortByKeyAndWindow/Read
root: INFO: 2019-04-12T13:37:44.142Z: JOB_MESSAGE_DETAILED: Fusing consumer
pair_with_one into split
root: INFO: 2019-04-12T13:37:44.184Z: JOB_MESSAGE_DETAILED: Fusing consumer
split into read/Read/ReadSplits
root: INFO: 2019-04-12T13:37:44.228Z: JOB_MESSAGE_DETAILED: Fusing consumer
write/Write/WriteImpl/FinalizeWrite/_DataflowIterableSideInput(MapToVoidKey1.out.0)/ToIsmRecordForMultimap
into
write/Write/WriteImpl/FinalizeWrite/_DataflowIterableSideInput(MapToVoidKey1.out.0)/GroupByKeyHashAndSortByKeyAndWindow/Read
root: INFO: 2019-04-12T13:37:44.274Z: JOB_MESSAGE_DETAILED: Fusing consumer
read/Read/ReadSplits into read/Read/Reshuffle/RemoveRandomKeys
root: INFO: 2019-04-12T13:37:44.315Z: JOB_MESSAGE_DETAILED: Fusing consumer
read/Read/Reshuffle/RemoveRandomKeys into
read/Read/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps)
root: INFO: 2019-04-12T13:37:44.358Z: JOB_MESSAGE_DETAILED: Fusing consumer
write/Write/WriteImpl/PreFinalize/_DataflowIterableSideInput(MapToVoidKey1.out.0)/ToIsmRecordForMultimap
into
write/Write/WriteImpl/PreFinalize/_DataflowIterableSideInput(MapToVoidKey1.out.0)/GroupByKeyHashAndSortByKeyAndWindow/Read
root: INFO: 2019-04-12T13:37:44.397Z: JOB_MESSAGE_DETAILED: Fusing consumer
write/Write/WriteImpl/PreFinalize/_DataflowIterableSideInput(MapToVoidKey0.out.0)/CreateIsmShardKeyAndSortKey
into write/Write/WriteImpl/PreFinalize/MapToVoidKey0
root: INFO: 2019-04-12T13:37:44.440Z: JOB_MESSAGE_DETAILED: Fusing consumer
read/Read/Reshuffle/ReshufflePerKey/GroupByKey/Write into
read/Read/Reshuffle/ReshufflePerKey/GroupByKey/Reify
root: INFO: 2019-04-12T13:37:44.488Z: JOB_MESSAGE_DETAILED: Fusing consumer
read/Read/Reshuffle/ReshufflePerKey/GroupByKey/Reify into
read/Read/Reshuffle/ReshufflePerKey/Map(reify_timestamps)
root: INFO: 2019-04-12T13:37:44.518Z: JOB_MESSAGE_DETAILED: Fusing consumer
write/Write/WriteImpl/PreFinalize/_DataflowIterableSideInput(MapToVoidKey0.out.0)/GroupByKeyHashAndSortByKeyAndWindow/Write
into
write/Write/WriteImpl/PreFinalize/_DataflowIterableSideInput(MapToVoidKey0.out.0)/CreateIsmShardKeyAndSortKey
root: INFO: 2019-04-12T13:37:44.560Z: JOB_MESSAGE_DETAILED: Fusing consumer
write/Write/WriteImpl/DoOnce/Map(decode) into
write/Write/WriteImpl/DoOnce/FlatMap(<lambda at core.py:2172>)
root: INFO: 2019-04-12T13:37:44.607Z: JOB_MESSAGE_DETAILED: Fusing consumer
read/Read/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps) into
read/Read/Reshuffle/ReshufflePerKey/GroupByKey/GroupByWindow
root: INFO: 2019-04-12T13:37:44.657Z: JOB_MESSAGE_DEBUG: Workflow config is
missing a default resource spec.
root: INFO: 2019-04-12T13:37:44.713Z: JOB_MESSAGE_DEBUG: Adding StepResource
setup and teardown to workflow graph.
root: INFO: 2019-04-12T13:37:44.740Z: JOB_MESSAGE_DEBUG: Adding workflow start
and stop steps.
root: INFO: 2019-04-12T13:37:44.787Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
root: INFO: 2019-04-12T13:37:45.004Z: JOB_MESSAGE_DEBUG: Executing wait step
start120
root: INFO: 2019-04-12T13:37:45.101Z: JOB_MESSAGE_BASIC: Executing operation
write/Write/WriteImpl/WriteBundles/_DataflowIterableSideInput(MapToVoidKey0.out.0)/GroupByKeyHashAndSortByKeyAndWindow/Create
root: INFO: 2019-04-12T13:37:45.139Z: JOB_MESSAGE_BASIC: Executing operation
write/Write/WriteImpl/PreFinalize/_DataflowIterableSideInput(MapToVoidKey0.out.0)/GroupByKeyHashAndSortByKeyAndWindow/Create
root: INFO: 2019-04-12T13:37:45.151Z: JOB_MESSAGE_DEBUG: Starting worker pool
setup.
root: INFO: 2019-04-12T13:37:45.183Z: JOB_MESSAGE_BASIC: Executing operation
write/Write/WriteImpl/PreFinalize/_DataflowIterableSideInput(MapToVoidKey1.out.0)/GroupByKeyHashAndSortByKeyAndWindow/Create
root: INFO: 2019-04-12T13:37:45.201Z: JOB_MESSAGE_BASIC: Starting 1 workers in
us-central1-b...
root: INFO: 2019-04-12T13:37:45.212Z: JOB_MESSAGE_BASIC: Executing operation
write/Write/WriteImpl/FinalizeWrite/_DataflowIterableSideInput(MapToVoidKey0.out.0)/GroupByKeyHashAndSortByKeyAndWindow/Create
root: INFO: 2019-04-12T13:37:45.254Z: JOB_MESSAGE_BASIC: Executing operation
write/Write/WriteImpl/FinalizeWrite/_DataflowIterableSideInput(MapToVoidKey1.out.0)/GroupByKeyHashAndSortByKeyAndWindow/Create
root: INFO: 2019-04-12T13:37:45.291Z: JOB_MESSAGE_BASIC: Executing operation
write/Write/WriteImpl/FinalizeWrite/_DataflowIterableSideInput(MapToVoidKey2.out.0)/GroupByKeyHashAndSortByKeyAndWindow/Create
root: INFO: 2019-04-12T13:37:45.338Z: JOB_MESSAGE_BASIC: Executing operation
group/Create
root: INFO: 2019-04-12T13:37:45.384Z: JOB_MESSAGE_BASIC: Executing operation
write/Write/WriteImpl/GroupByKey/Create
root: INFO: 2019-04-12T13:37:45.426Z: JOB_MESSAGE_BASIC: Executing operation
read/Read/Reshuffle/ReshufflePerKey/GroupByKey/Create
root: INFO: 2019-04-12T13:37:45.475Z: JOB_MESSAGE_DEBUG: Value
"write/Write/WriteImpl/WriteBundles/_DataflowIterableSideInput(MapToVoidKey0.out.0)/GroupByKeyHashAndSortByKeyAndWindow/Session"
materialized.
root: INFO: 2019-04-12T13:37:45.517Z: JOB_MESSAGE_DEBUG: Value
"write/Write/WriteImpl/PreFinalize/_DataflowIterableSideInput(MapToVoidKey1.out.0)/GroupByKeyHashAndSortByKeyAndWindow/Session"
materialized.
root: INFO: 2019-04-12T13:37:45.566Z: JOB_MESSAGE_DEBUG: Value
"write/Write/WriteImpl/PreFinalize/_DataflowIterableSideInput(MapToVoidKey0.out.0)/GroupByKeyHashAndSortByKeyAndWindow/Session"
materialized.
root: INFO: 2019-04-12T13:37:45.606Z: JOB_MESSAGE_DEBUG: Value
"write/Write/WriteImpl/FinalizeWrite/_DataflowIterableSideInput(MapToVoidKey0.out.0)/GroupByKeyHashAndSortByKeyAndWindow/Session"
materialized.
root: INFO: 2019-04-12T13:37:45.653Z: JOB_MESSAGE_DEBUG: Value
"write/Write/WriteImpl/FinalizeWrite/_DataflowIterableSideInput(MapToVoidKey1.out.0)/GroupByKeyHashAndSortByKeyAndWindow/Session"
materialized.
root: INFO: 2019-04-12T13:37:45.701Z: JOB_MESSAGE_DEBUG: Value
"write/Write/WriteImpl/FinalizeWrite/_DataflowIterableSideInput(MapToVoidKey2.out.0)/GroupByKeyHashAndSortByKeyAndWindow/Session"
materialized.
root: INFO: 2019-04-12T13:37:45.744Z: JOB_MESSAGE_DEBUG: Value "group/Session"
materialized.
root: INFO: 2019-04-12T13:37:45.794Z: JOB_MESSAGE_DEBUG: Value
"write/Write/WriteImpl/GroupByKey/Session" materialized.
root: INFO: 2019-04-12T13:37:45.841Z: JOB_MESSAGE_DEBUG: Value
"read/Read/Reshuffle/ReshufflePerKey/GroupByKey/Session" materialized.
root: INFO: 2019-04-12T13:37:45.889Z: JOB_MESSAGE_BASIC: Executing operation
write/Write/WriteImpl/DoOnce/Impulse+write/Write/WriteImpl/DoOnce/FlatMap(<lambda
at
core.py:2172>)+write/Write/WriteImpl/DoOnce/Map(decode)+write/Write/WriteImpl/InitializeWrite+write/Write/WriteImpl/WriteBundles/MapToVoidKey0+write/Write/WriteImpl/PreFinalize/MapToVoidKey0+write/Write/WriteImpl/FinalizeWrite/MapToVoidKey0+write/Write/WriteImpl/WriteBundles/MapToVoidKey0+write/Write/WriteImpl/PreFinalize/MapToVoidKey0+write/Write/WriteImpl/FinalizeWrite/MapToVoidKey0+write/Write/WriteImpl/FinalizeWrite/_DataflowIterableSideInput(MapToVoidKey0.out.0)/CreateIsmShardKeyAndSortKey+write/Write/WriteImpl/FinalizeWrite/_DataflowIterableSideInput(MapToVoidKey0.out.0)/GroupByKeyHashAndSortByKeyAndWindow/Write+write/Write/WriteImpl/WriteBundles/_DataflowIterableSideInput(MapToVoidKey0.out.0)/CreateIsmShardKeyAndSortKey+write/Write/WriteImpl/WriteBundles/_DataflowIterableSideInput(MapToVoidKey0.out.0)/GroupByKeyHashAndSortByKeyAndWindow/Write+write/Write/WriteImpl/PreFinalize/_DataflowIterableSideInput(MapToVoidKey0.out.0)/CreateIsmShardKeyAndSortKey+write/Write/WriteImpl/PreFinalize/_DataflowIterableSideInput(MapToVoidKey0.out.0)/GroupByKeyHashAndSortByKeyAndWindow/Write
root: INFO: 2019-04-12T13:37:45.935Z: JOB_MESSAGE_BASIC: Executing operation
read/Read/Impulse+read/Read/Split+read/Read/Reshuffle/AddRandomKeys+read/Read/Reshuffle/ReshufflePerKey/Map(reify_timestamps)+read/Read/Reshuffle/ReshufflePerKey/GroupByKey/Reify+read/Read/Reshuffle/ReshufflePerKey/GroupByKey/Write
root: INFO: 2019-04-12T13:37:57.193Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised
the number of workers to 0 based on the rate of progress in the currently
running step(s).
root: INFO: 2019-04-12T13:38:26.007Z: JOB_MESSAGE_DETAILED: Workers have
started successfully.
root: INFO: 2019-04-12T13:38:35.553Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised
the number of workers to 1 based on the rate of progress in the currently
running step(s).
root: INFO: 2019-04-12T13:38:35.600Z: JOB_MESSAGE_DETAILED: Autoscaling: Would
further reduce the number of workers but reached the minimum number allowed for
the job.
root: INFO: 2019-04-12T13:38:57.060Z: JOB_MESSAGE_DETAILED: Workers have
started successfully.
root: INFO: 2019-04-12T13:41:21.988Z: JOB_MESSAGE_DEBUG: Value
"write/Write/WriteImpl/DoOnce/Map(decode).out" materialized.
root: INFO: 2019-04-12T13:41:22.031Z: JOB_MESSAGE_BASIC: Executing operation
write/Write/WriteImpl/WriteBundles/_DataflowIterableSideInput(MapToVoidKey0.out.0)/GroupByKeyHashAndSortByKeyAndWindow/Close
root: INFO: 2019-04-12T13:41:22.078Z: JOB_MESSAGE_BASIC: Executing operation
write/Write/WriteImpl/FinalizeWrite/_DataflowIterableSideInput(MapToVoidKey0.out.0)/GroupByKeyHashAndSortByKeyAndWindow/Close
root: INFO: 2019-04-12T13:41:22.125Z: JOB_MESSAGE_BASIC: Executing operation
write/Write/WriteImpl/PreFinalize/_DataflowIterableSideInput(MapToVoidKey0.out.0)/GroupByKeyHashAndSortByKeyAndWindow/Close
root: INFO: 2019-04-12T13:41:22.153Z: JOB_MESSAGE_BASIC: Executing operation
write/Write/WriteImpl/WriteBundles/_DataflowIterableSideInput(MapToVoidKey0.out.0)/GroupByKeyHashAndSortByKeyAndWindow/Read+write/Write/WriteImpl/WriteBundles/_DataflowIterableSideInput(MapToVoidKey0.out.0)/ToIsmRecordForMultimap
root: INFO: 2019-04-12T13:41:22.197Z: JOB_MESSAGE_BASIC: Executing operation
write/Write/WriteImpl/FinalizeWrite/_DataflowIterableSideInput(MapToVoidKey0.out.0)/GroupByKeyHashAndSortByKeyAndWindow/Read+write/Write/WriteImpl/FinalizeWrite/_DataflowIterableSideInput(MapToVoidKey0.out.0)/ToIsmRecordForMultimap
root: INFO: 2019-04-12T13:41:22.241Z: JOB_MESSAGE_BASIC: Executing operation
write/Write/WriteImpl/PreFinalize/_DataflowIterableSideInput(MapToVoidKey0.out.0)/GroupByKeyHashAndSortByKeyAndWindow/Read+write/Write/WriteImpl/PreFinalize/_DataflowIterableSideInput(MapToVoidKey0.out.0)/ToIsmRecordForMultimap
root: INFO: 2019-04-12T13:41:23.885Z: JOB_MESSAGE_BASIC: Executing operation
read/Read/Reshuffle/ReshufflePerKey/GroupByKey/Close
root: INFO: 2019-04-12T13:41:23.979Z: JOB_MESSAGE_BASIC: Executing operation
read/Read/Reshuffle/ReshufflePerKey/GroupByKey/Read+read/Read/Reshuffle/ReshufflePerKey/GroupByKey/GroupByWindow+read/Read/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps)+read/Read/Reshuffle/RemoveRandomKeys+read/Read/ReadSplits+split+pair_with_one+group/Reify+group/Write
root: INFO: 2019-04-12T13:41:37.681Z: JOB_MESSAGE_BASIC: Executing operation
group/Close
root: INFO: 2019-04-12T13:41:46.448Z: JOB_MESSAGE_DEBUG: Value
"write/Write/WriteImpl/FinalizeWrite/_DataflowIterableSideInput(MapToVoidKey0.out.0)/ToIsmRecordForMultimap.out0"
materialized.
root: INFO: 2019-04-12T13:41:46.518Z: JOB_MESSAGE_BASIC: Executing operation
write/Write/WriteImpl/FinalizeWrite/_DataflowIterableSideInput(MapToVoidKey0.out.0)/Materialize
root: INFO: 2019-04-12T13:41:46.632Z: JOB_MESSAGE_DEBUG: Value
"write/Write/WriteImpl/FinalizeWrite/_DataflowIterableSideInput(MapToVoidKey0.out.0)/Materialize.out0"
materialized.
root: INFO: 2019-04-12T13:41:52.972Z: JOB_MESSAGE_DEBUG: Value
"write/Write/WriteImpl/WriteBundles/_DataflowIterableSideInput(MapToVoidKey0.out.0)/ToIsmRecordForMultimap.out0"
materialized.
root: INFO: 2019-04-12T13:41:53.058Z: JOB_MESSAGE_BASIC: Executing operation
write/Write/WriteImpl/WriteBundles/_DataflowIterableSideInput(MapToVoidKey0.out.0)/Materialize
root: INFO: 2019-04-12T13:41:53.156Z: JOB_MESSAGE_DEBUG: Value
"write/Write/WriteImpl/WriteBundles/_DataflowIterableSideInput(MapToVoidKey0.out.0)/Materialize.out0"
materialized.
root: INFO: 2019-04-12T13:41:53.254Z: JOB_MESSAGE_BASIC: Executing operation
group/Read+group/GroupByWindow+count+format+write/Write/WriteImpl/WriteBundles/WriteBundles+write/Write/WriteImpl/Pair+write/Write/WriteImpl/WindowInto(WindowIntoFn)+write/Write/WriteImpl/GroupByKey/Reify+write/Write/WriteImpl/GroupByKey/Write
root: INFO: 2019-04-12T13:41:56.606Z: JOB_MESSAGE_ERROR:
java.lang.IllegalArgumentException: This handler is only capable of dealing
with urn:beam:sideinput:materialization:multimap:0.1 materializations but was
asked to handle beam:side_input:multimap:v1 for PCollectionView with tag
side0-write/Write/WriteImpl/WriteBundles.
at
org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument(Preconditions.java:399)
at
org.apache.beam.runners.dataflow.worker.graph.RegisterNodeFunction.transformSideInputForRunner(RegisterNodeFunction.java:506)
at
org.apache.beam.runners.dataflow.worker.graph.RegisterNodeFunction.apply(RegisterNodeFunction.java:327)
at
org.apache.beam.runners.dataflow.worker.graph.RegisterNodeFunction.apply(RegisterNodeFunction.java:97)
at java.util.function.Function.lambda$andThen$1(Function.java:88)
at
org.apache.beam.runners.dataflow.worker.graph.CreateRegisterFnOperationFunction.apply(CreateRegisterFnOperationFunction.java:208)
at
org.apache.beam.runners.dataflow.worker.graph.CreateRegisterFnOperationFunction.apply(CreateRegisterFnOperationFunction.java:75)
at java.util.function.Function.lambda$andThen$1(Function.java:88)
at java.util.function.Function.lambda$andThen$1(Function.java:88)
at
org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:347)
at
org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:306)
at
org.apache.beam.runners.dataflow.worker.DataflowRunnerHarness.start(DataflowRunnerHarness.java:195)
at
org.apache.beam.runners.dataflow.worker.DataflowRunnerHarness.main(DataflowRunnerHarness.java:123)
root: INFO: 2019-04-12T13:41:58.744Z: JOB_MESSAGE_ERROR:
java.lang.IllegalArgumentException: This handler is only capable of dealing
with urn:beam:sideinput:materialization:multimap:0.1 materializations but was
asked to handle beam:side_input:multimap:v1 for PCollectionView with tag
side0-write/Write/WriteImpl/WriteBundles.
at
org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument(Preconditions.java:399)
at
org.apache.beam.runners.dataflow.worker.graph.RegisterNodeFunction.transformSideInputForRunner(RegisterNodeFunction.java:506)
at
org.apache.beam.runners.dataflow.worker.graph.RegisterNodeFunction.apply(RegisterNodeFunction.java:327)
at
org.apache.beam.runners.dataflow.worker.graph.RegisterNodeFunction.apply(RegisterNodeFunction.java:97)
at java.util.function.Function.lambda$andThen$1(Function.java:88)
at
org.apache.beam.runners.dataflow.worker.graph.CreateRegisterFnOperationFunction.apply(CreateRegisterFnOperationFunction.java:208)
at
org.apache.beam.runners.dataflow.worker.graph.CreateRegisterFnOperationFunction.apply(CreateRegisterFnOperationFunction.java:75)
at java.util.function.Function.lambda$andThen$1(Function.java:88)
at java.util.function.Function.lambda$andThen$1(Function.java:88)
at
org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:347)
at
org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:306)
at
org.apache.beam.runners.dataflow.worker.DataflowRunnerHarness.start(DataflowRunnerHarness.java:195)
at
org.apache.beam.runners.dataflow.worker.DataflowRunnerHarness.main(DataflowRunnerHarness.java:123)
root: INFO: 2019-04-12T13:41:59.862Z: JOB_MESSAGE_ERROR:
java.lang.IllegalArgumentException: This handler is only capable of dealing
with urn:beam:sideinput:materialization:multimap:0.1 materializations but was
asked to handle beam:side_input:multimap:v1 for PCollectionView with tag
side0-write/Write/WriteImpl/WriteBundles.
at
org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument(Preconditions.java:399)
at
org.apache.beam.runners.dataflow.worker.graph.RegisterNodeFunction.transformSideInputForRunner(RegisterNodeFunction.java:506)
at
org.apache.beam.runners.dataflow.worker.graph.RegisterNodeFunction.apply(RegisterNodeFunction.java:327)
at
org.apache.beam.runners.dataflow.worker.graph.RegisterNodeFunction.apply(RegisterNodeFunction.java:97)
at java.util.function.Function.lambda$andThen$1(Function.java:88)
at
org.apache.beam.runners.dataflow.worker.graph.CreateRegisterFnOperationFunction.apply(CreateRegisterFnOperationFunction.java:208)
at
org.apache.beam.runners.dataflow.worker.graph.CreateRegisterFnOperationFunction.apply(CreateRegisterFnOperationFunction.java:75)
at java.util.function.Function.lambda$andThen$1(Function.java:88)
at java.util.function.Function.lambda$andThen$1(Function.java:88)
at
org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:347)
at
org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:306)
at
org.apache.beam.runners.dataflow.worker.DataflowRunnerHarness.start(DataflowRunnerHarness.java:195)
at
org.apache.beam.runners.dataflow.worker.DataflowRunnerHarness.main(DataflowRunnerHarness.java:123)
root: INFO: 2019-04-12T13:42:00.989Z: JOB_MESSAGE_ERROR:
java.lang.IllegalArgumentException: This handler is only capable of dealing
with urn:beam:sideinput:materialization:multimap:0.1 materializations but was
asked to handle beam:side_input:multimap:v1 for PCollectionView with tag
side0-write/Write/WriteImpl/WriteBundles.
at
org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument(Preconditions.java:399)
at
org.apache.beam.runners.dataflow.worker.graph.RegisterNodeFunction.transformSideInputForRunner(RegisterNodeFunction.java:506)
at
org.apache.beam.runners.dataflow.worker.graph.RegisterNodeFunction.apply(RegisterNodeFunction.java:327)
at
org.apache.beam.runners.dataflow.worker.graph.RegisterNodeFunction.apply(RegisterNodeFunction.java:97)
at java.util.function.Function.lambda$andThen$1(Function.java:88)
at
org.apache.beam.runners.dataflow.worker.graph.CreateRegisterFnOperationFunction.apply(CreateRegisterFnOperationFunction.java:208)
at
org.apache.beam.runners.dataflow.worker.graph.CreateRegisterFnOperationFunction.apply(CreateRegisterFnOperationFunction.java:75)
at java.util.function.Function.lambda$andThen$1(Function.java:88)
at java.util.function.Function.lambda$andThen$1(Function.java:88)
at
org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:347)
at
org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:306)
at
org.apache.beam.runners.dataflow.worker.DataflowRunnerHarness.start(DataflowRunnerHarness.java:195)
at
org.apache.beam.runners.dataflow.worker.DataflowRunnerHarness.main(DataflowRunnerHarness.java:123)
root: INFO: 2019-04-12T13:42:01.047Z: JOB_MESSAGE_DEBUG: Executing failure step
failure119
root: INFO: 2019-04-12T13:42:01.103Z: JOB_MESSAGE_ERROR: Workflow failed.
Causes:
S20:group/Read+group/GroupByWindow+count+format+write/Write/WriteImpl/WriteBundles/WriteBundles+write/Write/WriteImpl/Pair+write/Write/WriteImpl/WindowInto(WindowIntoFn)+write/Write/WriteImpl/GroupByKey/Reify+write/Write/WriteImpl/GroupByKey/Write
failed., A work item was attempted 4 times without success. Each time the
worker eventually lost contact with the service. The work item was attempted
on:
beamapp-jenkins-041213373-04120637-rgsl-harness-mvd6,
beamapp-jenkins-041213373-04120637-rgsl-harness-mvd6,
beamapp-jenkins-041213373-04120637-rgsl-harness-mvd6,
beamapp-jenkins-041213373-04120637-rgsl-harness-mvd6
root: INFO: 2019-04-12T13:42:01.301Z: JOB_MESSAGE_DETAILED: Cleaning up.
root: INFO: 2019-04-12T13:42:01.693Z: JOB_MESSAGE_DEBUG: Starting worker pool
teardown.
root: INFO: 2019-04-12T13:42:01.737Z: JOB_MESSAGE_BASIC: Stopping worker pool...
root: INFO: 2019-04-12T13:43:40.506Z: JOB_MESSAGE_DETAILED: Autoscaling:
Reduced the number of workers to 0 based on the rate of progress in the
currently running step(s).
root: INFO: 2019-04-12T13:43:40.554Z: JOB_MESSAGE_BASIC: Worker pool stopped.
root: INFO: 2019-04-12T13:43:40.600Z: JOB_MESSAGE_DEBUG: Tearing down pending
resources...
root: INFO: Job 2019-04-12_06_37_37-3825886325144229012 is in state
JOB_STATE_FAILED
--------------------- >> end captured logging << ---------------------
----------------------------------------------------------------------
XML:
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/nosetests.xml>
----------------------------------------------------------------------
Ran 34 tests in 3368.005s
FAILED (SKIP=1, errors=2)
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-04-12_06_29_28-3869843636575664555?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-04-12_06_44_54-10972290681193205558?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-04-12_06_52_57-17893954555689082920?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-04-12_06_29_19-2055797310439501395?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-04-12_06_51_20-11774921032435053968?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-04-12_06_29_26-6481832782687299781?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-04-12_06_42_39-2276081416852307920?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-04-12_06_50_55-17290013332180616669?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-04-12_06_29_21-2590636554618760178?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-04-12_06_47_19-7860883682887714197?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-04-12_06_54_51-11160948418655198913?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-04-12_06_29_19-15618963448592485771?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-04-12_06_37_37-3825886325144229012?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-04-12_06_44_00-7105441621596942447?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-04-12_06_52_17-7965511292241692840?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-04-12_06_58_55-918672728942100472?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-04-12_06_29_18-14246337652071497550?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-04-12_06_36_35-5567917242472926604?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-04-12_06_45_12-5674834788132103608?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-04-12_06_52_29-14740857881088092075?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-04-12_07_00_19-2962286773282130605?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-04-12_07_09_04-7009857319109823001?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-04-12_07_17_12-14992238505611808173?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-04-12_06_29_19-8955487910613127020?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-04-12_06_37_47-17904147272831626109?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-04-12_06_40_41-1826921772470364102?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-04-12_06_47_48-16919822412340629674?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-04-12_06_55_12-13165826499690722312?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-04-12_06_29_19-11217613619256072897?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-04-12_06_38_48-886017693501556942?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-04-12_06_48_45-17726567207598275353?project=apache-beam-testing.
Found:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-04-12_06_56_57-17939619763123965518?project=apache-beam-testing.
> Task :beam-sdks-python:postCommitIT FAILED
FAILURE: Build completed with 2 failures.
1: Task failed with an exception.
-----------
* Where:
Build file
'<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/build.gradle'>
line: 127
* What went wrong:
Execution failed for task ':beam-sdks-python:directRunnerIT'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug
option to get more log output. Run with --scan to get full insights.
==============================================================================
2: Task failed with an exception.
-----------
* Where:
Build file
'<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/src/sdks/python/build.gradle'>
line: 229
* What went wrong:
Execution failed for task ':beam-sdks-python:postCommitIT'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug
option to get more log output. Run with --scan to get full insights.
==============================================================================
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with
Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See
https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 1h 0m 5s
6 actionable tasks: 6 executed
Publishing build scan...
https://gradle.com/s/43zbe3pzwz2kq
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
