See
<https://builds.apache.org/job/beam_PerformanceTests_TFRecordIOIT/2021/display/redirect>
------------------------------------------
[...truncated 657.62 KB...]
Jun 02, 2019 7:12:15 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding
PAssert$0/GroupGlobally/RemoveDummyTriggering/Flatten.PCollections as step s29
Jun 02, 2019 7:12:15 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$0/GroupGlobally/FlattenDummyAndContents as step s30
Jun 02, 2019 7:12:15 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$0/GroupGlobally/NeverTrigger/Flatten.PCollections as
step s31
Jun 02, 2019 7:12:15 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$0/GroupGlobally/GroupDummyAndContents as step s32
Jun 02, 2019 7:12:15 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$0/GroupGlobally/Values/Values/Map as step s33
Jun 02, 2019 7:12:15 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$0/GroupGlobally/ParDo(Concat) as step s34
Jun 02, 2019 7:12:15 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$0/GetPane/Map as step s35
Jun 02, 2019 7:12:15 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$0/RunChecks as step s36
Jun 02, 2019 7:12:15 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$0/VerifyAssertions/ParDo(DefaultConclude) as step s37
Jun 02, 2019 7:12:15 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Create.Values/Read(CreateSource) as step s38
Jun 02, 2019 7:12:15 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding
View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map
as step s39
Jun 02, 2019 7:12:15 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding
View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey
as step s40
Jun 02, 2019 7:12:15 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding
View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues
as step s41
Jun 02, 2019 7:12:15 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding
View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map
as step s42
Jun 02, 2019 7:12:15 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding
View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
as step s43
Jun 02, 2019 7:12:15 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding
View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly
as step s44
Jun 02, 2019 7:12:15 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding
View.AsSingleton/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow)
as step s45
Jun 02, 2019 7:12:15 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding
View.AsSingleton/Combine.GloballyAsSingletonView/CreateDataflowView as step s46
Jun 02, 2019 7:12:15 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Delete test files as step s47
Jun 02, 2019 7:12:15 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to
gs://temp-storage-for-perf-tests/tfrecordioit0writethenreadall-jenkins-0602191056-330d3679/output/results/staging/
Jun 02, 2019 7:12:15 PM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage
INFO: Uploading <104173 bytes, hash TjXHICv_aucuE9c-tezObA> to
gs://temp-storage-for-perf-tests/tfrecordioit0writethenreadall-jenkins-0602191056-330d3679/output/results/staging/pipeline-TjXHICv_aucuE9c-tezObA.pb
org.apache.beam.sdk.io.tfrecord.TFRecordIOIT > writeThenReadAll STANDARD_OUT
Dataflow SDK version: 2.14.0-SNAPSHOT
org.apache.beam.sdk.io.tfrecord.TFRecordIOIT > writeThenReadAll STANDARD_ERROR
Jun 02, 2019 7:12:18 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-06-02_12_12_17-1749624827998496452?project=apache-beam-testing
org.apache.beam.sdk.io.tfrecord.TFRecordIOIT > writeThenReadAll STANDARD_OUT
Submitted job: 2019-06-02_12_12_17-1749624827998496452
org.apache.beam.sdk.io.tfrecord.TFRecordIOIT > writeThenReadAll STANDARD_ERROR
Jun 02, 2019 7:12:18 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel
--region=us-central1 2019-06-02_12_12_17-1749624827998496452
Jun 02, 2019 7:12:18 PM org.apache.beam.runners.dataflow.TestDataflowRunner
run
INFO: Running Dataflow job 2019-06-02_12_12_17-1749624827998496452 with 1
expected assertions.
Jun 02, 2019 7:12:18 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:17.152Z: Autoscaling is enabled for job
2019-06-02_12_12_17-1749624827998496452. The number of workers will be between
1 and 1000.
Jun 02, 2019 7:12:18 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:17.267Z: Autoscaling was automatically enabled for
job 2019-06-02_12_12_17-1749624827998496452.
Jun 02, 2019 7:12:23 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:20.836Z: Checking permissions granted to controller
Service Account.
Jun 02, 2019 7:12:42 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:42.753Z: Worker configuration: n1-standard-1 in
us-central1-c.
Jun 02, 2019 7:12:47 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:46.458Z: Expanding CoGroupByKey operations into
optimizable parts.
Jun 02, 2019 7:12:47 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:47.047Z: Expanding GroupByKey operations into
optimizable parts.
Jun 02, 2019 7:12:49 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:47.162Z: Lifting ValueCombiningMappingFns into
MergeBucketsMappingFns
Jun 02, 2019 7:12:49 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:48.008Z: Fusing adjacent ParDo, Read, Write, and
Flatten operations
Jun 02, 2019 7:12:49 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:48.121Z: Elided trivial flatten
Jun 02, 2019 7:12:49 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:48.229Z: Elided trivial flatten
Jun 02, 2019 7:12:49 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:48.348Z: Elided trivial flatten
Jun 02, 2019 7:12:49 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:48.464Z: Fusing consumer
PAssert$0/GroupGlobally/Window.Into()/Window.Assign into
Reshuffle.ViaRandomKey/Values/Values/Map
Jun 02, 2019 7:12:49 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:48.577Z: Fusing consumer
View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map
into Reshuffle.ViaRandomKey/Values/Values/Map
Jun 02, 2019 7:12:49 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:48.700Z: Fusing consumer
PAssert$0/GroupGlobally/GroupDummyAndContents/Write into
PAssert$0/GroupGlobally/GroupDummyAndContents/Reify
Jun 02, 2019 7:12:49 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:48.826Z: Fusing consumer
PAssert$0/GroupGlobally/ParDo(Concat) into
PAssert$0/GroupGlobally/Values/Values/Map
Jun 02, 2019 7:12:49 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:48.946Z: Fusing consumer PAssert$0/GetPane/Map into
PAssert$0/GroupGlobally/ParDo(Concat)
Jun 02, 2019 7:12:49 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:49.075Z: Fusing consumer PAssert$0/RunChecks into
PAssert$0/GetPane/Map
Jun 02, 2019 7:12:49 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:49.194Z: Fusing consumer
PAssert$0/VerifyAssertions/ParDo(DefaultConclude) into PAssert$0/RunChecks
Jun 02, 2019 7:12:49 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:49.305Z: Fusing consumer
PAssert$0/GroupGlobally/GroupDummyAndContents/GroupByWindow into
PAssert$0/GroupGlobally/GroupDummyAndContents/Read
Jun 02, 2019 7:12:49 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:49.429Z: Fusing consumer
PAssert$0/GroupGlobally/Values/Values/Map into
PAssert$0/GroupGlobally/GroupDummyAndContents/GroupByWindow
Jun 02, 2019 7:12:51 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:49.556Z: Unzipping flatten s30 for input
s25.org.apache.beam.sdk.values.PCollection.<init>:402#710158705210b615
Jun 02, 2019 7:12:51 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:49.678Z: Fusing unzipped copy of
PAssert$0/GroupGlobally/GroupDummyAndContents/Reify, through flatten
PAssert$0/GroupGlobally/FlattenDummyAndContents, into producer
PAssert$0/GroupGlobally/KeyForDummy/AddKeys/Map
Jun 02, 2019 7:12:51 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:49.787Z: Fusing consumer
Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign into
Reshuffle.ViaRandomKey/Pair with random key
Jun 02, 2019 7:12:51 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:49.899Z: Fusing consumer
PAssert$0/GroupGlobally/GatherAllOutputs/GroupByKey/Write into
PAssert$0/GroupGlobally/GatherAllOutputs/GroupByKey/Reify
Jun 02, 2019 7:12:51 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:50.019Z: Fusing consumer
PAssert$0/GroupGlobally/GatherAllOutputs/Values/Values/Map into
PAssert$0/GroupGlobally/GatherAllOutputs/GroupByKey/GroupByWindow
Jun 02, 2019 7:12:51 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:50.132Z: Fusing consumer
PAssert$0/GroupGlobally/RewindowActuals/Window.Assign into
PAssert$0/GroupGlobally/GatherAllOutputs/Values/Values/Map
Jun 02, 2019 7:12:51 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:50.254Z: Fusing consumer
View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
into
View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
Jun 02, 2019 7:12:51 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:50.381Z: Fusing consumer
PAssert$0/GroupGlobally/GatherAllOutputs/GroupByKey/GroupByWindow into
PAssert$0/GroupGlobally/GatherAllOutputs/GroupByKey/Read
Jun 02, 2019 7:12:51 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:50.507Z: Fusing consumer
PAssert$0/GroupGlobally/GatherAllOutputs/Reify.Window/ParDo(Anonymous) into
PAssert$0/GroupGlobally/Window.Into()/Window.Assign
Jun 02, 2019 7:12:51 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:50.635Z: Fusing consumer
PAssert$0/GroupGlobally/GatherAllOutputs/Window.Into()/Window.Assign into
PAssert$0/GroupGlobally/GatherAllOutputs/WithKeys/AddKeys/Map
Jun 02, 2019 7:12:51 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:50.746Z: Fusing consumer
PAssert$0/GroupGlobally/GatherAllOutputs/GroupByKey/Reify into
PAssert$0/GroupGlobally/GatherAllOutputs/Window.Into()/Window.Assign
Jun 02, 2019 7:12:51 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:50.868Z: Fusing consumer
View.AsSingleton/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow)
into
View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
Jun 02, 2019 7:12:53 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:50.974Z: Fusing consumer
Reshuffle.ViaRandomKey/Values/Values/Map into
Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable
Jun 02, 2019 7:12:53 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:51.089Z: Fusing consumer
PAssert$0/GroupGlobally/GatherAllOutputs/WithKeys/AddKeys/Map into
PAssert$0/GroupGlobally/GatherAllOutputs/Reify.Window/ParDo(Anonymous)
Jun 02, 2019 7:12:53 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:51.191Z: Fusing consumer
View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
into
View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map
Jun 02, 2019 7:12:53 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:51.318Z: Fusing consumer
View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Extract
into
View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues
Jun 02, 2019 7:12:53 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:51.441Z: Fusing consumer
Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow into
Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read
Jun 02, 2019 7:12:53 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:51.568Z: Fusing consumer
View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey+View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Partial
into
View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map
Jun 02, 2019 7:12:53 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:51.691Z: Fusing consumer
View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Reify
into
View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey+View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Partial
Jun 02, 2019 7:12:53 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:51.811Z: Fusing consumer
View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues
into
View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Read
Jun 02, 2019 7:12:53 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:51.920Z: Fusing consumer
View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Write
into
View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Reify
Jun 02, 2019 7:12:53 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:52.034Z: Fusing consumer
Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify into
Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign
Jun 02, 2019 7:12:53 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:52.155Z: Unzipping flatten s12 for input
s11.org.apache.beam.sdk.values.PCollection.<init>:402#dabfabce15317cd8
Jun 02, 2019 7:12:53 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:52.273Z: Fusing unzipped copy of
Reshuffle.ViaRandomKey/Pair with random key, through flatten Calculate
hashcode/Flatten.PCollections, into producer Calculate hashcode/ProduceDefault
Jun 02, 2019 7:12:53 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:52.396Z: Fusing consumer
Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable into
Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow
Jun 02, 2019 7:12:53 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:52.523Z: Unzipping flatten s12-u80 for input
s15-reify-value18-c78
Jun 02, 2019 7:12:53 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:52.649Z: Fusing unzipped copy of
Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write, through flatten Calculate
hashcode/Flatten.PCollections/Unzipped-1, into producer
Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify
Jun 02, 2019 7:12:53 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:52.765Z: Fusing consumer
View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map
into
View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Extract
Jun 02, 2019 7:12:53 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:52.888Z: Fusing consumer
PAssert$0/GroupGlobally/KeyForDummy/AddKeys/Map into
PAssert$0/GroupGlobally/RewindowActuals/Window.Assign
Jun 02, 2019 7:12:53 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:53.015Z: Fusing consumer Reshuffle.ViaRandomKey/Pair
with random key into Calculate hashcode/Values/Values/Map
Jun 02, 2019 7:12:53 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:53.116Z: Fusing consumer Calculate
hashcode/View.AsIterable/ParDo(ToIsmRecordForGlobalWindow) into Calculate
hashcode/Values/Values/Map
Jun 02, 2019 7:12:53 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:53.223Z: Fusing consumer
Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write into
Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify
Jun 02, 2019 7:12:53 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:53.342Z: Fusing consumer Calculate
hashcode/ProduceDefault into Calculate hashcode/CreateVoid/Read(CreateSource)
Jun 02, 2019 7:12:53 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:53.451Z: Fusing consumer Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey+Calculate
hashcode/Combine.perKey(Hashing)/Combine.GroupedValues/Partial into Calculate
hashcode/WithKeys/AddKeys/Map
Jun 02, 2019 7:12:53 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:53.561Z: Fusing consumer Calculate
hashcode/Combine.perKey(Hashing)/Combine.GroupedValues/Extract into Calculate
hashcode/Combine.perKey(Hashing)/Combine.GroupedValues
Jun 02, 2019 7:12:54 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:53.675Z: Fusing consumer Transform bytes to
strings/Map into Record time after reading
Jun 02, 2019 7:12:54 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:53.798Z: Fusing consumer Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Reify into Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey+Calculate
hashcode/Combine.perKey(Hashing)/Combine.GroupedValues/Partial
Jun 02, 2019 7:12:54 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:53.919Z: Fusing consumer Calculate
hashcode/Combine.perKey(Hashing)/Combine.GroupedValues into Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Read
Jun 02, 2019 7:12:54 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:54.037Z: Fusing consumer Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Write into Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Reify
Jun 02, 2019 7:12:54 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:54.146Z: Fusing consumer Record time after reading
into TFRecordIO.Read/Read
Jun 02, 2019 7:12:54 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:54.267Z: Fusing consumer Calculate
hashcode/Values/Values/Map into Calculate
hashcode/Combine.perKey(Hashing)/Combine.GroupedValues/Extract
Jun 02, 2019 7:12:54 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:54.394Z: Fusing consumer Calculate
hashcode/WithKeys/AddKeys/Map into Transform bytes to strings/Map
Jun 02, 2019 7:12:54 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:54.516Z: Fusing consumer Delete test files into
Create.Values/Read(CreateSource)
Jun 02, 2019 7:12:54 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:54.640Z: Fusing consumer
PAssert$0/GroupGlobally/GroupDummyAndContents/Reify into
PAssert$0/GroupGlobally/WindowIntoDummy/Window.Assign
Jun 02, 2019 7:12:54 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:54.763Z: Fusing consumer
PAssert$0/GroupGlobally/WindowIntoDummy/Window.Assign into
PAssert$0/GroupGlobally/Create.Values/Read(CreateSource)
Jun 02, 2019 7:12:57 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:56.032Z: Executing operation
Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Create
Jun 02, 2019 7:12:57 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:56.153Z: Executing operation
PAssert$0/GroupGlobally/GatherAllOutputs/GroupByKey/Create
Jun 02, 2019 7:12:57 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:56.271Z: Starting 1 workers in us-central1-c...
Jun 02, 2019 7:12:57 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:56.271Z: Executing operation
PAssert$0/GroupGlobally/GroupDummyAndContents/Create
Jun 02, 2019 7:12:57 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:56.399Z: Executing operation
View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Create
Jun 02, 2019 7:12:57 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:56.526Z: Executing operation
View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Create
Jun 02, 2019 7:12:57 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:56.648Z: Executing operation Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Create
Jun 02, 2019 7:12:58 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:57.494Z: Executing operation
PAssert$0/GroupGlobally/Create.Values/Read(CreateSource)+PAssert$0/GroupGlobally/WindowIntoDummy/Window.Assign+PAssert$0/GroupGlobally/GroupDummyAndContents/Reify+PAssert$0/GroupGlobally/GroupDummyAndContents/Write
Jun 02, 2019 7:12:58 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:12:57.624Z: Executing operation
TFRecordIO.Read/Read+Record time after reading+Transform bytes to
strings/Map+Calculate hashcode/WithKeys/AddKeys/Map+Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey+Calculate
hashcode/Combine.perKey(Hashing)/Combine.GroupedValues/Partial+Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Reify+Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Write
Jun 02, 2019 7:13:56 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:13:54.668Z: Workers have started successfully.
Jun 02, 2019 7:14:28 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-02T19:14:28.125Z: Autoscaling: Raised the number of workers
to 1 based on the rate of progress in the currently running step(s).
STDERR:
2019-06-02 19:14:44,679 86182668 MainThread beam_integration_benchmark(1/1)
ERROR Error during benchmark beam_integration_benchmark
Traceback (most recent call last):
File
"<https://builds.apache.org/job/beam_PerformanceTests_TFRecordIOIT/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py";,>
line 760, in RunBenchmark
DoRunPhase(spec, collector, detailed_timer)
File
"<https://builds.apache.org/job/beam_PerformanceTests_TFRecordIOIT/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py";,>
line 609, in DoRunPhase
samples = spec.BenchmarkRun(spec)
File
"<https://builds.apache.org/job/beam_PerformanceTests_TFRecordIOIT/ws/PerfKitBenchmarker/perfkitbenchmarker/linux_benchmarks/beam_integration_benchmark.py";,>
line 160, in Run
job_type=job_type)
File
"<https://builds.apache.org/job/beam_PerformanceTests_TFRecordIOIT/ws/PerfKitBenchmarker/perfkitbenchmarker/providers/gcp/gcp_dpb_dataflow.py";,>
line 90, in SubmitJob
assert retcode == 0, "Integration Test Failed."
AssertionError: Integration Test Failed.
2019-06-02 19:14:44,682 86182668 MainThread beam_integration_benchmark(1/1)
INFO Cleaning up benchmark beam_integration_benchmark
2019-06-02 19:14:44,688 86182668 MainThread beam_integration_benchmark(1/1)
ERROR Exception running benchmark
Traceback (most recent call last):
File
"<https://builds.apache.org/job/beam_PerformanceTests_TFRecordIOIT/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py";,>
line 903, in RunBenchmarkTask
RunBenchmark(spec, collector)
File
"<https://builds.apache.org/job/beam_PerformanceTests_TFRecordIOIT/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py";,>
line 760, in RunBenchmark
DoRunPhase(spec, collector, detailed_timer)
File
"<https://builds.apache.org/job/beam_PerformanceTests_TFRecordIOIT/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py";,>
line 609, in DoRunPhase
samples = spec.BenchmarkRun(spec)
File
"<https://builds.apache.org/job/beam_PerformanceTests_TFRecordIOIT/ws/PerfKitBenchmarker/perfkitbenchmarker/linux_benchmarks/beam_integration_benchmark.py";,>
line 160, in Run
job_type=job_type)
File
"<https://builds.apache.org/job/beam_PerformanceTests_TFRecordIOIT/ws/PerfKitBenchmarker/perfkitbenchmarker/providers/gcp/gcp_dpb_dataflow.py";,>
line 90, in SubmitJob
assert retcode == 0, "Integration Test Failed."
AssertionError: Integration Test Failed.
2019-06-02 19:14:44,689 86182668 MainThread beam_integration_benchmark(1/1)
ERROR Benchmark 1/1 beam_integration_benchmark (UID:
beam_integration_benchmark0) failed. Execution will continue.
2019-06-02 19:14:44,690 86182668 MainThread beam_integration_benchmark(1/1)
INFO Benchmark run statuses:
---------------------------------------------------------------------------------
Name UID Status Failed
Substatus
---------------------------------------------------------------------------------
beam_integration_benchmark beam_integration_benchmark0 FAILED
---------------------------------------------------------------------------------
Success rate: 0.00% (0/1)
2019-06-02 19:14:44,690 86182668 MainThread beam_integration_benchmark(1/1)
INFO Complete logs can be found at:
<https://builds.apache.org/job/beam_PerformanceTests_TFRecordIOIT/ws/runs/86182668/pkb.log>
2019-06-02 19:14:44,690 86182668 MainThread beam_integration_benchmark(1/1)
INFO Completion statuses can be found at:
<https://builds.apache.org/job/beam_PerformanceTests_TFRecordIOIT/ws/runs/86182668/completion_statuses.json>
Build step 'Execute shell' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
