See
<https://builds.apache.org/job/beam_PerformanceTests_HadoopInputFormat/976/display/redirect?page=changes>
Changes:
[rangadi] Deprecate KafkaIO withTimestampFn().
[amaliujia] [BEAM-6006] Test TIME +/- INTERVAL
------------------------------------------
[...truncated 440.84 KB...]
INFO: Adding PAssert$0/CreateActual/GatherPanes/Values/Values/Map as step
s18
Nov 07, 2018 6:38:28 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$0/CreateActual/ExtractPane/Map as step s19
Nov 07, 2018 6:38:28 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding
PAssert$0/CreateActual/Flatten.Iterables/FlattenIterables/FlatMap as step s20
Nov 07, 2018 6:38:28 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$0/CreateActual/RewindowActuals/Window.Assign as step
s21
Nov 07, 2018 6:38:28 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$0/CreateActual/ParDo(Anonymous) as step s22
Nov 07, 2018 6:38:28 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding
PAssert$0/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map
as step s23
Nov 07, 2018 6:38:28 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding
PAssert$0/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey
as step s24
Nov 07, 2018 6:38:28 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding
PAssert$0/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues
as step s25
Nov 07, 2018 6:38:28 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding
PAssert$0/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map
as step s26
Nov 07, 2018 6:38:28 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding
PAssert$0/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
as step s27
Nov 07, 2018 6:38:28 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding
PAssert$0/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly
as step s28
Nov 07, 2018 6:38:28 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding
PAssert$0/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow)
as step s29
Nov 07, 2018 6:38:28 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding
PAssert$0/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/CreateDataflowView
as step s30
Nov 07, 2018 6:38:28 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$0/Create.Values/Read(CreateSource) as step s31
Nov 07, 2018 6:38:28 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$0/WindowToken/Window.Assign as step s32
Nov 07, 2018 6:38:28 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$0/RunChecks as step s33
Nov 07, 2018 6:38:28 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$0/VerifyAssertions/ParDo(DefaultConclude) as step s34
Nov 07, 2018 6:38:28 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to
gs://temp-storage-for-perf-tests/hadoopinputformatioit0readusinghadoopinputformat-jenkins-1107183822-b4aa250b/output/results/staging/
Nov 07, 2018 6:38:28 PM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage
INFO: Uploading <91801 bytes, hash 00Rwzs4qGn8mDf_GaSzzOA> to
gs://temp-storage-for-perf-tests/hadoopinputformatioit0readusinghadoopinputformat-jenkins-1107183822-b4aa250b/output/results/staging/pipeline-00Rwzs4qGn8mDf_GaSzzOA.pb
org.apache.beam.sdk.io.hadoop.inputformat.HadoopInputFormatIOIT >
readUsingHadoopInputFormat STANDARD_OUT
Dataflow SDK version: 2.9.0-SNAPSHOT
org.apache.beam.sdk.io.hadoop.inputformat.HadoopInputFormatIOIT >
readUsingHadoopInputFormat STANDARD_ERROR
Nov 07, 2018 6:38:32 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-11-07_10_38_29-15219366204489868810?project=apache-beam-testing
org.apache.beam.sdk.io.hadoop.inputformat.HadoopInputFormatIOIT >
readUsingHadoopInputFormat STANDARD_OUT
Submitted job: 2018-11-07_10_38_29-15219366204489868810
org.apache.beam.sdk.io.hadoop.inputformat.HadoopInputFormatIOIT >
readUsingHadoopInputFormat STANDARD_ERROR
Nov 07, 2018 6:38:32 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel
--region=us-central1 2018-11-07_10_38_29-15219366204489868810
Nov 07, 2018 6:38:32 PM org.apache.beam.runners.dataflow.TestDataflowRunner
run
INFO: Running Dataflow job 2018-11-07_10_38_29-15219366204489868810 with 1
expected assertions.
Nov 07, 2018 6:38:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:38:29.312Z: Autoscaling is enabled for job
2018-11-07_10_38_29-15219366204489868810. The number of workers will be between
1 and 1000.
Nov 07, 2018 6:38:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:38:29.352Z: Autoscaling was automatically enabled for
job 2018-11-07_10_38_29-15219366204489868810.
Nov 07, 2018 6:38:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
WARNING: 2018-11-07T18:38:33.725Z: The workflow name is not a valid Cloud
Label. Labels applied to Cloud resources (such as GCE Instances) for monitoring
will be labeled with this modified job name:
hadoopinputformatioit0readusinghadoopinputformat-jenkins-1-iz1p. For the best
monitoring experience, please name your job with a valid Cloud Label. For
details, see:
https://cloud.google.com/compute/docs/labeling-resources#restrictions
Nov 07, 2018 6:38:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:38:33.919Z: Checking permissions granted to controller
Service Account.
Nov 07, 2018 6:38:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:38:42.175Z: Worker configuration: n1-standard-1 in
us-central1-a.
Nov 07, 2018 6:38:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:38:42.779Z: Expanding CoGroupByKey operations into
optimizable parts.
Nov 07, 2018 6:38:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:38:42.917Z: Expanding GroupByKey operations into
optimizable parts.
Nov 07, 2018 6:38:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:38:42.965Z: Lifting ValueCombiningMappingFns into
MergeBucketsMappingFns
Nov 07, 2018 6:38:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:38:43.213Z: Fusing adjacent ParDo, Read, Write, and
Flatten operations
Nov 07, 2018 6:38:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:38:43.252Z: Unzipping flatten s12 for input
s11.org.apache.beam.sdk.values.PCollection.<init>:402#d6c184af69871a3b
Nov 07, 2018 6:38:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:38:43.303Z: Fusing unzipped copy of
PAssert$0/CreateActual/FilterActuals/Window.Assign, through flatten Calculate
hashcode/Flatten.PCollections, into producer Calculate hashcode/ProduceDefault
Nov 07, 2018 6:38:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:38:43.371Z: Fusing consumer
PAssert$0/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Reify
into
PAssert$0/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey+PAssert$0/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Partial
Nov 07, 2018 6:38:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:38:43.453Z: Fusing consumer
PAssert$0/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues
into
PAssert$0/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Read
Nov 07, 2018 6:38:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:38:43.510Z: Fusing consumer
PAssert$0/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Extract
into
PAssert$0/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues
Nov 07, 2018 6:38:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:38:43.559Z: Fusing consumer
PAssert$0/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Write
into
PAssert$0/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Reify
Nov 07, 2018 6:38:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:38:43.604Z: Fusing consumer
PAssert$0/CreateActual/RewindowActuals/Window.Assign into
PAssert$0/CreateActual/Flatten.Iterables/FlattenIterables/FlatMap
Nov 07, 2018 6:38:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:38:43.640Z: Fusing consumer
PAssert$0/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map
into PAssert$0/CreateActual/ParDo(Anonymous)
Nov 07, 2018 6:38:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:38:43.688Z: Fusing consumer
PAssert$0/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey+PAssert$0/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Partial
into
PAssert$0/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map
Nov 07, 2018 6:38:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:38:43.735Z: Fusing consumer
PAssert$0/CreateActual/ParDo(Anonymous) into
PAssert$0/CreateActual/RewindowActuals/Window.Assign
Nov 07, 2018 6:38:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:38:43.779Z: Fusing consumer
PAssert$0/CreateActual/GatherPanes/WithKeys/AddKeys/Map into
PAssert$0/CreateActual/GatherPanes/Reify.Window/ParDo(Anonymous)
Nov 07, 2018 6:38:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:38:43.823Z: Fusing consumer
PAssert$0/CreateActual/GatherPanes/GroupByKey/GroupByWindow into
PAssert$0/CreateActual/GatherPanes/GroupByKey/Read
Nov 07, 2018 6:38:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:38:43.870Z: Fusing consumer
PAssert$0/CreateActual/ExtractPane/Map into
PAssert$0/CreateActual/GatherPanes/Values/Values/Map
Nov 07, 2018 6:38:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:38:43.918Z: Fusing consumer
PAssert$0/CreateActual/Flatten.Iterables/FlattenIterables/FlatMap into
PAssert$0/CreateActual/ExtractPane/Map
Nov 07, 2018 6:38:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:38:43.972Z: Fusing consumer
PAssert$0/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow)
into
PAssert$0/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
Nov 07, 2018 6:38:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:38:44.016Z: Fusing consumer
PAssert$0/CreateActual/GatherPanes/Values/Values/Map into
PAssert$0/CreateActual/GatherPanes/GroupByKey/GroupByWindow
Nov 07, 2018 6:38:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:38:44.064Z: Unzipping flatten s12-u54 for input
s13.output-c52
Nov 07, 2018 6:38:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:38:44.140Z: Fusing unzipped copy of
PAssert$0/CreateActual/GatherPanes/Reify.Window/ParDo(Anonymous), through
flatten s12-u54, into producer
PAssert$0/CreateActual/FilterActuals/Window.Assign
Nov 07, 2018 6:38:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:38:44.190Z: Unzipping flatten s12-u54-u61 for input
s15.org.apache.beam.sdk.values.PCollection.<init>:402#b8ec6571abcba715-c59
Nov 07, 2018 6:38:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:38:44.233Z: Fusing unzipped copy of
PAssert$0/CreateActual/GatherPanes/Window.Into()/Window.Assign, through flatten
s12-u54-u61, into producer
PAssert$0/CreateActual/GatherPanes/WithKeys/AddKeys/Map
Nov 07, 2018 6:38:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:38:44.269Z: Fusing consumer
PAssert$0/CreateActual/GatherPanes/GroupByKey/Write into
PAssert$0/CreateActual/GatherPanes/GroupByKey/Reify
Nov 07, 2018 6:38:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:38:44.317Z: Fusing consumer
PAssert$0/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map
into
PAssert$0/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Extract
Nov 07, 2018 6:38:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:38:44.365Z: Fusing consumer
PAssert$0/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
into
PAssert$0/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
Nov 07, 2018 6:38:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:38:44.410Z: Fusing consumer
PAssert$0/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
into
PAssert$0/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map
Nov 07, 2018 6:38:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:38:44.458Z: Unzipping flatten s12-u54-u61-u66 for input
s16.output-c64
Nov 07, 2018 6:38:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:38:44.504Z: Fusing unzipped copy of
PAssert$0/CreateActual/GatherPanes/GroupByKey/Reify, through flatten
s12-u54-u61-u66, into producer
PAssert$0/CreateActual/GatherPanes/Window.Into()/Window.Assign
Nov 07, 2018 6:38:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:38:44.554Z: Fusing consumer Calculate
hashcode/ProduceDefault into Calculate hashcode/CreateVoid/Read(CreateSource)
Nov 07, 2018 6:38:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:38:44.601Z: Fusing consumer
PAssert$0/CreateActual/FilterActuals/Window.Assign into Calculate
hashcode/Values/Values/Map
Nov 07, 2018 6:38:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:38:44.640Z: Fusing consumer Calculate
hashcode/View.AsIterable/ParDo(ToIsmRecordForGlobalWindow) into Calculate
hashcode/Values/Values/Map
Nov 07, 2018 6:38:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:38:44.688Z: Fusing consumer
PAssert$0/CreateActual/GatherPanes/GroupByKey/Reify into
PAssert$0/CreateActual/GatherPanes/Window.Into()/Window.Assign
Nov 07, 2018 6:38:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:38:44.735Z: Fusing consumer
PAssert$0/CreateActual/GatherPanes/Reify.Window/ParDo(Anonymous) into
PAssert$0/CreateActual/FilterActuals/Window.Assign
Nov 07, 2018 6:38:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:38:44.782Z: Fusing consumer
PAssert$0/CreateActual/GatherPanes/Window.Into()/Window.Assign into
PAssert$0/CreateActual/GatherPanes/WithKeys/AddKeys/Map
Nov 07, 2018 6:38:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:38:44.825Z: Fusing consumer Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Reify into Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey+Calculate
hashcode/Combine.perKey(Hashing)/Combine.GroupedValues/Partial
Nov 07, 2018 6:38:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:38:44.873Z: Fusing consumer Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey+Calculate
hashcode/Combine.perKey(Hashing)/Combine.GroupedValues/Partial into Calculate
hashcode/WithKeys/AddKeys/Map
Nov 07, 2018 6:38:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:38:44.920Z: Fusing consumer Calculate
hashcode/Values/Values/Map into Calculate
hashcode/Combine.perKey(Hashing)/Combine.GroupedValues/Extract
Nov 07, 2018 6:38:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:38:44.967Z: Fusing consumer Values as string into Get
values only/Values/Map
Nov 07, 2018 6:38:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:38:45.014Z: Fusing consumer Calculate
hashcode/WithKeys/AddKeys/Map into Values as string
Nov 07, 2018 6:38:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:38:45.061Z: Fusing consumer Calculate
hashcode/Combine.perKey(Hashing)/Combine.GroupedValues into Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Read
Nov 07, 2018 6:38:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:38:45.108Z: Fusing consumer Calculate
hashcode/Combine.perKey(Hashing)/Combine.GroupedValues/Extract into Calculate
hashcode/Combine.perKey(Hashing)/Combine.GroupedValues
Nov 07, 2018 6:38:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:38:45.151Z: Fusing consumer Get values only/Values/Map
into Read using HadoopInputFormat/Read(HadoopInputFormatBoundedSource)
Nov 07, 2018 6:38:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:38:45.194Z: Fusing consumer Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Write into Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Reify
Nov 07, 2018 6:38:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:38:45.243Z: Fusing consumer
PAssert$0/WindowToken/Window.Assign into
PAssert$0/Create.Values/Read(CreateSource)
Nov 07, 2018 6:38:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:38:45.271Z: Fusing consumer PAssert$0/RunChecks into
PAssert$0/WindowToken/Window.Assign
Nov 07, 2018 6:38:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:38:45.319Z: Fusing consumer
PAssert$0/VerifyAssertions/ParDo(DefaultConclude) into PAssert$0/RunChecks
Nov 07, 2018 6:38:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:38:45.789Z: Executing operation
PAssert$0/CreateActual/GatherPanes/GroupByKey/Create
Nov 07, 2018 6:38:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:38:45.835Z: Executing operation
PAssert$0/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Create
Nov 07, 2018 6:38:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:38:45.885Z: Starting 1 workers in us-central1-a...
Nov 07, 2018 6:38:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:38:45.885Z: Executing operation
PAssert$0/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Create
Nov 07, 2018 6:38:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:38:45.928Z: Executing operation Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Create
Nov 07, 2018 6:38:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:38:46.234Z: Executing operation Read using
HadoopInputFormat/Read(HadoopInputFormatBoundedSource)+Get values
only/Values/Map+Values as string+Calculate
hashcode/WithKeys/AddKeys/Map+Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey+Calculate
hashcode/Combine.perKey(Hashing)/Combine.GroupedValues/Partial+Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Reify+Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Write
Nov 07, 2018 6:38:57 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:38:54.810Z: Autoscaling: Raised the number of workers
to 0 based on the rate of progress in the currently running step(s).
Nov 07, 2018 6:39:34 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:39:33.081Z: Workers have started successfully.
Nov 07, 2018 6:39:55 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:39:53.178Z: Autoscaling: Raised the number of workers
to 1 based on the rate of progress in the currently running step(s).
Nov 07, 2018 6:39:55 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:39:53.209Z: Autoscaling: Would further reduce the
number of workers but reached the minimum number allowed for the job.
Nov 07, 2018 6:40:04 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:40:04.330Z: Workers have started successfully.
Nov 07, 2018 6:41:08 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:41:07.043Z: Executing operation Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Close
Nov 07, 2018 6:41:08 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:41:07.142Z: Executing operation Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Read+Calculate
hashcode/Combine.perKey(Hashing)/Combine.GroupedValues+Calculate
hashcode/Combine.perKey(Hashing)/Combine.GroupedValues/Extract+Calculate
hashcode/Values/Values/Map+PAssert$0/CreateActual/FilterActuals/Window.Assign+Calculate
hashcode/View.AsIterable/ParDo(ToIsmRecordForGlobalWindow)+PAssert$0/CreateActual/GatherPanes/Reify.Window/ParDo(Anonymous)+PAssert$0/CreateActual/GatherPanes/WithKeys/AddKeys/Map+PAssert$0/CreateActual/GatherPanes/Window.Into()/Window.Assign+PAssert$0/CreateActual/GatherPanes/GroupByKey/Reify+PAssert$0/CreateActual/GatherPanes/GroupByKey/Write
Nov 07, 2018 6:41:16 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:41:15.680Z: Executing operation Calculate
hashcode/View.AsIterable/CreateDataflowView
Nov 07, 2018 6:41:16 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:41:15.907Z: Executing operation Calculate
hashcode/CreateVoid/Read(CreateSource)+Calculate
hashcode/ProduceDefault+PAssert$0/CreateActual/FilterActuals/Window.Assign+PAssert$0/CreateActual/GatherPanes/Reify.Window/ParDo(Anonymous)+PAssert$0/CreateActual/GatherPanes/WithKeys/AddKeys/Map+PAssert$0/CreateActual/GatherPanes/Window.Into()/Window.Assign+PAssert$0/CreateActual/GatherPanes/GroupByKey/Reify+PAssert$0/CreateActual/GatherPanes/GroupByKey/Write
Nov 07, 2018 6:41:20 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:41:18.725Z: Executing operation
PAssert$0/CreateActual/GatherPanes/GroupByKey/Close
Nov 07, 2018 6:41:20 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:41:18.827Z: Executing operation
PAssert$0/CreateActual/GatherPanes/GroupByKey/Read+PAssert$0/CreateActual/GatherPanes/GroupByKey/GroupByWindow+PAssert$0/CreateActual/GatherPanes/Values/Values/Map+PAssert$0/CreateActual/ExtractPane/Map+PAssert$0/CreateActual/Flatten.Iterables/FlattenIterables/FlatMap+PAssert$0/CreateActual/RewindowActuals/Window.Assign+PAssert$0/CreateActual/ParDo(Anonymous)+PAssert$0/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map+PAssert$0/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey+PAssert$0/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Partial+PAssert$0/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Reify+PAssert$0/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Write
Nov 07, 2018 6:41:23 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:41:23.532Z: Executing operation
PAssert$0/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Close
Nov 07, 2018 6:41:23 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:41:23.629Z: Executing operation
PAssert$0/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Read+PAssert$0/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues+PAssert$0/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Extract+PAssert$0/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map+PAssert$0/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)+PAssert$0/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
Nov 07, 2018 6:41:27 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:41:26.938Z: Executing operation
PAssert$0/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Close
Nov 07, 2018 6:41:27 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:41:27.036Z: Executing operation
PAssert$0/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read+PAssert$0/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow)
Nov 07, 2018 6:41:32 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:41:32.701Z: Executing operation
PAssert$0/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/CreateDataflowView
Nov 07, 2018 6:41:35 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:41:32.917Z: Executing operation
PAssert$0/Create.Values/Read(CreateSource)+PAssert$0/WindowToken/Window.Assign+PAssert$0/RunChecks+PAssert$0/VerifyAssertions/ParDo(DefaultConclude)
Nov 07, 2018 6:41:35 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:41:33.698Z: Cleaning up.
Nov 07, 2018 6:41:35 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-07T18:41:33.794Z: Stopping worker pool...
STDERR:
2018-11-07 18:44:57,134 0846d0ee MainThread beam_integration_benchmark(1/1)
ERROR Error during benchmark beam_integration_benchmark
Traceback (most recent call last):
File
"<https://builds.apache.org/job/beam_PerformanceTests_HadoopInputFormat/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py";,>
line 719, in RunBenchmark
DoRunPhase(spec, collector, detailed_timer)
File
"<https://builds.apache.org/job/beam_PerformanceTests_HadoopInputFormat/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py";,>
line 580, in DoRunPhase
samples = spec.BenchmarkRun(spec)
File
"<https://builds.apache.org/job/beam_PerformanceTests_HadoopInputFormat/ws/PerfKitBenchmarker/perfkitbenchmarker/linux_benchmarks/beam_integration_benchmark.py";,>
line 160, in Run
job_type=job_type)
File
"<https://builds.apache.org/job/beam_PerformanceTests_HadoopInputFormat/ws/PerfKitBenchmarker/perfkitbenchmarker/providers/gcp/gcp_dpb_dataflow.py";,>
line 90, in SubmitJob
assert retcode == 0, "Integration Test Failed."
AssertionError: Integration Test Failed.
2018-11-07 18:44:57,136 0846d0ee MainThread beam_integration_benchmark(1/1)
INFO Cleaning up benchmark beam_integration_benchmark
2018-11-07 18:44:57,137 0846d0ee MainThread beam_integration_benchmark(1/1)
INFO Running: kubectl
--kubeconfig=<https://builds.apache.org/job/beam_PerformanceTests_HadoopInputFormat/ws/config-beam-performancetests-hadoopinputformat-976>
delete -f
<https://builds.apache.org/job/beam_PerformanceTests_HadoopInputFormat/ws/src/.test-infra/kubernetes/postgres/postgres-service-for-local-dev.yml>
--ignore-not-found
2018-11-07 18:44:57,947 0846d0ee MainThread beam_integration_benchmark(1/1)
ERROR Exception running benchmark
Traceback (most recent call last):
File
"<https://builds.apache.org/job/beam_PerformanceTests_HadoopInputFormat/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py";,>
line 860, in RunBenchmarkTask
RunBenchmark(spec, collector)
File
"<https://builds.apache.org/job/beam_PerformanceTests_HadoopInputFormat/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py";,>
line 719, in RunBenchmark
DoRunPhase(spec, collector, detailed_timer)
File
"<https://builds.apache.org/job/beam_PerformanceTests_HadoopInputFormat/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py";,>
line 580, in DoRunPhase
samples = spec.BenchmarkRun(spec)
File
"<https://builds.apache.org/job/beam_PerformanceTests_HadoopInputFormat/ws/PerfKitBenchmarker/perfkitbenchmarker/linux_benchmarks/beam_integration_benchmark.py";,>
line 160, in Run
job_type=job_type)
File
"<https://builds.apache.org/job/beam_PerformanceTests_HadoopInputFormat/ws/PerfKitBenchmarker/perfkitbenchmarker/providers/gcp/gcp_dpb_dataflow.py";,>
line 90, in SubmitJob
assert retcode == 0, "Integration Test Failed."
AssertionError: Integration Test Failed.
2018-11-07 18:44:57,948 0846d0ee MainThread beam_integration_benchmark(1/1)
ERROR Benchmark 1/1 beam_integration_benchmark (UID:
beam_integration_benchmark0) failed. Execution will continue.
2018-11-07 18:44:57,997 0846d0ee MainThread INFO Benchmark run statuses:
---------------------------------------------------------------------------------
Name UID Status Failed
Substatus
---------------------------------------------------------------------------------
beam_integration_benchmark beam_integration_benchmark0 FAILED
---------------------------------------------------------------------------------
Success rate: 0.00% (0/1)
2018-11-07 18:44:57,998 0846d0ee MainThread INFO Complete logs can be found
at:
<https://builds.apache.org/job/beam_PerformanceTests_HadoopInputFormat/ws/runs/0846d0ee/pkb.log>
2018-11-07 18:44:57,998 0846d0ee MainThread INFO Completion statuses can be
found at:
<https://builds.apache.org/job/beam_PerformanceTests_HadoopInputFormat/ws/runs/0846d0ee/completion_statuses.json>
Build step 'Execute shell' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscr...@beam.apache.org
For additional commands, e-mail: builds-h...@beam.apache.org
