See
<https://builds.apache.org/job/beam_PerformanceTests_HadoopFormat/180/display/redirect?page=changes>
Changes:
[ehudm] [BEAM-7432] Set input and output files for sdist
[ehudm] [BEAM-7531] Remove and disallow KMS key setting
[ehudm] Reduce gradle build verbosity
[mxm] [BEAM-7551] Checkpoint Flink's ImpulseSourceFunction
[kanterov] Merge pull request #8858: [BEAM-7542] Fix faulty cast in
BigQueryUtils
[robertwb] Remove unused **kwargs arguments for various transforms.
[github] Adding a Link to The Doc for Cost Estimation
------------------------------------------
[...truncated 534.73 KB...]
Submitted job: 2019-06-14_11_49_53-848578026723844113
org.apache.beam.sdk.io.hadoop.format.HadoopFormatIOIT >
writeAndReadUsingHadoopFormat STANDARD_ERROR
Jun 14, 2019 6:49:54 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel
--region=us-central1 2019-06-14_11_49_53-848578026723844113
Jun 14, 2019 6:49:54 PM org.apache.beam.runners.dataflow.TestDataflowRunner
run
INFO: Running Dataflow job 2019-06-14_11_49_53-848578026723844113 with 1
expected assertions.
Jun 14, 2019 6:49:55 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:49:53.561Z: Autoscaling is enabled for job
2019-06-14_11_49_53-848578026723844113. The number of workers will be between 1
and 1000.
Jun 14, 2019 6:49:55 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:49:53.735Z: Autoscaling was automatically enabled for
job 2019-06-14_11_49_53-848578026723844113.
Jun 14, 2019 6:49:57 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
WARNING: 2019-06-14T18:49:56.850Z: The workflow name is not a valid Cloud
Label. Labels applied to Cloud resources (such as GCE Instances) for monitoring
will be labeled with this modified job name:
hadoopformatioit0writeandreadusinghadoopformat-jenkins-061-eaxs. For the best
monitoring experience, please name your job with a valid Cloud Label. For
details, see:
https://cloud.google.com/compute/docs/labeling-resources#restrictions
Jun 14, 2019 6:49:59 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:49:57.424Z: Checking permissions granted to controller
Service Account.
Jun 14, 2019 6:50:09 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:50:08.039Z: Worker configuration: n1-standard-1 in
us-central1-f.
Jun 14, 2019 6:50:09 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:50:08.656Z: Expanding CoGroupByKey operations into
optimizable parts.
Jun 14, 2019 6:50:09 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:50:08.801Z: Expanding GroupByKey operations into
optimizable parts.
Jun 14, 2019 6:50:09 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:50:08.836Z: Lifting ValueCombiningMappingFns into
MergeBucketsMappingFns
Jun 14, 2019 6:50:09 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:50:09.086Z: Fusing adjacent ParDo, Read, Write, and
Flatten operations
Jun 14, 2019 6:50:09 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:50:09.127Z: Elided trivial flatten
Jun 14, 2019 6:50:09 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:50:09.167Z: Elided trivial flatten
Jun 14, 2019 6:50:09 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:50:09.204Z: Elided trivial flatten
Jun 14, 2019 6:50:09 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:50:09.257Z: Fusing consumer
PAssert$0/GroupGlobally/GroupDummyAndContents/Write into
PAssert$0/GroupGlobally/GroupDummyAndContents/Reify
Jun 14, 2019 6:50:09 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:50:09.301Z: Fusing consumer
PAssert$0/GroupGlobally/GroupDummyAndContents/GroupByWindow into
PAssert$0/GroupGlobally/GroupDummyAndContents/Read
Jun 14, 2019 6:50:09 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:50:09.359Z: Fusing consumer
PAssert$0/GroupGlobally/Values/Values/Map into
PAssert$0/GroupGlobally/GroupDummyAndContents/GroupByWindow
Jun 14, 2019 6:50:09 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:50:09.405Z: Fusing consumer
PAssert$0/GroupGlobally/ParDo(Concat) into
PAssert$0/GroupGlobally/Values/Values/Map
Jun 14, 2019 6:50:10 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:50:09.446Z: Fusing consumer PAssert$0/GetPane/Map into
PAssert$0/GroupGlobally/ParDo(Concat)
Jun 14, 2019 6:50:10 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:50:09.498Z: Fusing consumer PAssert$0/RunChecks into
PAssert$0/GetPane/Map
Jun 14, 2019 6:50:10 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:50:09.542Z: Fusing consumer
PAssert$0/VerifyAssertions/ParDo(DefaultConclude) into PAssert$0/RunChecks
Jun 14, 2019 6:50:10 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:50:09.594Z: Unzipping flatten s26 for input
s21.org.apache.beam.sdk.values.PCollection.<init>:402#80783fbab681bb5c
Jun 14, 2019 6:50:10 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:50:09.635Z: Fusing unzipped copy of
PAssert$0/GroupGlobally/GroupDummyAndContents/Reify, through flatten
PAssert$0/GroupGlobally/FlattenDummyAndContents, into producer
PAssert$0/GroupGlobally/KeyForDummy/AddKeys/Map
Jun 14, 2019 6:50:10 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:50:09.675Z: Unzipping flatten s13 for input
s12.org.apache.beam.sdk.values.PCollection.<init>:402#c93a13e4ee1094aa
Jun 14, 2019 6:50:10 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:50:09.726Z: Fusing unzipped copy of
PAssert$0/GroupGlobally/Window.Into()/Window.Assign, through flatten Calculate
hashcode/Flatten.PCollections, into producer Calculate hashcode/ProduceDefault
Jun 14, 2019 6:50:10 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:50:09.771Z: Fusing consumer
PAssert$0/GroupGlobally/KeyForDummy/AddKeys/Map into
PAssert$0/GroupGlobally/RewindowActuals/Window.Assign
Jun 14, 2019 6:50:10 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:50:09.822Z: Unzipping flatten s13-u44 for input
s14.output-c42
Jun 14, 2019 6:50:10 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:50:09.866Z: Fusing unzipped copy of
PAssert$0/GroupGlobally/GatherAllOutputs/Reify.Window/ParDo(Anonymous), through
flatten Calculate hashcode/Flatten.PCollections/Unzipped-1, into producer
PAssert$0/GroupGlobally/Window.Into()/Window.Assign
Jun 14, 2019 6:50:10 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:50:09.918Z: Fusing consumer
PAssert$0/GroupGlobally/GatherAllOutputs/GroupByKey/Write into
PAssert$0/GroupGlobally/GatherAllOutputs/GroupByKey/Reify
Jun 14, 2019 6:50:10 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:50:09.960Z: Fusing consumer
PAssert$0/GroupGlobally/GatherAllOutputs/Values/Values/Map into
PAssert$0/GroupGlobally/GatherAllOutputs/GroupByKey/GroupByWindow
Jun 14, 2019 6:50:10 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:50:10.015Z: Fusing consumer
PAssert$0/GroupGlobally/RewindowActuals/Window.Assign into
PAssert$0/GroupGlobally/GatherAllOutputs/Values/Values/Map
Jun 14, 2019 6:50:10 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:50:10.062Z: Fusing consumer
PAssert$0/GroupGlobally/GatherAllOutputs/GroupByKey/GroupByWindow into
PAssert$0/GroupGlobally/GatherAllOutputs/GroupByKey/Read
Jun 14, 2019 6:50:10 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:50:10.099Z: Fusing consumer
PAssert$0/GroupGlobally/GatherAllOutputs/Window.Into()/Window.Assign into
PAssert$0/GroupGlobally/GatherAllOutputs/WithKeys/AddKeys/Map
Jun 14, 2019 6:50:10 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:50:10.147Z: Fusing consumer
PAssert$0/GroupGlobally/GatherAllOutputs/GroupByKey/Reify into
PAssert$0/GroupGlobally/GatherAllOutputs/Window.Into()/Window.Assign
Jun 14, 2019 6:50:10 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:50:10.200Z: Unzipping flatten s13-u44-u49 for input
s15.org.apache.beam.sdk.values.PCollection.<init>:402#b96b6624e7817fb3-c47
Jun 14, 2019 6:50:10 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:50:10.246Z: Fusing unzipped copy of
PAssert$0/GroupGlobally/GatherAllOutputs/WithKeys/AddKeys/Map, through flatten
Calculate hashcode/Flatten.PCollections/Unzipped-2, into producer
PAssert$0/GroupGlobally/GatherAllOutputs/Reify.Window/ParDo(Anonymous)
Jun 14, 2019 6:50:10 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:50:10.293Z: Fusing consumer Calculate
hashcode/ProduceDefault into Calculate hashcode/CreateVoid/Read(CreateSource)
Jun 14, 2019 6:50:10 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:50:10.344Z: Fusing consumer
PAssert$0/GroupGlobally/Window.Into()/Window.Assign into Calculate
hashcode/Values/Values/Map
Jun 14, 2019 6:50:10 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:50:10.388Z: Fusing consumer Calculate
hashcode/View.AsIterable/ParDo(ToIsmRecordForGlobalWindow) into Calculate
hashcode/Values/Values/Map
Jun 14, 2019 6:50:10 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:50:10.436Z: Fusing consumer
PAssert$0/GroupGlobally/GatherAllOutputs/WithKeys/AddKeys/Map into
PAssert$0/GroupGlobally/GatherAllOutputs/Reify.Window/ParDo(Anonymous)
Jun 14, 2019 6:50:10 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:50:10.478Z: Fusing consumer
PAssert$0/GroupGlobally/GatherAllOutputs/Reify.Window/ParDo(Anonymous) into
PAssert$0/GroupGlobally/Window.Into()/Window.Assign
Jun 14, 2019 6:50:10 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:50:10.522Z: Fusing consumer Calculate
hashcode/Combine.perKey(Hashing)/Combine.GroupedValues into Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Read
Jun 14, 2019 6:50:10 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:50:10.585Z: Fusing consumer Get values only/Values/Map
into Collect read time
Jun 14, 2019 6:50:11 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:50:10.635Z: Fusing consumer Values as string into Get
values only/Values/Map
Jun 14, 2019 6:50:11 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:50:10.684Z: Fusing consumer Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Reify into Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey+Calculate
hashcode/Combine.perKey(Hashing)/Combine.GroupedValues/Partial
Jun 14, 2019 6:50:11 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:50:10.733Z: Fusing consumer Calculate
hashcode/Combine.perKey(Hashing)/Combine.GroupedValues/Extract into Calculate
hashcode/Combine.perKey(Hashing)/Combine.GroupedValues
Jun 14, 2019 6:50:11 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:50:10.783Z: Fusing consumer Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Write into Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Reify
Jun 14, 2019 6:50:11 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:50:10.820Z: Fusing consumer Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey+Calculate
hashcode/Combine.perKey(Hashing)/Combine.GroupedValues/Partial into Calculate
hashcode/WithKeys/AddKeys/Map
Jun 14, 2019 6:50:11 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:50:10.863Z: Fusing consumer Calculate
hashcode/Values/Values/Map into Calculate
hashcode/Combine.perKey(Hashing)/Combine.GroupedValues/Extract
Jun 14, 2019 6:50:11 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:50:10.916Z: Fusing consumer Calculate
hashcode/WithKeys/AddKeys/Map into Values as string
Jun 14, 2019 6:50:11 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:50:10.966Z: Fusing consumer Collect read time into Read
using Hadoop InputFormat/Read(HadoopInputFormatBoundedSource)
Jun 14, 2019 6:50:11 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:50:11.015Z: Fusing consumer
PAssert$0/GroupGlobally/GroupDummyAndContents/Reify into
PAssert$0/GroupGlobally/WindowIntoDummy/Window.Assign
Jun 14, 2019 6:50:11 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:50:11.053Z: Fusing consumer
PAssert$0/GroupGlobally/WindowIntoDummy/Window.Assign into
PAssert$0/GroupGlobally/Create.Values/Read(CreateSource)
Jun 14, 2019 6:50:11 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:50:11.560Z: Executing operation
PAssert$0/GroupGlobally/GatherAllOutputs/GroupByKey/Create
Jun 14, 2019 6:50:11 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:50:11.608Z: Executing operation
PAssert$0/GroupGlobally/GroupDummyAndContents/Create
Jun 14, 2019 6:50:11 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:50:11.657Z: Executing operation Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Create
Jun 14, 2019 6:50:11 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:50:11.657Z: Starting 1 workers in us-central1-f...
Jun 14, 2019 6:50:11 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:50:11.724Z: Finished operation
PAssert$0/GroupGlobally/GatherAllOutputs/GroupByKey/Create
Jun 14, 2019 6:50:11 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:50:11.744Z: Finished operation
PAssert$0/GroupGlobally/GroupDummyAndContents/Create
Jun 14, 2019 6:50:11 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:50:11.744Z: Finished operation Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Create
Jun 14, 2019 6:50:14 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:50:11.973Z: Executing operation Read using Hadoop
InputFormat/Read(HadoopInputFormatBoundedSource)+Collect read time+Get values
only/Values/Map+Values as string+Calculate
hashcode/WithKeys/AddKeys/Map+Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey+Calculate
hashcode/Combine.perKey(Hashing)/Combine.GroupedValues/Partial+Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Reify+Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Write
Jun 14, 2019 6:50:14 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:50:12.040Z: Executing operation
PAssert$0/GroupGlobally/Create.Values/Read(CreateSource)+PAssert$0/GroupGlobally/WindowIntoDummy/Window.Assign+PAssert$0/GroupGlobally/GroupDummyAndContents/Reify+PAssert$0/GroupGlobally/GroupDummyAndContents/Write
Jun 14, 2019 6:51:04 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:51:01.655Z: Workers have started successfully.
Jun 14, 2019 6:51:20 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:51:19.932Z: Autoscaling: Raised the number of workers
to 1 based on the rate of progress in the currently running step(s).
Jun 14, 2019 6:51:33 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:51:33.024Z: Workers have started successfully.
Jun 14, 2019 6:51:41 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:51:39.806Z: Finished operation
PAssert$0/GroupGlobally/Create.Values/Read(CreateSource)+PAssert$0/GroupGlobally/WindowIntoDummy/Window.Assign+PAssert$0/GroupGlobally/GroupDummyAndContents/Reify+PAssert$0/GroupGlobally/GroupDummyAndContents/Write
Jun 14, 2019 6:51:46 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:51:44.246Z: Finished operation Read using Hadoop
InputFormat/Read(HadoopInputFormatBoundedSource)+Collect read time+Get values
only/Values/Map+Values as string+Calculate
hashcode/WithKeys/AddKeys/Map+Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey+Calculate
hashcode/Combine.perKey(Hashing)/Combine.GroupedValues/Partial+Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Reify+Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Write
Jun 14, 2019 6:51:46 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:51:44.356Z: Executing operation Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Close
Jun 14, 2019 6:51:46 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:51:44.407Z: Finished operation Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Close
Jun 14, 2019 6:51:46 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:51:44.516Z: Executing operation Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Read+Calculate
hashcode/Combine.perKey(Hashing)/Combine.GroupedValues+Calculate
hashcode/Combine.perKey(Hashing)/Combine.GroupedValues/Extract+Calculate
hashcode/Values/Values/Map+PAssert$0/GroupGlobally/Window.Into()/Window.Assign+Calculate
hashcode/View.AsIterable/ParDo(ToIsmRecordForGlobalWindow)+PAssert$0/GroupGlobally/GatherAllOutputs/Reify.Window/ParDo(Anonymous)+PAssert$0/GroupGlobally/GatherAllOutputs/WithKeys/AddKeys/Map+PAssert$0/GroupGlobally/GatherAllOutputs/Window.Into()/Window.Assign+PAssert$0/GroupGlobally/GatherAllOutputs/GroupByKey/Reify+PAssert$0/GroupGlobally/GatherAllOutputs/GroupByKey/Write
Jun 14, 2019 6:51:54 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:51:52.338Z: Finished operation Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Read+Calculate
hashcode/Combine.perKey(Hashing)/Combine.GroupedValues+Calculate
hashcode/Combine.perKey(Hashing)/Combine.GroupedValues/Extract+Calculate
hashcode/Values/Values/Map+PAssert$0/GroupGlobally/Window.Into()/Window.Assign+Calculate
hashcode/View.AsIterable/ParDo(ToIsmRecordForGlobalWindow)+PAssert$0/GroupGlobally/GatherAllOutputs/Reify.Window/ParDo(Anonymous)+PAssert$0/GroupGlobally/GatherAllOutputs/WithKeys/AddKeys/Map+PAssert$0/GroupGlobally/GatherAllOutputs/Window.Into()/Window.Assign+PAssert$0/GroupGlobally/GatherAllOutputs/GroupByKey/Reify+PAssert$0/GroupGlobally/GatherAllOutputs/GroupByKey/Write
Jun 14, 2019 6:51:54 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:51:52.545Z: Executing operation Calculate
hashcode/View.AsIterable/CreateDataflowView
Jun 14, 2019 6:51:54 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:51:52.608Z: Finished operation Calculate
hashcode/View.AsIterable/CreateDataflowView
Jun 14, 2019 6:51:54 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:51:52.835Z: Executing operation Calculate
hashcode/CreateVoid/Read(CreateSource)+Calculate
hashcode/ProduceDefault+PAssert$0/GroupGlobally/Window.Into()/Window.Assign+PAssert$0/GroupGlobally/GatherAllOutputs/Reify.Window/ParDo(Anonymous)+PAssert$0/GroupGlobally/GatherAllOutputs/WithKeys/AddKeys/Map+PAssert$0/GroupGlobally/GatherAllOutputs/Window.Into()/Window.Assign+PAssert$0/GroupGlobally/GatherAllOutputs/GroupByKey/Reify+PAssert$0/GroupGlobally/GatherAllOutputs/GroupByKey/Write
Jun 14, 2019 6:51:55 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:51:54.507Z: Finished operation Calculate
hashcode/CreateVoid/Read(CreateSource)+Calculate
hashcode/ProduceDefault+PAssert$0/GroupGlobally/Window.Into()/Window.Assign+PAssert$0/GroupGlobally/GatherAllOutputs/Reify.Window/ParDo(Anonymous)+PAssert$0/GroupGlobally/GatherAllOutputs/WithKeys/AddKeys/Map+PAssert$0/GroupGlobally/GatherAllOutputs/Window.Into()/Window.Assign+PAssert$0/GroupGlobally/GatherAllOutputs/GroupByKey/Reify+PAssert$0/GroupGlobally/GatherAllOutputs/GroupByKey/Write
Jun 14, 2019 6:51:55 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:51:54.605Z: Executing operation
PAssert$0/GroupGlobally/GatherAllOutputs/GroupByKey/Close
Jun 14, 2019 6:51:55 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:51:54.653Z: Finished operation
PAssert$0/GroupGlobally/GatherAllOutputs/GroupByKey/Close
Jun 14, 2019 6:51:55 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:51:54.746Z: Executing operation
PAssert$0/GroupGlobally/GatherAllOutputs/GroupByKey/Read+PAssert$0/GroupGlobally/GatherAllOutputs/GroupByKey/GroupByWindow+PAssert$0/GroupGlobally/GatherAllOutputs/Values/Values/Map+PAssert$0/GroupGlobally/RewindowActuals/Window.Assign+PAssert$0/GroupGlobally/KeyForDummy/AddKeys/Map+PAssert$0/GroupGlobally/GroupDummyAndContents/Reify+PAssert$0/GroupGlobally/GroupDummyAndContents/Write
Jun 14, 2019 6:52:00 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:51:59.372Z: Finished operation
PAssert$0/GroupGlobally/GatherAllOutputs/GroupByKey/Read+PAssert$0/GroupGlobally/GatherAllOutputs/GroupByKey/GroupByWindow+PAssert$0/GroupGlobally/GatherAllOutputs/Values/Values/Map+PAssert$0/GroupGlobally/RewindowActuals/Window.Assign+PAssert$0/GroupGlobally/KeyForDummy/AddKeys/Map+PAssert$0/GroupGlobally/GroupDummyAndContents/Reify+PAssert$0/GroupGlobally/GroupDummyAndContents/Write
Jun 14, 2019 6:52:00 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:51:59.489Z: Executing operation
PAssert$0/GroupGlobally/GroupDummyAndContents/Close
Jun 14, 2019 6:52:00 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:51:59.545Z: Finished operation
PAssert$0/GroupGlobally/GroupDummyAndContents/Close
Jun 14, 2019 6:52:00 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:51:59.666Z: Executing operation
PAssert$0/GroupGlobally/GroupDummyAndContents/Read+PAssert$0/GroupGlobally/GroupDummyAndContents/GroupByWindow+PAssert$0/GroupGlobally/Values/Values/Map+PAssert$0/GroupGlobally/ParDo(Concat)+PAssert$0/GetPane/Map+PAssert$0/RunChecks+PAssert$0/VerifyAssertions/ParDo(DefaultConclude)
Jun 14, 2019 6:52:06 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:52:04.903Z: Finished operation
PAssert$0/GroupGlobally/GroupDummyAndContents/Read+PAssert$0/GroupGlobally/GroupDummyAndContents/GroupByWindow+PAssert$0/GroupGlobally/Values/Values/Map+PAssert$0/GroupGlobally/ParDo(Concat)+PAssert$0/GetPane/Map+PAssert$0/RunChecks+PAssert$0/VerifyAssertions/ParDo(DefaultConclude)
Jun 14, 2019 6:52:06 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:52:05.175Z: Cleaning up.
Jun 14, 2019 6:52:06 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-06-14T18:52:05.304Z: Stopping worker pool...
STDERR: Note:
<https://builds.apache.org/job/beam_PerformanceTests_HadoopFormat/ws/src/sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/IOITHelperTest.java>
uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note:
<https://builds.apache.org/job/beam_PerformanceTests_HadoopFormat/ws/src/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java>
uses unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
2019-06-14 18:54:15,513 d1d953fe MainThread beam_integration_benchmark(1/1)
ERROR Error during benchmark beam_integration_benchmark
Traceback (most recent call last):
File
"<https://builds.apache.org/job/beam_PerformanceTests_HadoopFormat/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py";,>
line 760, in RunBenchmark
DoRunPhase(spec, collector, detailed_timer)
File
"<https://builds.apache.org/job/beam_PerformanceTests_HadoopFormat/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py";,>
line 609, in DoRunPhase
samples = spec.BenchmarkRun(spec)
File
"<https://builds.apache.org/job/beam_PerformanceTests_HadoopFormat/ws/PerfKitBenchmarker/perfkitbenchmarker/linux_benchmarks/beam_integration_benchmark.py";,>
line 160, in Run
job_type=job_type)
File
"<https://builds.apache.org/job/beam_PerformanceTests_HadoopFormat/ws/PerfKitBenchmarker/perfkitbenchmarker/providers/gcp/gcp_dpb_dataflow.py";,>
line 90, in SubmitJob
assert retcode == 0, "Integration Test Failed."
AssertionError: Integration Test Failed.
2019-06-14 18:54:15,514 d1d953fe MainThread beam_integration_benchmark(1/1)
INFO Cleaning up benchmark beam_integration_benchmark
2019-06-14 18:54:15,515 d1d953fe MainThread beam_integration_benchmark(1/1)
INFO Running: kubectl
--kubeconfig=<https://builds.apache.org/job/beam_PerformanceTests_HadoopFormat/ws/config-beam-performancetests-hadoopformat-180>
delete -f
<https://builds.apache.org/job/beam_PerformanceTests_HadoopFormat/ws/src/.test-infra/kubernetes/postgres/postgres-service-for-local-dev.yml>
--ignore-not-found
2019-06-14 18:54:16,187 d1d953fe MainThread beam_integration_benchmark(1/1)
ERROR Exception running benchmark
Traceback (most recent call last):
File
"<https://builds.apache.org/job/beam_PerformanceTests_HadoopFormat/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py";,>
line 903, in RunBenchmarkTask
RunBenchmark(spec, collector)
File
"<https://builds.apache.org/job/beam_PerformanceTests_HadoopFormat/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py";,>
line 760, in RunBenchmark
DoRunPhase(spec, collector, detailed_timer)
File
"<https://builds.apache.org/job/beam_PerformanceTests_HadoopFormat/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py";,>
line 609, in DoRunPhase
samples = spec.BenchmarkRun(spec)
File
"<https://builds.apache.org/job/beam_PerformanceTests_HadoopFormat/ws/PerfKitBenchmarker/perfkitbenchmarker/linux_benchmarks/beam_integration_benchmark.py";,>
line 160, in Run
job_type=job_type)
File
"<https://builds.apache.org/job/beam_PerformanceTests_HadoopFormat/ws/PerfKitBenchmarker/perfkitbenchmarker/providers/gcp/gcp_dpb_dataflow.py";,>
line 90, in SubmitJob
assert retcode == 0, "Integration Test Failed."
AssertionError: Integration Test Failed.
2019-06-14 18:54:16,188 d1d953fe MainThread beam_integration_benchmark(1/1)
ERROR Benchmark 1/1 beam_integration_benchmark (UID:
beam_integration_benchmark0) failed. Execution will continue.
2019-06-14 18:54:16,189 d1d953fe MainThread beam_integration_benchmark(1/1)
INFO Benchmark run statuses:
---------------------------------------------------------------------------------
Name UID Status Failed
Substatus
---------------------------------------------------------------------------------
beam_integration_benchmark beam_integration_benchmark0 FAILED
---------------------------------------------------------------------------------
Success rate: 0.00% (0/1)
2019-06-14 18:54:16,189 d1d953fe MainThread beam_integration_benchmark(1/1)
INFO Complete logs can be found at:
<https://builds.apache.org/job/beam_PerformanceTests_HadoopFormat/ws/runs/d1d953fe/pkb.log>
2019-06-14 18:54:16,189 d1d953fe MainThread beam_integration_benchmark(1/1)
INFO Completion statuses can be found at:
<https://builds.apache.org/job/beam_PerformanceTests_HadoopFormat/ws/runs/d1d953fe/completion_statuses.json>
Build step 'Execute shell' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
