See
<https://ci-beam.apache.org/job/beam_PerformanceTests_AvroIOIT_HDFS/5617/display/redirect>
Changes:
------------------------------------------
[...truncated 357.97 KB...]
Feb 18, 2022 6:23:26 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:23:24.705Z: Unzipping flatten s48-c98 for input
s43.org.apache.beam.sdk.values.PCollection.<init>:408#fe542dba50f8fd8a
Feb 18, 2022 6:23:26 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:23:24.734Z: Fusing unzipped copy of
PAssert$0/GroupGlobally/Reify.Window/ParDo(Anonymous), through flatten
Calculate hashcode/Flatten.PCollections, into producer Calculate
hashcode/Values/Values/Map
Feb 18, 2022 6:23:26 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:23:24.756Z: Fusing consumer
PAssert$0/GroupGlobally/Reify.Window/ParDo(Anonymous) into Calculate
hashcode/ProduceDefault
Feb 18, 2022 6:23:26 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:23:24.791Z: Fusing consumer
View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map
into Calculate hashcode/ProduceDefault
Feb 18, 2022 6:23:26 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:23:24.816Z: Fusing consumer Match all files/Match
filepatterns into Values/Values/Map
Feb 18, 2022 6:23:26 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:23:24.851Z: Fusing consumer Calculate
hashcode/View.AsIterable/ParDo(ToIsmRecordForGlobalWindow) into Calculate
hashcode/Values/Values/Map
Feb 18, 2022 6:23:26 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:23:24.895Z: Fusing consumer Write Avro records to
files/Write/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Reify.ReifyView/ParDo(Anonymous)
into Write Avro records to
files/Write/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Create.Values/Read(CreateSource)
Feb 18, 2022 6:23:26 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:23:24.927Z: Fusing consumer Write Avro records to
files/Write/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Values/Values/Map
into Write Avro records to
files/Write/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Reify.ReifyView/ParDo(Anonymous)
Feb 18, 2022 6:23:26 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:23:24.961Z: Fusing consumer Write Avro records to
files/Write/FinalizeTempFileBundles/Finalize into Write Avro records to
files/Write/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Values/Values/Map
Feb 18, 2022 6:23:26 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:23:24.994Z: Fusing consumer Write Avro records to
files/Write/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair with random key
into Write Avro records to files/Write/FinalizeTempFileBundles/Finalize
Feb 18, 2022 6:23:26 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:23:25.031Z: Fusing consumer Write Avro records to
files/Write/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign
into Write Avro records to
files/Write/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair with random key
Feb 18, 2022 6:23:26 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:23:25.071Z: Fusing consumer Write Avro records to
files/Write/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify
into Write Avro records to
files/Write/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign
Feb 18, 2022 6:23:26 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:23:25.115Z: Fusing consumer Write Avro records to
files/Write/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write
into Write Avro records to
files/Write/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify
Feb 18, 2022 6:23:26 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:23:25.145Z: Fusing consumer Write Avro records to
files/Write/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow
into Write Avro records to
files/Write/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read
Feb 18, 2022 6:23:26 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:23:25.178Z: Fusing consumer Write Avro records to
files/Write/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable
into Write Avro records to
files/Write/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow
Feb 18, 2022 6:23:26 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:23:25.202Z: Fusing consumer Write Avro records to
files/Write/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Values/Values/Map
into Write Avro records to
files/Write/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable
Feb 18, 2022 6:23:26 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:23:25.232Z: Fusing consumer Collect middle time into
Write Avro records to
files/Write/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Values/Values/Map
Feb 18, 2022 6:23:26 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:23:25.266Z: Fusing consumer Values/Values/Map into
Collect middle time
Feb 18, 2022 6:23:26 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:23:25.301Z: Fusing consumer Match all
files/Reshuffle.ViaRandomKey/Pair with random key into Match all files/Match
filepatterns
Feb 18, 2022 6:23:26 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:23:25.322Z: Fusing consumer Match all
files/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign into Match
all files/Reshuffle.ViaRandomKey/Pair with random key
Feb 18, 2022 6:23:26 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:23:25.355Z: Fusing consumer Match all
files/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify into Match all
files/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign
Feb 18, 2022 6:23:26 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:23:25.381Z: Fusing consumer Match all
files/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write into Match all
files/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify
Feb 18, 2022 6:23:26 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:23:25.418Z: Fusing consumer Match all
files/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow into Match all
files/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read
Feb 18, 2022 6:23:26 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:23:25.455Z: Fusing consumer Match all
files/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable into Match all
files/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow
Feb 18, 2022 6:23:26 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:23:25.486Z: Fusing consumer Match all
files/Reshuffle.ViaRandomKey/Values/Values/Map into Match all
files/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable
Feb 18, 2022 6:23:26 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:23:25.511Z: Fusing consumer Read
matches/ParDo(ToReadableFile) into Match all
files/Reshuffle.ViaRandomKey/Values/Values/Map
Feb 18, 2022 6:23:26 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:23:25.538Z: Fusing consumer Read files/Read all via
FileBasedSource/Split into ranges into Read matches/ParDo(ToReadableFile)
Feb 18, 2022 6:23:26 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:23:25.568Z: Fusing consumer Read files/Read all via
FileBasedSource/Reshuffle/Pair with random key into Read files/Read all via
FileBasedSource/Split into ranges
Feb 18, 2022 6:23:26 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:23:25.605Z: Fusing consumer Read files/Read all via
FileBasedSource/Reshuffle/Reshuffle/Window.Into()/Window.Assign into Read
files/Read all via FileBasedSource/Reshuffle/Pair with random key
Feb 18, 2022 6:23:26 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:23:25.637Z: Fusing consumer Read files/Read all via
FileBasedSource/Reshuffle/Reshuffle/GroupByKey/Reify into Read files/Read all
via FileBasedSource/Reshuffle/Reshuffle/Window.Into()/Window.Assign
Feb 18, 2022 6:23:26 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:23:25.670Z: Fusing consumer Read files/Read all via
FileBasedSource/Reshuffle/Reshuffle/GroupByKey/Write into Read files/Read all
via FileBasedSource/Reshuffle/Reshuffle/GroupByKey/Reify
Feb 18, 2022 6:23:26 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:23:25.705Z: Fusing consumer Read files/Read all via
FileBasedSource/Reshuffle/Reshuffle/GroupByKey/GroupByWindow into Read
files/Read all via FileBasedSource/Reshuffle/Reshuffle/GroupByKey/Read
Feb 18, 2022 6:23:26 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:23:25.740Z: Fusing consumer Read files/Read all via
FileBasedSource/Reshuffle/Reshuffle/ExpandIterable into Read files/Read all via
FileBasedSource/Reshuffle/Reshuffle/GroupByKey/GroupByWindow
Feb 18, 2022 6:23:26 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:23:25.762Z: Fusing consumer Read files/Read all via
FileBasedSource/Reshuffle/Values/Values/Map into Read files/Read all via
FileBasedSource/Reshuffle/Reshuffle/ExpandIterable
Feb 18, 2022 6:23:26 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:23:25.790Z: Fusing consumer Read files/Read all via
FileBasedSource/Read ranges into Read files/Read all via
FileBasedSource/Reshuffle/Values/Values/Map
Feb 18, 2022 6:23:26 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:23:25.816Z: Fusing consumer Collect end time into Read
files/Read all via FileBasedSource/Read ranges
Feb 18, 2022 6:23:26 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:23:25.844Z: Fusing consumer Parse Avro records to
Strings into Collect end time
Feb 18, 2022 6:23:26 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:23:25.895Z: Fusing consumer Calculate
hashcode/WithKeys/AddKeys/Map into Parse Avro records to Strings
Feb 18, 2022 6:23:26 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:23:25.921Z: Fusing consumer Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey+Calculate
hashcode/Combine.perKey(Hashing)/Combine.GroupedValues/Partial into Calculate
hashcode/WithKeys/AddKeys/Map
Feb 18, 2022 6:23:26 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:23:25.954Z: Fusing consumer Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Reify into Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey+Calculate
hashcode/Combine.perKey(Hashing)/Combine.GroupedValues/Partial
Feb 18, 2022 6:23:26 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:23:25.991Z: Fusing consumer Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Write into Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Reify
Feb 18, 2022 6:23:26 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:23:26.023Z: Fusing consumer Calculate
hashcode/Combine.perKey(Hashing)/Combine.GroupedValues into Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Read
Feb 18, 2022 6:23:26 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:23:26.073Z: Fusing consumer Calculate
hashcode/Combine.perKey(Hashing)/Combine.GroupedValues/Extract into Calculate
hashcode/Combine.perKey(Hashing)/Combine.GroupedValues
Feb 18, 2022 6:23:28 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:23:26.105Z: Fusing consumer Calculate
hashcode/Values/Values/Map into Calculate
hashcode/Combine.perKey(Hashing)/Combine.GroupedValues/Extract
Feb 18, 2022 6:23:28 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:23:26.144Z: Fusing consumer Calculate
hashcode/ProduceDefault into Calculate hashcode/CreateVoid/Read(CreateSource)
Feb 18, 2022 6:23:28 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:23:26.178Z: Fusing consumer
PAssert$0/GroupGlobally/WithKeys/AddKeys/Map into
PAssert$0/GroupGlobally/Create.Values/Read(CreateSource)
Feb 18, 2022 6:23:28 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:23:26.203Z: Fusing consumer
PAssert$0/GroupGlobally/GroupByKey/Reify into
PAssert$0/GroupGlobally/WithKeys/AddKeys/Map
Feb 18, 2022 6:23:28 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:23:26.616Z: Executing operation Write Avro records to
files/Write/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Create
Feb 18, 2022 6:23:28 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:23:26.678Z: Starting 5 ****s in us-central1-a...
Feb 18, 2022 6:23:28 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:23:27.146Z: Finished operation Write Avro records to
files/Write/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Create
Feb 18, 2022 6:23:28 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:23:27.293Z: Executing operation Generate
sequence/Read(BoundedCountingSource)+Produce text lines+Produce Avro
records+Collect start time+Write Avro records to
files/Write/RewindowIntoGlobal/Window.Assign+Write Avro records to
files/Write/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles+Write Avro
records to
files/Write/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow)+Write
Avro records to
files/Write/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Reify+Write Avro
records to files/Write/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Write
Feb 18, 2022 6:23:32 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:23:31.070Z: Your project already contains 100
Dataflow-created metric descriptors, so new user metrics of the form
custom.googleapis.com/* will not be created. However, all user metrics are also
available in the metric dataflow.googleapis.com/job/user_counter. If you rely
on the custom metrics, you can delete old / unused metric descriptors. See
https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list
and
https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
Feb 18, 2022 6:24:11 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:24:10.699Z: Autoscaling: Raised the number of ****s to
5 based on the rate of progress in the currently running stage(s).
Feb 18, 2022 6:24:35 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:24:35.109Z: Workers have started successfully.
Feb 18, 2022 6:24:35 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:24:35.162Z: Workers have started successfully.
Feb 18, 2022 6:26:40 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:26:40.035Z: Finished operation Generate
sequence/Read(BoundedCountingSource)+Produce text lines+Produce Avro
records+Collect start time+Write Avro records to
files/Write/RewindowIntoGlobal/Window.Assign+Write Avro records to
files/Write/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles+Write Avro
records to
files/Write/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow)+Write
Avro records to
files/Write/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Reify+Write Avro
records to files/Write/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Write
Feb 18, 2022 6:26:40 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:26:40.143Z: Executing operation Write Avro records to
files/Write/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Close
Feb 18, 2022 6:26:40 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:26:40.204Z: Finished operation Write Avro records to
files/Write/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Close
Feb 18, 2022 6:26:40 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:26:40.269Z: Executing operation Write Avro records to
files/Write/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Read+Write Avro
records to
files/Write/WriteUnshardedBundlesToTempFiles/GroupUnwritten/GroupByWindow+Write
Avro records to
files/Write/WriteUnshardedBundlesToTempFiles/WriteUnwritten+Write Avro records
to files/Write/WriteUnshardedBundlesToTempFiles/DropShardNum+Write Avro records
to
files/Write/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow)
Feb 18, 2022 6:26:43 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:26:41.546Z: Finished operation Write Avro records to
files/Write/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Read+Write Avro
records to
files/Write/WriteUnshardedBundlesToTempFiles/GroupUnwritten/GroupByWindow+Write
Avro records to
files/Write/WriteUnshardedBundlesToTempFiles/WriteUnwritten+Write Avro records
to files/Write/WriteUnshardedBundlesToTempFiles/DropShardNum+Write Avro records
to
files/Write/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow)
Feb 18, 2022 6:26:43 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:26:41.668Z: Executing operation Write Avro records to
files/Write/WriteUnshardedBundlesToTempFiles/Flatten.PCollections/Unzipped-1
Feb 18, 2022 6:26:43 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:26:41.745Z: Finished operation Write Avro records to
files/Write/WriteUnshardedBundlesToTempFiles/Flatten.PCollections/Unzipped-1
Feb 18, 2022 6:26:43 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:26:41.873Z: Executing operation Write Avro records to
files/Write/GatherTempFileResults/View.AsList/CreateDataflowView
Feb 18, 2022 6:26:43 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:26:41.926Z: Finished operation Write Avro records to
files/Write/GatherTempFileResults/View.AsList/CreateDataflowView
Feb 18, 2022 6:26:43 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:26:42.042Z: Executing operation Write Avro records to
files/Write/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Create
Feb 18, 2022 6:26:43 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:26:42.743Z: Finished operation Write Avro records to
files/Write/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Create
Feb 18, 2022 6:26:43 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:26:42.886Z: Executing operation Write Avro records to
files/Write/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Create.Values/Read(CreateSource)+Write
Avro records to
files/Write/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Reify.ReifyView/ParDo(Anonymous)+Write
Avro records to
files/Write/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Values/Values/Map+Write
Avro records to files/Write/FinalizeTempFileBundles/Finalize+Write Avro
records to files/Write/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair with
random key+Write Avro records to
files/Write/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign+Write
Avro records to
files/Write/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify+Write
Avro records to
files/Write/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write
Feb 18, 2022 6:26:44 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:26:44.895Z: Finished operation Write Avro records to
files/Write/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Create.Values/Read(CreateSource)+Write
Avro records to
files/Write/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Reify.ReifyView/ParDo(Anonymous)+Write
Avro records to
files/Write/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Values/Values/Map+Write
Avro records to files/Write/FinalizeTempFileBundles/Finalize+Write Avro
records to files/Write/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair with
random key+Write Avro records to
files/Write/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign+Write
Avro records to
files/Write/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify+Write
Avro records to
files/Write/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write
Feb 18, 2022 6:26:46 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:26:44.973Z: Executing operation Write Avro records to
files/Write/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Close
Feb 18, 2022 6:26:46 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:26:45.025Z: Finished operation Write Avro records to
files/Write/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Close
Feb 18, 2022 6:26:46 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:26:45.097Z: Executing operation Match all
files/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Create
Feb 18, 2022 6:26:46 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:26:45.290Z: Finished operation Match all
files/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Create
Feb 18, 2022 6:26:46 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:26:45.402Z: Executing operation Write Avro records to
files/Write/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read+Write
Avro records to
files/Write/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow+Write
Avro records to
files/Write/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable+Write
Avro records to
files/Write/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Values/Values/Map+Collect
middle time+Values/Values/Map+Match all files/Match filepatterns+Match all
files/Reshuffle.ViaRandomKey/Pair with random key+Match all
files/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign+Match all
files/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify+Match all
files/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write
Feb 18, 2022 6:26:49 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:26:47.366Z: Finished operation Write Avro records to
files/Write/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read+Write
Avro records to
files/Write/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow+Write
Avro records to
files/Write/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable+Write
Avro records to
files/Write/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Values/Values/Map+Collect
middle time+Values/Values/Map+Match all files/Match filepatterns+Match all
files/Reshuffle.ViaRandomKey/Pair with random key+Match all
files/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign+Match all
files/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify+Match all
files/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write
Feb 18, 2022 6:26:49 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:26:47.483Z: Executing operation Match all
files/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Close
Feb 18, 2022 6:26:49 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:26:47.533Z: Finished operation Match all
files/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Close
Feb 18, 2022 6:26:49 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:26:47.599Z: Executing operation Read files/Read all via
FileBasedSource/Reshuffle/Reshuffle/GroupByKey/Create
Feb 18, 2022 6:26:49 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:26:47.863Z: Finished operation Read files/Read all via
FileBasedSource/Reshuffle/Reshuffle/GroupByKey/Create
Feb 18, 2022 6:26:49 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:26:48.006Z: Executing operation Match all
files/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read+Match all
files/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow+Match all
files/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable+Match all
files/Reshuffle.ViaRandomKey/Values/Values/Map+Read
matches/ParDo(ToReadableFile)+Read files/Read all via FileBasedSource/Split
into ranges+Read files/Read all via FileBasedSource/Reshuffle/Pair with random
key+Read files/Read all via
FileBasedSource/Reshuffle/Reshuffle/Window.Into()/Window.Assign+Read files/Read
all via FileBasedSource/Reshuffle/Reshuffle/GroupByKey/Reify+Read files/Read
all via FileBasedSource/Reshuffle/Reshuffle/GroupByKey/Write
Feb 18, 2022 6:26:49 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:26:48.447Z: Finished operation Match all
files/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read+Match all
files/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow+Match all
files/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable+Match all
files/Reshuffle.ViaRandomKey/Values/Values/Map+Read
matches/ParDo(ToReadableFile)+Read files/Read all via FileBasedSource/Split
into ranges+Read files/Read all via FileBasedSource/Reshuffle/Pair with random
key+Read files/Read all via
FileBasedSource/Reshuffle/Reshuffle/Window.Into()/Window.Assign+Read files/Read
all via FileBasedSource/Reshuffle/Reshuffle/GroupByKey/Reify+Read files/Read
all via FileBasedSource/Reshuffle/Reshuffle/GroupByKey/Write
Feb 18, 2022 6:26:49 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:26:48.509Z: Executing operation Read files/Read all via
FileBasedSource/Reshuffle/Reshuffle/GroupByKey/Close
Feb 18, 2022 6:26:49 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:26:48.569Z: Finished operation Read files/Read all via
FileBasedSource/Reshuffle/Reshuffle/GroupByKey/Close
Feb 18, 2022 6:26:49 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:26:48.634Z: Executing operation Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Create
Feb 18, 2022 6:26:49 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:26:48.839Z: Finished operation Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Create
Feb 18, 2022 6:26:49 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-02-18T18:26:48.988Z: Executing operation Read files/Read all via
FileBasedSource/Reshuffle/Reshuffle/GroupByKey/Read+Read files/Read all via
FileBasedSource/Reshuffle/Reshuffle/GroupByKey/GroupByWindow+Read files/Read
all via FileBasedSource/Reshuffle/Reshuffle/ExpandIterable+Read files/Read all
via FileBasedSource/Reshuffle/Values/Values/Map+Read files/Read all via
FileBasedSource/Read ranges+Collect end time+Parse Avro records to
Strings+Calculate hashcode/WithKeys/AddKeys/Map+Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey+Calculate
hashcode/Combine.perKey(Hashing)/Combine.GroupedValues/Partial+Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Reify+Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Write
Feb 18, 2022 6:27:28 PM
org.apache.beam.runners.dataflow.DataflowPipelineJob lambda$waitUntilFinish$0
WARNING: Job is already running in Google Cloud Platform, Ctrl-C will not
cancel it.
To cancel the job in the cloud, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel
--region=us-central1 2022-02-18_10_22_58-3243822149252866010
Remove shutdown hook failed
java.lang.IllegalStateException: Shutdown in progress
at
java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:82)
at java.lang.Runtime.removeShutdownHook(Runtime.java:239)
at
org.gradle.process.internal.shutdown.ShutdownHooks.removeShutdownHook(ShutdownHooks.java:38)
at
org.gradle.process.internal.DefaultExecHandle.setEndStateInfo(DefaultExecHandle.java:208)
at
org.gradle.process.internal.DefaultExecHandle.aborted(DefaultExecHandle.java:365)
at
org.gradle.process.internal.ExecHandleRunner.completed(ExecHandleRunner.java:108)
at
org.gradle.process.internal.ExecHandleRunner.run(ExecHandleRunner.java:84)
at
org.gradle.internal.operations.CurrentBuildOperationPreservingRunnable.run(CurrentBuildOperationPreservingRunnable.java:42)
at
org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
at
org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at
org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:61)
at java.lang.Thread.run(Thread.java:748)
Remove shutdown hook failed
java.lang.IllegalStateException: Shutdown in progress
at
java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:82)
at java.lang.Runtime.removeShutdownHook(Runtime.java:239)
at
org.gradle.process.internal.shutdown.ShutdownHooks.removeShutdownHook(ShutdownHooks.java:38)
at
org.gradle.process.internal.DefaultExecHandle.setEndStateInfo(DefaultExecHandle.java:208)
at
org.gradle.process.internal.DefaultExecHandle.aborted(DefaultExecHandle.java:365)
at
org.gradle.process.internal.ExecHandleRunner.completed(ExecHandleRunner.java:108)
at
org.gradle.process.internal.ExecHandleRunner.run(ExecHandleRunner.java:84)
at
org.gradle.internal.operations.CurrentBuildOperationPreservingRunnable.run(CurrentBuildOperationPreservingRunnable.java:42)
at
org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
at
org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at
org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:61)
at java.lang.Thread.run(Thread.java:748)
The message received from the daemon indicates that the daemon has disappeared.
Build request sent: Build{id=7c780d32-56cd-44dc-ae67-928923e8840c,
currentDir=<https://ci-beam.apache.org/job/beam_PerformanceTests_AvroIOIT_HDFS/ws/src}>
Attempting to read last messages from the daemon log...
Daemon pid: 3244220
log file: /home/jenkins/.gradle/daemon/7.3.2/daemon-3244220.out.log
----- Last 20 lines from daemon log file - daemon-3244220.out.log -----
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at
org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:61)
at java.lang.Thread.run(Thread.java:748)
Remove shutdown hook failed
java.lang.IllegalStateException: Shutdown in progress
at
java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:82)
at java.lang.Runtime.removeShutdownHook(Runtime.java:239)
at
org.gradle.process.internal.shutdown.ShutdownHooks.removeShutdownHook(ShutdownHooks.java:38)
at
org.gradle.process.internal.DefaultExecHandle.setEndStateInfo(DefaultExecHandle.java:208)
at
org.gradle.process.internal.DefaultExecHandle.aborted(DefaultExecHandle.java:365)
at
org.gradle.process.internal.ExecHandleRunner.completed(ExecHandleRunner.java:108)
at
org.gradle.process.internal.ExecHandleRunner.run(ExecHandleRunner.java:84)
at
org.gradle.internal.operations.CurrentBuildOperationPreservingRunnable.run(CurrentBuildOperationPreservingRunnable.java:42)
at
org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
at
org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at
org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:61)
at java.lang.Thread.run(Thread.java:748)
----- End of the daemon log -----
FAILURE: Build failed with an exception.
* What went wrong:
Gradle build daemon disappeared unexpectedly (it may have been killed or may
have crashed)
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --debug option to get more log output.
> Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
