See
<https://ci-beam.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/2237/display/redirect?page=changes>
Changes:
[Kamil Wasilewski] [BEAM-9889] Publish InfluxDB backup to public GCS bucket
[Kamil Wasilewski] [BEAM-9889] Populate local instance of InfluxDB with data
------------------------------------------
[...truncated 327.21 KB...]
Jul 15, 2020 12:58:30 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T12:58:29.104Z: Fusing consumer Match all
files/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write into Match all
files/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify
Jul 15, 2020 12:58:30 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T12:58:29.163Z: Fusing consumer Match all
files/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow into Match all
files/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read
Jul 15, 2020 12:58:30 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T12:58:29.213Z: Fusing consumer Match all
files/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable into Match all
files/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow
Jul 15, 2020 12:58:30 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T12:58:29.274Z: Fusing consumer Match all
files/Reshuffle.ViaRandomKey/Values/Values/Map into Match all
files/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable
Jul 15, 2020 12:58:30 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T12:58:29.337Z: Fusing consumer Read
matches/ParDo(ToReadableFile) into Match all
files/Reshuffle.ViaRandomKey/Values/Values/Map
Jul 15, 2020 12:58:30 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T12:58:29.400Z: Fusing consumer Read files/Read all via
FileBasedSource/Split into ranges into Read matches/ParDo(ToReadableFile)
Jul 15, 2020 12:58:30 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T12:58:29.458Z: Fusing consumer Read files/Read all via
FileBasedSource/Reshuffle/Pair with random key into Read files/Read all via
FileBasedSource/Split into ranges
Jul 15, 2020 12:58:30 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T12:58:29.514Z: Fusing consumer Read files/Read all via
FileBasedSource/Reshuffle/Reshuffle/Window.Into()/Window.Assign into Read
files/Read all via FileBasedSource/Reshuffle/Pair with random key
Jul 15, 2020 12:58:30 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T12:58:29.578Z: Fusing consumer Read files/Read all via
FileBasedSource/Reshuffle/Reshuffle/GroupByKey/Reify into Read files/Read all
via FileBasedSource/Reshuffle/Reshuffle/Window.Into()/Window.Assign
Jul 15, 2020 12:58:30 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T12:58:29.636Z: Fusing consumer Read files/Read all via
FileBasedSource/Reshuffle/Reshuffle/GroupByKey/Write into Read files/Read all
via FileBasedSource/Reshuffle/Reshuffle/GroupByKey/Reify
Jul 15, 2020 12:58:30 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T12:58:29.696Z: Fusing consumer Read files/Read all via
FileBasedSource/Reshuffle/Reshuffle/GroupByKey/GroupByWindow into Read
files/Read all via FileBasedSource/Reshuffle/Reshuffle/GroupByKey/Read
Jul 15, 2020 12:58:30 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T12:58:29.770Z: Fusing consumer Read files/Read all via
FileBasedSource/Reshuffle/Reshuffle/ExpandIterable into Read files/Read all via
FileBasedSource/Reshuffle/Reshuffle/GroupByKey/GroupByWindow
Jul 15, 2020 12:58:30 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T12:58:29.826Z: Fusing consumer Read files/Read all via
FileBasedSource/Reshuffle/Values/Values/Map into Read files/Read all via
FileBasedSource/Reshuffle/Reshuffle/ExpandIterable
Jul 15, 2020 12:58:30 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T12:58:29.885Z: Fusing consumer Read files/Read all via
FileBasedSource/Read ranges into Read files/Read all via
FileBasedSource/Reshuffle/Values/Values/Map
Jul 15, 2020 12:58:30 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T12:58:29.954Z: Fusing consumer Collect read end time into
Read files/Read all via FileBasedSource/Read ranges
Jul 15, 2020 12:58:30 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T12:58:30.011Z: Fusing consumer Calculate
hashcode/WithKeys/AddKeys/Map into Collect read end time
Jul 15, 2020 12:58:30 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T12:58:30.070Z: Fusing consumer Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey+Calculate
hashcode/Combine.perKey(Hashing)/Combine.GroupedValues/Partial into Calculate
hashcode/WithKeys/AddKeys/Map
Jul 15, 2020 12:58:30 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T12:58:30.117Z: Fusing consumer Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Reify into Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey+Calculate
hashcode/Combine.perKey(Hashing)/Combine.GroupedValues/Partial
Jul 15, 2020 12:58:30 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T12:58:30.160Z: Fusing consumer Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Write into Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Reify
Jul 15, 2020 12:58:30 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T12:58:30.195Z: Fusing consumer Calculate
hashcode/Combine.perKey(Hashing)/Combine.GroupedValues into Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Read
Jul 15, 2020 12:58:30 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T12:58:30.228Z: Fusing consumer Calculate
hashcode/Combine.perKey(Hashing)/Combine.GroupedValues/Extract into Calculate
hashcode/Combine.perKey(Hashing)/Combine.GroupedValues
Jul 15, 2020 12:58:30 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T12:58:30.267Z: Fusing consumer Calculate
hashcode/Values/Values/Map into Calculate
hashcode/Combine.perKey(Hashing)/Combine.GroupedValues/Extract
Jul 15, 2020 12:58:30 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T12:58:30.299Z: Fusing consumer Calculate
hashcode/ProduceDefault into Calculate hashcode/CreateVoid/Read(CreateSource)
Jul 15, 2020 12:58:32 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T12:58:30.335Z: Fusing consumer
PAssert$0/GroupGlobally/WithKeys/AddKeys/Map into
PAssert$0/GroupGlobally/Create.Values/Read(CreateSource)
Jul 15, 2020 12:58:32 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T12:58:30.359Z: Fusing consumer
PAssert$0/GroupGlobally/GroupByKey/Reify into
PAssert$0/GroupGlobally/WithKeys/AddKeys/Map
Jul 15, 2020 12:58:32 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T12:58:30.879Z: Executing operation Write content to
files/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Create
Jul 15, 2020 12:58:32 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T12:58:30.918Z: Executing operation Match all
files/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Create
Jul 15, 2020 12:58:32 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T12:58:30.949Z: Executing operation Read files/Read all via
FileBasedSource/Reshuffle/Reshuffle/GroupByKey/Create
Jul 15, 2020 12:58:32 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T12:58:30.962Z: Starting 5 ****s in us-central1-a...
Jul 15, 2020 12:58:32 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T12:58:30.987Z: Executing operation Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Create
Jul 15, 2020 12:58:32 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T12:58:31.010Z: Finished operation Write content to
files/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Create
Jul 15, 2020 12:58:32 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T12:58:31.019Z: Executing operation
PAssert$0/GroupGlobally/GroupByKey/Create
Jul 15, 2020 12:58:32 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T12:58:31.035Z: Finished operation Match all
files/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Create
Jul 15, 2020 12:58:32 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T12:58:31.035Z: Finished operation Read files/Read all via
FileBasedSource/Reshuffle/Reshuffle/GroupByKey/Create
Jul 15, 2020 12:58:32 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T12:58:31.060Z: Finished operation Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Create
Jul 15, 2020 12:58:32 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T12:58:31.064Z: Executing operation
View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Create
Jul 15, 2020 12:58:32 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T12:58:31.098Z: Finished operation
PAssert$0/GroupGlobally/GroupByKey/Create
Jul 15, 2020 12:58:32 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T12:58:31.104Z: Executing operation Write content to
files/WriteFiles/WriteShardedBundlesToTempFiles/GroupIntoShards/Create
Jul 15, 2020 12:58:32 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T12:58:31.133Z: Finished operation
View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Create
Jul 15, 2020 12:58:32 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T12:58:31.145Z: Executing operation
View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Create
Jul 15, 2020 12:58:32 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T12:58:31.176Z: Finished operation Write content to
files/WriteFiles/WriteShardedBundlesToTempFiles/GroupIntoShards/Create
Jul 15, 2020 12:58:32 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T12:58:31.223Z: Finished operation
View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Create
Jul 15, 2020 12:58:32 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T12:58:31.490Z: Executing operation
PAssert$0/GroupGlobally/Create.Values/Read(CreateSource)+PAssert$0/GroupGlobally/WithKeys/AddKeys/Map+PAssert$0/GroupGlobally/GroupByKey/Reify+PAssert$0/GroupGlobally/GroupByKey/Write
Jul 15, 2020 12:58:32 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T12:58:31.524Z: Executing operation Generate
sequence/Read(BoundedCountingSource)+Produce text lines+Collect write start
time+Write content to files/WriteFiles/RewindowIntoGlobal/Window.Assign+Write
content to
files/WriteFiles/WriteShardedBundlesToTempFiles/ApplyShardingKey+Write content
to files/WriteFiles/WriteShardedBundlesToTempFiles/GroupIntoShards/Reify+Write
content to files/WriteFiles/WriteShardedBundlesToTempFiles/GroupIntoShards/Write
Jul 15, 2020 12:58:35 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
WARNING: 2020-07-15T12:58:32.939Z: Your project already contains 100
Dataflow-created metric descriptors and Stackdriver will not create new
Dataflow custom metrics for this job. Each unique user-defined metric name
(independent of the DoFn in which it is defined) produces a new metric
descriptor. To delete old / unused metric descriptors see
https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list
and
https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
Jul 15, 2020 1:00:59 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T13:00:58.065Z: Workers have started successfully.
Jul 15, 2020 1:01:30 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T13:01:27.496Z: Autoscaling: Raised the number of ****s to
5 based on the rate of progress in the currently running stage(s).
Jul 15, 2020 1:01:30 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T13:01:28.308Z: Workers have started successfully.
Jul 15, 2020 1:01:46 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T13:01:44.182Z: Finished operation
PAssert$0/GroupGlobally/Create.Values/Read(CreateSource)+PAssert$0/GroupGlobally/WithKeys/AddKeys/Map+PAssert$0/GroupGlobally/GroupByKey/Reify+PAssert$0/GroupGlobally/GroupByKey/Write
Jul 15, 2020 1:02:03 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T13:02:02.583Z: Finished operation Generate
sequence/Read(BoundedCountingSource)+Produce text lines+Collect write start
time+Write content to files/WriteFiles/RewindowIntoGlobal/Window.Assign+Write
content to
files/WriteFiles/WriteShardedBundlesToTempFiles/ApplyShardingKey+Write content
to files/WriteFiles/WriteShardedBundlesToTempFiles/GroupIntoShards/Reify+Write
content to files/WriteFiles/WriteShardedBundlesToTempFiles/GroupIntoShards/Write
Jul 15, 2020 1:02:03 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T13:02:02.699Z: Executing operation Write content to
files/WriteFiles/WriteShardedBundlesToTempFiles/GroupIntoShards/Close
Jul 15, 2020 1:02:03 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T13:02:02.777Z: Finished operation Write content to
files/WriteFiles/WriteShardedBundlesToTempFiles/GroupIntoShards/Close
Jul 15, 2020 1:02:03 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T13:02:02.857Z: Executing operation Write content to
files/WriteFiles/WriteShardedBundlesToTempFiles/GroupIntoShards/Read+Write
content to
files/WriteFiles/WriteShardedBundlesToTempFiles/GroupIntoShards/GroupByWindow+Write
content to
files/WriteFiles/WriteShardedBundlesToTempFiles/WriteShardsIntoTempFiles+Write
content to
files/WriteFiles/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow)
Jul 15, 2020 1:03:07 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T13:03:06.330Z: Finished operation Write content to
files/WriteFiles/WriteShardedBundlesToTempFiles/GroupIntoShards/Read+Write
content to
files/WriteFiles/WriteShardedBundlesToTempFiles/GroupIntoShards/GroupByWindow+Write
content to
files/WriteFiles/WriteShardedBundlesToTempFiles/WriteShardsIntoTempFiles+Write
content to
files/WriteFiles/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow)
Jul 15, 2020 1:03:07 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T13:03:06.561Z: Executing operation Write content to
files/WriteFiles/GatherTempFileResults/View.AsList/CreateDataflowView
Jul 15, 2020 1:03:07 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T13:03:06.647Z: Finished operation Write content to
files/WriteFiles/GatherTempFileResults/View.AsList/CreateDataflowView
Jul 15, 2020 1:03:07 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T13:03:06.855Z: Executing operation Write content to
files/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Create.Values/Read(CreateSource)+Write
content to
files/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Reify.ReifyView/ParDo(Anonymous)+Write
content to
files/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Values/Values/Map+Write
content to files/WriteFiles/FinalizeTempFileBundles/Finalize+Write content to
files/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair with
random key+Write content to
files/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign+Write
content to
files/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify+Write
content to
files/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write
Jul 15, 2020 1:03:17 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T13:03:15.693Z: Finished operation Write content to
files/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Create.Values/Read(CreateSource)+Write
content to
files/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Reify.ReifyView/ParDo(Anonymous)+Write
content to
files/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Values/Values/Map+Write
content to files/WriteFiles/FinalizeTempFileBundles/Finalize+Write content to
files/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair with
random key+Write content to
files/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign+Write
content to
files/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify+Write
content to
files/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write
Jul 15, 2020 1:03:17 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T13:03:15.923Z: Executing operation Write content to
files/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Close
Jul 15, 2020 1:03:17 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T13:03:16.172Z: Finished operation Write content to
files/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Close
Jul 15, 2020 1:03:17 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T13:03:16.388Z: Executing operation Write content to
files/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read+Write
content to
files/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow+Write
content to
files/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable+Write
content to
files/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Values/Values/Map+Values/Values/Map+Collect
write end time+Match all files/Match filepatterns+Match all
files/Reshuffle.ViaRandomKey/Pair with random key+Match all
files/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign+Match all
files/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify+Match all
files/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write
Jul 15, 2020 1:03:21 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T13:03:20.333Z: Finished operation Write content to
files/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read+Write
content to
files/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow+Write
content to
files/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable+Write
content to
files/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Values/Values/Map+Values/Values/Map+Collect
write end time+Match all files/Match filepatterns+Match all
files/Reshuffle.ViaRandomKey/Pair with random key+Match all
files/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign+Match all
files/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify+Match all
files/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write
Jul 15, 2020 1:03:21 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T13:03:20.457Z: Executing operation Match all
files/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Close
Jul 15, 2020 1:03:21 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T13:03:20.525Z: Finished operation Match all
files/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Close
Jul 15, 2020 1:03:21 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T13:03:20.618Z: Executing operation Match all
files/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read+Match all
files/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow+Match all
files/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable+Match all
files/Reshuffle.ViaRandomKey/Values/Values/Map+Read
matches/ParDo(ToReadableFile)+Read files/Read all via FileBasedSource/Split
into ranges+Read files/Read all via FileBasedSource/Reshuffle/Pair with random
key+Read files/Read all via
FileBasedSource/Reshuffle/Reshuffle/Window.Into()/Window.Assign+Read files/Read
all via FileBasedSource/Reshuffle/Reshuffle/GroupByKey/Reify+Read files/Read
all via FileBasedSource/Reshuffle/Reshuffle/GroupByKey/Write
Jul 15, 2020 1:03:24 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T13:03:22.810Z: Finished operation Match all
files/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read+Match all
files/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow+Match all
files/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable+Match all
files/Reshuffle.ViaRandomKey/Values/Values/Map+Read
matches/ParDo(ToReadableFile)+Read files/Read all via FileBasedSource/Split
into ranges+Read files/Read all via FileBasedSource/Reshuffle/Pair with random
key+Read files/Read all via
FileBasedSource/Reshuffle/Reshuffle/Window.Into()/Window.Assign+Read files/Read
all via FileBasedSource/Reshuffle/Reshuffle/GroupByKey/Reify+Read files/Read
all via FileBasedSource/Reshuffle/Reshuffle/GroupByKey/Write
Jul 15, 2020 1:03:24 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T13:03:22.907Z: Executing operation Read files/Read all via
FileBasedSource/Reshuffle/Reshuffle/GroupByKey/Close
Jul 15, 2020 1:03:24 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T13:03:22.969Z: Finished operation Read files/Read all via
FileBasedSource/Reshuffle/Reshuffle/GroupByKey/Close
Jul 15, 2020 1:03:24 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T13:03:23.050Z: Executing operation Read files/Read all via
FileBasedSource/Reshuffle/Reshuffle/GroupByKey/Read+Read files/Read all via
FileBasedSource/Reshuffle/Reshuffle/GroupByKey/GroupByWindow+Read files/Read
all via FileBasedSource/Reshuffle/Reshuffle/ExpandIterable+Read files/Read all
via FileBasedSource/Reshuffle/Values/Values/Map+Read files/Read all via
FileBasedSource/Read ranges+Collect read end time+Calculate
hashcode/WithKeys/AddKeys/Map+Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey+Calculate
hashcode/Combine.perKey(Hashing)/Combine.GroupedValues/Partial+Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Reify+Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Write
Jul 15, 2020 1:03:52 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T13:03:50.185Z: Finished operation Read files/Read all via
FileBasedSource/Reshuffle/Reshuffle/GroupByKey/Read+Read files/Read all via
FileBasedSource/Reshuffle/Reshuffle/GroupByKey/GroupByWindow+Read files/Read
all via FileBasedSource/Reshuffle/Reshuffle/ExpandIterable+Read files/Read all
via FileBasedSource/Reshuffle/Values/Values/Map+Read files/Read all via
FileBasedSource/Read ranges+Collect read end time+Calculate
hashcode/WithKeys/AddKeys/Map+Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey+Calculate
hashcode/Combine.perKey(Hashing)/Combine.GroupedValues/Partial+Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Reify+Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Write
Jul 15, 2020 1:03:52 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T13:03:50.265Z: Executing operation Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Close
Jul 15, 2020 1:03:52 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T13:03:50.324Z: Finished operation Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Close
Jul 15, 2020 1:03:52 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T13:03:50.389Z: Executing operation Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Read+Calculate
hashcode/Combine.perKey(Hashing)/Combine.GroupedValues+Calculate
hashcode/Combine.perKey(Hashing)/Combine.GroupedValues/Extract+Calculate
hashcode/Values/Values/Map+View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map+View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey+View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Partial+View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Reify+View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Write+PAssert$0/GroupGlobally/Reify.Window/ParDo(Anonymous)+PAssert$0/GroupGlobally/ParDo(ToSingletonIterables)+PAssert$0/GroupGlobally/WithKeys/AddKeys/Map+PAssert$0/GroupGlobally/GroupByKey/Reify+PAssert$0/GroupGlobally/GroupByKey/Write+Calculate
hashcode/View.AsIterable/ParDo(ToIsmRecordForGlobalWindow)
Jul 15, 2020 1:03:54 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T13:03:53.669Z: Finished operation Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Read+Calculate
hashcode/Combine.perKey(Hashing)/Combine.GroupedValues+Calculate
hashcode/Combine.perKey(Hashing)/Combine.GroupedValues/Extract+Calculate
hashcode/Values/Values/Map+View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map+View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey+View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Partial+View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Reify+View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Write+PAssert$0/GroupGlobally/Reify.Window/ParDo(Anonymous)+PAssert$0/GroupGlobally/ParDo(ToSingletonIterables)+PAssert$0/GroupGlobally/WithKeys/AddKeys/Map+PAssert$0/GroupGlobally/GroupByKey/Reify+PAssert$0/GroupGlobally/GroupByKey/Write+Calculate
hashcode/View.AsIterable/ParDo(ToIsmRecordForGlobalWindow)
Jul 15, 2020 1:03:54 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T13:03:54.003Z: Executing operation Calculate
hashcode/View.AsIterable/CreateDataflowView
Jul 15, 2020 1:03:54 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T13:03:54.071Z: Finished operation Calculate
hashcode/View.AsIterable/CreateDataflowView
Jul 15, 2020 1:03:54 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T13:03:54.218Z: Executing operation Calculate
hashcode/CreateVoid/Read(CreateSource)+Calculate
hashcode/ProduceDefault+PAssert$0/GroupGlobally/Reify.Window/ParDo(Anonymous)+PAssert$0/GroupGlobally/ParDo(ToSingletonIterables)+PAssert$0/GroupGlobally/WithKeys/AddKeys/Map+PAssert$0/GroupGlobally/GroupByKey/Reify+PAssert$0/GroupGlobally/GroupByKey/Write+View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map+View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey+View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Partial+View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Reify+View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Write
Jul 15, 2020 1:03:54 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T13:03:54.853Z: Finished operation Calculate
hashcode/CreateVoid/Read(CreateSource)+Calculate
hashcode/ProduceDefault+PAssert$0/GroupGlobally/Reify.Window/ParDo(Anonymous)+PAssert$0/GroupGlobally/ParDo(ToSingletonIterables)+PAssert$0/GroupGlobally/WithKeys/AddKeys/Map+PAssert$0/GroupGlobally/GroupByKey/Reify+PAssert$0/GroupGlobally/GroupByKey/Write+View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map+View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey+View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Partial+View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Reify+View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Write
Jul 15, 2020 1:03:56 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T13:03:54.928Z: Executing operation
View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Close
Jul 15, 2020 1:03:56 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T13:03:54.976Z: Executing operation
PAssert$0/GroupGlobally/GroupByKey/Close
Jul 15, 2020 1:03:56 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T13:03:54.988Z: Finished operation
View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Close
Jul 15, 2020 1:03:56 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T13:03:55.022Z: Finished operation
PAssert$0/GroupGlobally/GroupByKey/Close
Jul 15, 2020 1:03:56 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T13:03:55.062Z: Executing operation
View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Read+View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues+View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Extract+View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map+View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)+View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
Jul 15, 2020 1:03:56 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T13:03:55.100Z: Executing operation
PAssert$0/GroupGlobally/GroupByKey/Read+PAssert$0/GroupGlobally/GroupByKey/GroupByWindow+PAssert$0/GroupGlobally/Values/Values/Map+PAssert$0/GroupGlobally/ParDo(Concat)+PAssert$0/GetPane/Map+PAssert$0/RunChecks+PAssert$0/VerifyAssertions/ParDo(DefaultConclude)
Jul 15, 2020 1:03:56 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T13:03:55.717Z: Finished operation
PAssert$0/GroupGlobally/GroupByKey/Read+PAssert$0/GroupGlobally/GroupByKey/GroupByWindow+PAssert$0/GroupGlobally/Values/Values/Map+PAssert$0/GroupGlobally/ParDo(Concat)+PAssert$0/GetPane/Map+PAssert$0/RunChecks+PAssert$0/VerifyAssertions/ParDo(DefaultConclude)
Jul 15, 2020 1:03:56 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T13:03:55.750Z: Finished operation
View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Read+View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues+View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Extract+View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map+View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)+View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
Jul 15, 2020 1:03:56 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T13:03:55.871Z: Executing operation
View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Close
Jul 15, 2020 1:03:56 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T13:03:55.927Z: Finished operation
View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Close
Jul 15, 2020 1:03:56 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-15T13:03:56.045Z: Executing operation
View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read+View.AsSingleton/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow)
Jul 15, 2020 1:03:59 PM
org.apache.beam.runners.dataflow.DataflowPipelineJob lambda$waitUntilFinish$0
WARNING: Job is already running in Google Cloud Platform, Ctrl-C will not
cancel it.
To cancel the job in the cloud, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel
--region=us-central1 2020-07-15_05_58_09-1437552358515980780
org.apache.beam.sdk.io.text.TextIOIT > writeThenReadAll SKIPPED
> Task :sdks:java:io:file-based-io-tests:integrationTest FAILED
:sdks:java:io:file-based-io-tests:integrationTest (Thread[Execution **** for
':' Thread 8,5,main]) completed. Took 6 mins 2.737 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':sdks:java:io:file-based-io-tests:integrationTest'.
> Process 'Gradle Test Executor 3' finished with non-zero exit value 143
This problem might be caused by incorrect test process configuration.
Please refer to the test execution section in the User Manual at
https://docs.gradle.org/5.2.1/userguide/java_testing.html#sec:test_execution
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to
get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with
Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See
https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 6m 41s
95 actionable tasks: 60 executed, 35 from cache
Publishing build scan...
The message received from the daemon indicates that the daemon has disappeared.
Build request sent: Build{id=ce1668af-ebb6-4503-92f2-0e3670426549,
currentDir=<https://ci-beam.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/src}>
Attempting to read last messages from the daemon log...
Daemon pid: 21478
log file: /home/jenkins/.gradle/daemon/5.2.1/daemon-21478.out.log
----- Last 20 lines from daemon log file - daemon-21478.out.log -----
* What went wrong:
Execution failed for task ':sdks:java:io:file-based-io-tests:integrationTest'.
> Process 'Gradle Test Executor 3' finished with non-zero exit value 143
This problem might be caused by incorrect test process configuration.
Please refer to the test execution section in the User Manual at
https://docs.gradle.org/5.2.1/userguide/java_testing.html#sec:test_execution
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to
get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with
Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See
https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 6m 41s
95 actionable tasks: 60 executed, 35 from cache
Publishing build scan...
Daemon vm is shutting down... The daemon has exited normally or was terminated
in response to a user interrupt.
----- End of the daemon log -----
FAILURE: Build failed with an exception.
* What went wrong:
Gradle build daemon disappeared unexpectedly (it may have been killed or may
have crashed)
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to
get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
