See
<https://ci-beam.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/2600/display/redirect?page=changes>
Changes:
[rohde.samuel] Add ability for HotKeyLogger to log a key.
[noreply] [BEAM-10861]Adds PubSub Runner API encoding to Read/Write transforms
[noreply] [BEAM-8251] Add worker_region and worker_zone options (#13072)
[noreply] [BEAM-10720] Finish implementing StringMethods (cat, repeat) (#12705)
[noreply] [BEAM-10009] Document micros_instant (#12921)
[noreply] [BEAM-10987] Skip test_with_main_session on windows (#13099)
------------------------------------------
[...truncated 310.05 KB...]
Oct 14, 2020 12:58:15 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T00:58:14.944Z: Fusing consumer Read files/Read all via
FileBasedSource/Reshuffle/Reshuffle/Window.Into()/Window.Assign into Read
files/Read all via FileBasedSource/Reshuffle/Pair with random key
Oct 14, 2020 12:58:15 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T00:58:15.022Z: Fusing consumer Read files/Read all via
FileBasedSource/Reshuffle/Reshuffle/GroupByKey/Reify into Read files/Read all
via FileBasedSource/Reshuffle/Reshuffle/Window.Into()/Window.Assign
Oct 14, 2020 12:58:15 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T00:58:15.050Z: Fusing consumer Read files/Read all via
FileBasedSource/Reshuffle/Reshuffle/GroupByKey/Write into Read files/Read all
via FileBasedSource/Reshuffle/Reshuffle/GroupByKey/Reify
Oct 14, 2020 12:58:15 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T00:58:15.081Z: Fusing consumer Read files/Read all via
FileBasedSource/Reshuffle/Reshuffle/GroupByKey/GroupByWindow into Read
files/Read all via FileBasedSource/Reshuffle/Reshuffle/GroupByKey/Read
Oct 14, 2020 12:58:15 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T00:58:15.105Z: Fusing consumer Read files/Read all via
FileBasedSource/Reshuffle/Reshuffle/ExpandIterable into Read files/Read all via
FileBasedSource/Reshuffle/Reshuffle/GroupByKey/GroupByWindow
Oct 14, 2020 12:58:15 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T00:58:15.140Z: Fusing consumer Read files/Read all via
FileBasedSource/Reshuffle/Values/Values/Map into Read files/Read all via
FileBasedSource/Reshuffle/Reshuffle/ExpandIterable
Oct 14, 2020 12:58:15 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T00:58:15.166Z: Fusing consumer Read files/Read all via
FileBasedSource/Read ranges into Read files/Read all via
FileBasedSource/Reshuffle/Values/Values/Map
Oct 14, 2020 12:58:15 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T00:58:15.201Z: Fusing consumer Collect read end time into
Read files/Read all via FileBasedSource/Read ranges
Oct 14, 2020 12:58:15 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T00:58:15.236Z: Fusing consumer Calculate
hashcode/WithKeys/AddKeys/Map into Collect read end time
Oct 14, 2020 12:58:15 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T00:58:15.261Z: Fusing consumer Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey+Calculate
hashcode/Combine.perKey(Hashing)/Combine.GroupedValues/Partial into Calculate
hashcode/WithKeys/AddKeys/Map
Oct 14, 2020 12:58:15 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T00:58:15.297Z: Fusing consumer Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Reify into Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey+Calculate
hashcode/Combine.perKey(Hashing)/Combine.GroupedValues/Partial
Oct 14, 2020 12:58:15 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T00:58:15.333Z: Fusing consumer Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Write into Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Reify
Oct 14, 2020 12:58:15 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T00:58:15.368Z: Fusing consumer Calculate
hashcode/Combine.perKey(Hashing)/Combine.GroupedValues into Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Read
Oct 14, 2020 12:58:15 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T00:58:15.404Z: Fusing consumer Calculate
hashcode/Combine.perKey(Hashing)/Combine.GroupedValues/Extract into Calculate
hashcode/Combine.perKey(Hashing)/Combine.GroupedValues
Oct 14, 2020 12:58:15 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T00:58:15.439Z: Fusing consumer Calculate
hashcode/Values/Values/Map into Calculate
hashcode/Combine.perKey(Hashing)/Combine.GroupedValues/Extract
Oct 14, 2020 12:58:15 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T00:58:15.477Z: Fusing consumer Calculate
hashcode/ProduceDefault into Calculate hashcode/CreateVoid/Read(CreateSource)
Oct 14, 2020 12:58:15 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T00:58:15.510Z: Fusing consumer
PAssert$0/GroupGlobally/WithKeys/AddKeys/Map into
PAssert$0/GroupGlobally/Create.Values/Read(CreateSource)
Oct 14, 2020 12:58:15 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T00:58:15.534Z: Fusing consumer
PAssert$0/GroupGlobally/GroupByKey/Reify into
PAssert$0/GroupGlobally/WithKeys/AddKeys/Map
Oct 14, 2020 12:58:18 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T00:58:15.931Z: Executing operation Write content to
files/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Create
Oct 14, 2020 12:58:18 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T00:58:15.966Z: Executing operation Match all
files/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Create
Oct 14, 2020 12:58:18 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T00:58:16.001Z: Executing operation Read files/Read all via
FileBasedSource/Reshuffle/Reshuffle/GroupByKey/Create
Oct 14, 2020 12:58:18 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T00:58:16.001Z: Starting 5 ****s in us-central1-c...
Oct 14, 2020 12:58:18 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T00:58:16.037Z: Executing operation Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Create
Oct 14, 2020 12:58:18 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T00:58:16.047Z: Finished operation Write content to
files/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Create
Oct 14, 2020 12:58:18 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T00:58:16.063Z: Finished operation Read files/Read all via
FileBasedSource/Reshuffle/Reshuffle/GroupByKey/Create
Oct 14, 2020 12:58:18 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T00:58:16.063Z: Finished operation Match all
files/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Create
Oct 14, 2020 12:58:18 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T00:58:16.080Z: Executing operation
PAssert$0/GroupGlobally/GroupByKey/Create
Oct 14, 2020 12:58:18 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T00:58:16.094Z: Finished operation Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Create
Oct 14, 2020 12:58:18 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T00:58:16.109Z: Executing operation
View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Create
Oct 14, 2020 12:58:18 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T00:58:16.131Z: Finished operation
PAssert$0/GroupGlobally/GroupByKey/Create
Oct 14, 2020 12:58:18 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T00:58:16.143Z: Executing operation Write content to
files/WriteFiles/WriteShardedBundlesToTempFiles/GroupIntoShards/Create
Oct 14, 2020 12:58:18 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T00:58:16.167Z: Finished operation
View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Create
Oct 14, 2020 12:58:18 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T00:58:16.180Z: Executing operation
View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Create
Oct 14, 2020 12:58:18 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T00:58:16.205Z: Finished operation Write content to
files/WriteFiles/WriteShardedBundlesToTempFiles/GroupIntoShards/Create
Oct 14, 2020 12:58:18 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T00:58:16.246Z: Finished operation
View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Create
Oct 14, 2020 12:58:18 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T00:58:16.512Z: Executing operation
PAssert$0/GroupGlobally/Create.Values/Read(CreateSource)+PAssert$0/GroupGlobally/WithKeys/AddKeys/Map+PAssert$0/GroupGlobally/GroupByKey/Reify+PAssert$0/GroupGlobally/GroupByKey/Write
Oct 14, 2020 12:58:18 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T00:58:16.548Z: Executing operation Generate
sequence/Read(BoundedCountingSource)+Produce text lines+Collect write start
time+Write content to files/WriteFiles/RewindowIntoGlobal/Window.Assign+Write
content to
files/WriteFiles/WriteShardedBundlesToTempFiles/ApplyShardingKey+Write content
to files/WriteFiles/WriteShardedBundlesToTempFiles/GroupIntoShards/Reify+Write
content to files/WriteFiles/WriteShardedBundlesToTempFiles/GroupIntoShards/Write
Oct 14, 2020 12:58:34 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T00:58:33.035Z: Your project already contains 100
Dataflow-created metric descriptors, so new user metrics of the form
custom.googleapis.com/* will not be created. However, all user metrics are also
available in the metric dataflow.googleapis.com/job/user_counter. If you rely
on the custom metrics, you can delete old / unused metric descriptors. See
https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list
and
https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
Oct 14, 2020 12:58:42 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T00:58:41.236Z: Autoscaling: Raised the number of ****s to
5 based on the rate of progress in the currently running stage(s).
Oct 14, 2020 12:59:01 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T00:59:00.683Z: Workers have started successfully.
Oct 14, 2020 12:59:01 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T00:59:00.736Z: Workers have started successfully.
Oct 14, 2020 12:59:44 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T00:59:42.853Z: Finished operation
PAssert$0/GroupGlobally/Create.Values/Read(CreateSource)+PAssert$0/GroupGlobally/WithKeys/AddKeys/Map+PAssert$0/GroupGlobally/GroupByKey/Reify+PAssert$0/GroupGlobally/GroupByKey/Write
Oct 14, 2020 1:00:02 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T01:00:02.180Z: Finished operation Generate
sequence/Read(BoundedCountingSource)+Produce text lines+Collect write start
time+Write content to files/WriteFiles/RewindowIntoGlobal/Window.Assign+Write
content to
files/WriteFiles/WriteShardedBundlesToTempFiles/ApplyShardingKey+Write content
to files/WriteFiles/WriteShardedBundlesToTempFiles/GroupIntoShards/Reify+Write
content to files/WriteFiles/WriteShardedBundlesToTempFiles/GroupIntoShards/Write
Oct 14, 2020 1:00:02 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T01:00:02.263Z: Executing operation Write content to
files/WriteFiles/WriteShardedBundlesToTempFiles/GroupIntoShards/Close
Oct 14, 2020 1:00:02 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T01:00:02.329Z: Finished operation Write content to
files/WriteFiles/WriteShardedBundlesToTempFiles/GroupIntoShards/Close
Oct 14, 2020 1:00:02 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T01:00:02.411Z: Executing operation Write content to
files/WriteFiles/WriteShardedBundlesToTempFiles/GroupIntoShards/Read+Write
content to
files/WriteFiles/WriteShardedBundlesToTempFiles/GroupIntoShards/GroupByWindow+Write
content to
files/WriteFiles/WriteShardedBundlesToTempFiles/WriteShardsIntoTempFiles+Write
content to
files/WriteFiles/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow)
Oct 14, 2020 1:01:15 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T01:01:14.226Z: Finished operation Write content to
files/WriteFiles/WriteShardedBundlesToTempFiles/GroupIntoShards/Read+Write
content to
files/WriteFiles/WriteShardedBundlesToTempFiles/GroupIntoShards/GroupByWindow+Write
content to
files/WriteFiles/WriteShardedBundlesToTempFiles/WriteShardsIntoTempFiles+Write
content to
files/WriteFiles/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow)
Oct 14, 2020 1:01:15 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T01:01:14.383Z: Executing operation Write content to
files/WriteFiles/GatherTempFileResults/View.AsList/CreateDataflowView
Oct 14, 2020 1:01:15 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T01:01:14.475Z: Finished operation Write content to
files/WriteFiles/GatherTempFileResults/View.AsList/CreateDataflowView
Oct 14, 2020 1:01:15 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T01:01:14.642Z: Executing operation Write content to
files/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Create.Values/Read(CreateSource)+Write
content to
files/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Reify.ReifyView/ParDo(Anonymous)+Write
content to
files/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Values/Values/Map+Write
content to files/WriteFiles/FinalizeTempFileBundles/Finalize+Write content to
files/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair with
random key+Write content to
files/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign+Write
content to
files/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify+Write
content to
files/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write
Oct 14, 2020 1:01:23 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T01:01:22.936Z: Finished operation Write content to
files/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Create.Values/Read(CreateSource)+Write
content to
files/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Reify.ReifyView/ParDo(Anonymous)+Write
content to
files/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Values/Values/Map+Write
content to files/WriteFiles/FinalizeTempFileBundles/Finalize+Write content to
files/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair with
random key+Write content to
files/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign+Write
content to
files/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify+Write
content to
files/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write
Oct 14, 2020 1:01:23 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T01:01:23.207Z: Executing operation Write content to
files/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Close
Oct 14, 2020 1:01:23 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T01:01:23.336Z: Finished operation Write content to
files/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Close
Oct 14, 2020 1:01:23 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T01:01:23.406Z: Executing operation Write content to
files/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read+Write
content to
files/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow+Write
content to
files/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable+Write
content to
files/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Values/Values/Map+Values/Values/Map+Collect
write end time+Match all files/Match filepatterns+Match all
files/Reshuffle.ViaRandomKey/Pair with random key+Match all
files/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign+Match all
files/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify+Match all
files/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write
Oct 14, 2020 1:01:28 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T01:01:26.723Z: Finished operation Write content to
files/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read+Write
content to
files/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow+Write
content to
files/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable+Write
content to
files/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Values/Values/Map+Values/Values/Map+Collect
write end time+Match all files/Match filepatterns+Match all
files/Reshuffle.ViaRandomKey/Pair with random key+Match all
files/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign+Match all
files/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify+Match all
files/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write
Oct 14, 2020 1:01:28 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T01:01:27.123Z: Executing operation Match all
files/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Close
Oct 14, 2020 1:01:28 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T01:01:27.188Z: Finished operation Match all
files/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Close
Oct 14, 2020 1:01:28 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T01:01:27.287Z: Executing operation Match all
files/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read+Match all
files/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow+Match all
files/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable+Match all
files/Reshuffle.ViaRandomKey/Values/Values/Map+Read
matches/ParDo(ToReadableFile)+Read files/Read all via FileBasedSource/Split
into ranges+Read files/Read all via FileBasedSource/Reshuffle/Pair with random
key+Read files/Read all via
FileBasedSource/Reshuffle/Reshuffle/Window.Into()/Window.Assign+Read files/Read
all via FileBasedSource/Reshuffle/Reshuffle/GroupByKey/Reify+Read files/Read
all via FileBasedSource/Reshuffle/Reshuffle/GroupByKey/Write
Oct 14, 2020 1:01:29 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T01:01:28.916Z: Finished operation Match all
files/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read+Match all
files/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow+Match all
files/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable+Match all
files/Reshuffle.ViaRandomKey/Values/Values/Map+Read
matches/ParDo(ToReadableFile)+Read files/Read all via FileBasedSource/Split
into ranges+Read files/Read all via FileBasedSource/Reshuffle/Pair with random
key+Read files/Read all via
FileBasedSource/Reshuffle/Reshuffle/Window.Into()/Window.Assign+Read files/Read
all via FileBasedSource/Reshuffle/Reshuffle/GroupByKey/Reify+Read files/Read
all via FileBasedSource/Reshuffle/Reshuffle/GroupByKey/Write
Oct 14, 2020 1:01:29 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T01:01:28.986Z: Executing operation Read files/Read all via
FileBasedSource/Reshuffle/Reshuffle/GroupByKey/Close
Oct 14, 2020 1:01:29 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T01:01:29.198Z: Finished operation Read files/Read all via
FileBasedSource/Reshuffle/Reshuffle/GroupByKey/Close
Oct 14, 2020 1:01:31 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T01:01:29.411Z: Executing operation Read files/Read all via
FileBasedSource/Reshuffle/Reshuffle/GroupByKey/Read+Read files/Read all via
FileBasedSource/Reshuffle/Reshuffle/GroupByKey/GroupByWindow+Read files/Read
all via FileBasedSource/Reshuffle/Reshuffle/ExpandIterable+Read files/Read all
via FileBasedSource/Reshuffle/Values/Values/Map+Read files/Read all via
FileBasedSource/Read ranges+Collect read end time+Calculate
hashcode/WithKeys/AddKeys/Map+Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey+Calculate
hashcode/Combine.perKey(Hashing)/Combine.GroupedValues/Partial+Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Reify+Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Write
Oct 14, 2020 1:01:50 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T01:01:49.150Z: Finished operation Read files/Read all via
FileBasedSource/Reshuffle/Reshuffle/GroupByKey/Read+Read files/Read all via
FileBasedSource/Reshuffle/Reshuffle/GroupByKey/GroupByWindow+Read files/Read
all via FileBasedSource/Reshuffle/Reshuffle/ExpandIterable+Read files/Read all
via FileBasedSource/Reshuffle/Values/Values/Map+Read files/Read all via
FileBasedSource/Read ranges+Collect read end time+Calculate
hashcode/WithKeys/AddKeys/Map+Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey+Calculate
hashcode/Combine.perKey(Hashing)/Combine.GroupedValues/Partial+Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Reify+Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Write
Oct 14, 2020 1:01:50 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T01:01:49.255Z: Executing operation Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Close
Oct 14, 2020 1:01:50 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T01:01:49.320Z: Finished operation Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Close
Oct 14, 2020 1:01:50 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T01:01:49.389Z: Executing operation Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Read+Calculate
hashcode/Combine.perKey(Hashing)/Combine.GroupedValues+Calculate
hashcode/Combine.perKey(Hashing)/Combine.GroupedValues/Extract+Calculate
hashcode/Values/Values/Map+View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map+View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey+View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Partial+View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Reify+View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Write+PAssert$0/GroupGlobally/Reify.Window/ParDo(Anonymous)+PAssert$0/GroupGlobally/ParDo(ToSingletonIterables)+PAssert$0/GroupGlobally/WithKeys/AddKeys/Map+PAssert$0/GroupGlobally/GroupByKey/Reify+PAssert$0/GroupGlobally/GroupByKey/Write+Calculate
hashcode/View.AsIterable/ParDo(ToIsmRecordForGlobalWindow)
Oct 14, 2020 1:01:52 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T01:01:51.291Z: Finished operation Calculate
hashcode/Combine.perKey(Hashing)/GroupByKey/Read+Calculate
hashcode/Combine.perKey(Hashing)/Combine.GroupedValues+Calculate
hashcode/Combine.perKey(Hashing)/Combine.GroupedValues/Extract+Calculate
hashcode/Values/Values/Map+View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map+View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey+View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Partial+View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Reify+View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Write+PAssert$0/GroupGlobally/Reify.Window/ParDo(Anonymous)+PAssert$0/GroupGlobally/ParDo(ToSingletonIterables)+PAssert$0/GroupGlobally/WithKeys/AddKeys/Map+PAssert$0/GroupGlobally/GroupByKey/Reify+PAssert$0/GroupGlobally/GroupByKey/Write+Calculate
hashcode/View.AsIterable/ParDo(ToIsmRecordForGlobalWindow)
Oct 14, 2020 1:01:52 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T01:01:51.473Z: Executing operation Calculate
hashcode/View.AsIterable/CreateDataflowView
Oct 14, 2020 1:01:52 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T01:01:51.526Z: Finished operation Calculate
hashcode/View.AsIterable/CreateDataflowView
Oct 14, 2020 1:01:52 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T01:01:51.693Z: Executing operation Calculate
hashcode/CreateVoid/Read(CreateSource)+Calculate
hashcode/ProduceDefault+PAssert$0/GroupGlobally/Reify.Window/ParDo(Anonymous)+PAssert$0/GroupGlobally/ParDo(ToSingletonIterables)+PAssert$0/GroupGlobally/WithKeys/AddKeys/Map+PAssert$0/GroupGlobally/GroupByKey/Reify+PAssert$0/GroupGlobally/GroupByKey/Write+View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map+View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey+View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Partial+View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Reify+View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Write
Oct 14, 2020 1:01:54 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T01:01:52.550Z: Finished operation Calculate
hashcode/CreateVoid/Read(CreateSource)+Calculate
hashcode/ProduceDefault+PAssert$0/GroupGlobally/Reify.Window/ParDo(Anonymous)+PAssert$0/GroupGlobally/ParDo(ToSingletonIterables)+PAssert$0/GroupGlobally/WithKeys/AddKeys/Map+PAssert$0/GroupGlobally/GroupByKey/Reify+PAssert$0/GroupGlobally/GroupByKey/Write+View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map+View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey+View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Partial+View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Reify+View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Write
Oct 14, 2020 1:01:54 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T01:01:52.628Z: Executing operation
View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Close
Oct 14, 2020 1:01:54 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T01:01:52.663Z: Executing operation
PAssert$0/GroupGlobally/GroupByKey/Close
Oct 14, 2020 1:01:54 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T01:01:52.679Z: Finished operation
View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Close
Oct 14, 2020 1:01:54 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T01:01:52.728Z: Finished operation
PAssert$0/GroupGlobally/GroupByKey/Close
Oct 14, 2020 1:01:54 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T01:01:52.753Z: Executing operation
View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Read+View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues+View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Extract+View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map+View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)+View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
Oct 14, 2020 1:01:54 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T01:01:52.802Z: Executing operation
PAssert$0/GroupGlobally/GroupByKey/Read+PAssert$0/GroupGlobally/GroupByKey/GroupByWindow+PAssert$0/GroupGlobally/Values/Values/Map+PAssert$0/GroupGlobally/ParDo(Concat)+PAssert$0/GetPane/Map+PAssert$0/RunChecks+PAssert$0/VerifyAssertions/ParDo(DefaultConclude)
Oct 14, 2020 1:01:54 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T01:01:53.378Z: Finished operation
PAssert$0/GroupGlobally/GroupByKey/Read+PAssert$0/GroupGlobally/GroupByKey/GroupByWindow+PAssert$0/GroupGlobally/Values/Values/Map+PAssert$0/GroupGlobally/ParDo(Concat)+PAssert$0/GetPane/Map+PAssert$0/RunChecks+PAssert$0/VerifyAssertions/ParDo(DefaultConclude)
Oct 14, 2020 1:01:54 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T01:01:53.423Z: Finished operation
View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Read+View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues+View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Extract+View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map+View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)+View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
Oct 14, 2020 1:01:54 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T01:01:53.509Z: Executing operation
View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Close
Oct 14, 2020 1:01:54 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T01:01:53.565Z: Finished operation
View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Close
Oct 14, 2020 1:01:54 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T01:01:53.637Z: Executing operation
View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read+View.AsSingleton/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow)
Oct 14, 2020 1:01:56 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T01:01:55.574Z: Finished operation
View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read+View.AsSingleton/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow)
Oct 14, 2020 1:01:56 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T01:01:55.718Z: Executing operation
View.AsSingleton/Combine.GloballyAsSingletonView/CreateDataflowView
Oct 14, 2020 1:01:56 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T01:01:55.771Z: Finished operation
View.AsSingleton/Combine.GloballyAsSingletonView/CreateDataflowView
Oct 14, 2020 1:01:56 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T01:01:55.937Z: Executing operation Delete test files
Oct 14, 2020 1:02:00 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T01:01:59.007Z: Finished operation Delete test files
Oct 14, 2020 1:02:00 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T01:01:59.175Z: Cleaning up.
Oct 14, 2020 1:02:00 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-10-14T01:01:59.255Z: Stopping **** pool...
Oct 14, 2020 1:02:43 AM
org.apache.beam.runners.dataflow.DataflowPipelineJob lambda$waitUntilFinish$0
WARNING: Job is already running in Google Cloud Platform, Ctrl-C will not
cancel it.
To cancel the job in the cloud, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel
--region=us-central1 2020-10-13_17_57_53-16364477530988434267
org.apache.beam.sdk.io.text.TextIOIT > writeThenReadAll SKIPPED
> Task :sdks:java:io:file-based-io-tests:integrationTest FAILED
:sdks:java:io:file-based-io-tests:integrationTest (Thread[Execution **** for
':' Thread 2,5,main]) completed. Took 5 mins 4.063 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':sdks:java:io:file-based-io-tests:integrationTest'.
> Process 'Gradle Test Executor 6' finished with non-zero exit value 143
This problem might be caused by incorrect test process configuration.
Please refer to the test execution section in the User Manual at
https://docs.gradle.org/6.6.1/userguide/java_testing.html#sec:test_execution
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to
get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with
Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See
https://docs.gradle.org/6.6.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 5m 48s
96 actionable tasks: 59 executed, 37 from cache
Publishing build scan...
The message received from the daemon indicates that the daemon has disappeared.
Build request sent: Build{id=332bc08b-412f-436e-a7e4-a41976f66b90,
currentDir=<https://ci-beam.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/ws/src}>
Attempting to read last messages from the daemon log...
Daemon pid: 20274
log file: /home/jenkins/.gradle/daemon/6.6.1/daemon-20274.out.log
----- Last 20 lines from daemon log file - daemon-20274.out.log -----
* What went wrong:
Execution failed for task ':sdks:java:io:file-based-io-tests:integrationTest'.
> Process 'Gradle Test Executor 6' finished with non-zero exit value 143
This problem might be caused by incorrect test process configuration.
Please refer to the test execution section in the User Manual at
https://docs.gradle.org/6.6.1/userguide/java_testing.html#sec:test_execution
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to
get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with
Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See
https://docs.gradle.org/6.6.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 5m 48s
96 actionable tasks: 59 executed, 37 from cache
Publishing build scan...
Daemon vm is shutting down... The daemon has exited normally or was terminated
in response to a user interrupt.
----- End of the daemon log -----
FAILURE: Build failed with an exception.
* What went wrong:
Gradle build daemon disappeared unexpectedly (it may have been killed or may
have crashed)
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to
get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
