See <https://ci-beam.apache.org/job/beam_PostRelease_NightlySnapshot/1364/display/redirect?page=changes>
Changes: [shoyer] Use __getstate__ & __setstate__ for deterministic coding [shoyer] Re-raise exceptions during deterministic encoding [shoyer] Add test for non-deterministic inputs [aromanenko.dev] [BEAM-12271] Make max width for code snippets adaptive with text blocks [Kenneth Knowles] [BEAM-12273] Support non-multimap materialization in Twister2 runner [Ismaël Mejía] [BEAM-12280] Upgrade Flink runner to Flink version 1.12.3 [noreply] [BEAM-11759] Create Beam glossary (#14717) ------------------------------------------ [...truncated 3.69 MB...] > gcloud dataflow jobs --project=apache-beam-testing cancel > --region=us-central1 2021-05-05_04_24_26-5692170606189967423 May 05, 2021 11:24:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-05-05T11:24:30.269Z: Autoscaling is enabled for job 2021-05-05_04_24_26-5692170606189967423. The number of workers will be between 1 and 1000. May 05, 2021 11:24:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-05-05T11:24:30.449Z: Autoscaling was automatically enabled for job 2021-05-05_04_24_26-5692170606189967423. May 05, 2021 11:24:40 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-05-05T11:24:39.622Z: Worker configuration: n1-standard-1 in us-central1-c. May 05, 2021 11:24:43 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-05-05T11:24:40.660Z: Expanding CoGroupByKey operations into optimizable parts. May 05, 2021 11:24:43 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-05-05T11:24:40.822Z: Expanding GroupByKey operations into optimizable parts. May 05, 2021 11:24:43 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-05-05T11:24:40.855Z: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns May 05, 2021 11:24:43 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-05-05T11:24:41.116Z: Fusing adjacent ParDo, Read, Write, and Flatten operations May 05, 2021 11:24:43 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-05-05T11:24:41.144Z: Fusing consumer WriteUserScoreSums/TextIO.Write/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Reify.ReifyView/ParDo(Anonymous) into WriteUserScoreSums/TextIO.Write/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Create.Values/Read(CreateSource) May 05, 2021 11:24:43 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-05-05T11:24:41.177Z: Fusing consumer WriteUserScoreSums/TextIO.Write/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Values/Values/Map into WriteUserScoreSums/TextIO.Write/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Reify.ReifyView/ParDo(Anonymous) May 05, 2021 11:24:43 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-05-05T11:24:41.204Z: Fusing consumer WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Finalize into WriteUserScoreSums/TextIO.Write/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Values/Values/Map May 05, 2021 11:24:43 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-05-05T11:24:41.241Z: Fusing consumer WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair with random key into WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Finalize May 05, 2021 11:24:43 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-05-05T11:24:41.275Z: Fusing consumer WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign into WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair with random key May 05, 2021 11:24:43 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-05-05T11:24:41.301Z: Fusing consumer WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify into WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign May 05, 2021 11:24:43 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-05-05T11:24:41.339Z: Fusing consumer WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write into WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify May 05, 2021 11:24:43 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-05-05T11:24:41.375Z: Fusing consumer WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow into WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read May 05, 2021 11:24:43 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-05-05T11:24:41.398Z: Fusing consumer WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable into WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow May 05, 2021 11:24:43 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-05-05T11:24:41.433Z: Fusing consumer WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Values/Values/Map into WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable May 05, 2021 11:24:43 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-05-05T11:24:41.461Z: Unzipping flatten s12 for input s11.org.apache.beam.sdk.values.PCollection.<init>:402#5f2ef1f005ae0b4 May 05, 2021 11:24:43 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-05-05T11:24:41.500Z: Fusing unzipped copy of WriteUserScoreSums/TextIO.Write/WriteFiles/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow), through flatten WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/Flatten.PCollections, into producer WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/DropShardNum May 05, 2021 11:24:43 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-05-05T11:24:41.530Z: Fusing consumer WriteUserScoreSums/TextIO.Write/WriteFiles/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow) into WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles May 05, 2021 11:24:43 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-05-05T11:24:41.566Z: Fusing consumer ParseGameEvent into TextIO.Read/Read May 05, 2021 11:24:43 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-05-05T11:24:41.604Z: Fusing consumer ExtractUserScore/MapElements/Map into ParseGameEvent May 05, 2021 11:24:43 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-05-05T11:24:41.643Z: Fusing consumer ExtractUserScore/Combine.perKey(SumInteger)/GroupByKey+ExtractUserScore/Combine.perKey(SumInteger)/Combine.GroupedValues/Partial into ExtractUserScore/MapElements/Map May 05, 2021 11:24:43 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-05-05T11:24:41.678Z: Fusing consumer ExtractUserScore/Combine.perKey(SumInteger)/GroupByKey/Reify into ExtractUserScore/Combine.perKey(SumInteger)/GroupByKey+ExtractUserScore/Combine.perKey(SumInteger)/Combine.GroupedValues/Partial May 05, 2021 11:24:43 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-05-05T11:24:41.716Z: Fusing consumer ExtractUserScore/Combine.perKey(SumInteger)/GroupByKey/Write into ExtractUserScore/Combine.perKey(SumInteger)/GroupByKey/Reify May 05, 2021 11:24:43 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-05-05T11:24:41.749Z: Fusing consumer ExtractUserScore/Combine.perKey(SumInteger)/Combine.GroupedValues into ExtractUserScore/Combine.perKey(SumInteger)/GroupByKey/Read May 05, 2021 11:24:43 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-05-05T11:24:41.784Z: Fusing consumer ExtractUserScore/Combine.perKey(SumInteger)/Combine.GroupedValues/Extract into ExtractUserScore/Combine.perKey(SumInteger)/Combine.GroupedValues May 05, 2021 11:24:43 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-05-05T11:24:41.821Z: Fusing consumer WriteUserScoreSums/ConvertToRow into ExtractUserScore/Combine.perKey(SumInteger)/Combine.GroupedValues/Extract May 05, 2021 11:24:43 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-05-05T11:24:41.850Z: Fusing consumer WriteUserScoreSums/TextIO.Write/WriteFiles/RewindowIntoGlobal/Window.Assign into WriteUserScoreSums/ConvertToRow May 05, 2021 11:24:43 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-05-05T11:24:41.893Z: Fusing consumer WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles into WriteUserScoreSums/TextIO.Write/WriteFiles/RewindowIntoGlobal/Window.Assign May 05, 2021 11:24:43 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-05-05T11:24:41.948Z: Fusing consumer WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Reify into WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles May 05, 2021 11:24:43 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-05-05T11:24:41.984Z: Fusing consumer WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Write into WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Reify May 05, 2021 11:24:43 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-05-05T11:24:42.011Z: Fusing consumer WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/GroupByWindow into WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Read May 05, 2021 11:24:43 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-05-05T11:24:42.045Z: Fusing consumer WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnwritten into WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/GroupByWindow May 05, 2021 11:24:43 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-05-05T11:24:42.081Z: Fusing consumer WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/DropShardNum into WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnwritten May 05, 2021 11:24:43 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-05-05T11:24:42.492Z: Executing operation ExtractUserScore/Combine.perKey(SumInteger)/GroupByKey/Create May 05, 2021 11:24:43 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-05-05T11:24:42.590Z: Starting 1 workers in us-central1-c... May 05, 2021 11:24:43 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-05-05T11:24:42.967Z: Finished operation ExtractUserScore/Combine.perKey(SumInteger)/GroupByKey/Create May 05, 2021 11:24:43 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-05-05T11:24:43.182Z: Executing operation TextIO.Read/Read+ParseGameEvent+ExtractUserScore/MapElements/Map+ExtractUserScore/Combine.perKey(SumInteger)/GroupByKey+ExtractUserScore/Combine.perKey(SumInteger)/Combine.GroupedValues/Partial+ExtractUserScore/Combine.perKey(SumInteger)/GroupByKey/Reify+ExtractUserScore/Combine.perKey(SumInteger)/GroupByKey/Write May 05, 2021 11:25:13 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-05-05T11:25:12.698Z: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete May 05, 2021 11:25:14 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-05-05T11:25:13.869Z: Autoscaling: Raised the number of workers to 1 based on the rate of progress in the currently running stage(s). May 05, 2021 11:25:50 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-05-05T11:25:49.987Z: Workers have started successfully. May 05, 2021 11:25:50 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-05-05T11:25:50.028Z: Workers have started successfully. May 05, 2021 11:26:11 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-05-05T11:26:09.078Z: Finished operation TextIO.Read/Read+ParseGameEvent+ExtractUserScore/MapElements/Map+ExtractUserScore/Combine.perKey(SumInteger)/GroupByKey+ExtractUserScore/Combine.perKey(SumInteger)/Combine.GroupedValues/Partial+ExtractUserScore/Combine.perKey(SumInteger)/GroupByKey/Reify+ExtractUserScore/Combine.perKey(SumInteger)/GroupByKey/Write May 05, 2021 11:26:11 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-05-05T11:26:09.159Z: Executing operation ExtractUserScore/Combine.perKey(SumInteger)/GroupByKey/Close May 05, 2021 11:26:11 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-05-05T11:26:09.244Z: Finished operation ExtractUserScore/Combine.perKey(SumInteger)/GroupByKey/Close May 05, 2021 11:26:11 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-05-05T11:26:09.309Z: Executing operation WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Create May 05, 2021 11:26:11 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-05-05T11:26:09.542Z: Finished operation WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Create May 05, 2021 11:26:11 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-05-05T11:26:09.788Z: Executing operation ExtractUserScore/Combine.perKey(SumInteger)/GroupByKey/Read+ExtractUserScore/Combine.perKey(SumInteger)/Combine.GroupedValues+ExtractUserScore/Combine.perKey(SumInteger)/Combine.GroupedValues/Extract+WriteUserScoreSums/ConvertToRow+WriteUserScoreSums/TextIO.Write/WriteFiles/RewindowIntoGlobal/Window.Assign+WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles+WriteUserScoreSums/TextIO.Write/WriteFiles/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow)+WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Reify+WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Write May 05, 2021 11:26:19 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-05-05T11:26:19.302Z: Finished operation ExtractUserScore/Combine.perKey(SumInteger)/GroupByKey/Read+ExtractUserScore/Combine.perKey(SumInteger)/Combine.GroupedValues+ExtractUserScore/Combine.perKey(SumInteger)/Combine.GroupedValues/Extract+WriteUserScoreSums/ConvertToRow+WriteUserScoreSums/TextIO.Write/WriteFiles/RewindowIntoGlobal/Window.Assign+WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles+WriteUserScoreSums/TextIO.Write/WriteFiles/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow)+WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Reify+WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Write May 05, 2021 11:26:21 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-05-05T11:26:19.421Z: Executing operation WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Close May 05, 2021 11:26:21 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-05-05T11:26:19.484Z: Finished operation WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Close May 05, 2021 11:26:21 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-05-05T11:26:19.566Z: Executing operation WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Read+WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/GroupByWindow+WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnwritten+WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/DropShardNum+WriteUserScoreSums/TextIO.Write/WriteFiles/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow) May 05, 2021 11:26:22 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-05-05T11:26:22.177Z: Finished operation WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Read+WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/GroupByWindow+WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnwritten+WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/DropShardNum+WriteUserScoreSums/TextIO.Write/WriteFiles/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow) May 05, 2021 11:26:22 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-05-05T11:26:22.366Z: Executing operation WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/Flatten.PCollections/Unzipped-1 May 05, 2021 11:26:22 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-05-05T11:26:22.455Z: Finished operation WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/Flatten.PCollections/Unzipped-1 May 05, 2021 11:26:22 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-05-05T11:26:22.625Z: Executing operation WriteUserScoreSums/TextIO.Write/WriteFiles/GatherTempFileResults/View.AsList/CreateDataflowView May 05, 2021 11:26:22 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-05-05T11:26:22.692Z: Finished operation WriteUserScoreSums/TextIO.Write/WriteFiles/GatherTempFileResults/View.AsList/CreateDataflowView May 05, 2021 11:26:25 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-05-05T11:26:22.873Z: Executing operation WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Create May 05, 2021 11:26:25 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-05-05T11:26:23.163Z: Finished operation WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Create May 05, 2021 11:26:25 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-05-05T11:26:23.331Z: Executing operation WriteUserScoreSums/TextIO.Write/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Create.Values/Read(CreateSource)+WriteUserScoreSums/TextIO.Write/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Reify.ReifyView/ParDo(Anonymous)+WriteUserScoreSums/TextIO.Write/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Values/Values/Map+WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Finalize+WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair with random key+WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign+WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify+WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write May 05, 2021 11:26:28 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process SEVERE: 2021-05-05T11:26:26.883Z: java.lang.IllegalAccessError: tried to access class org.apache.beam.sdk.values.PCollectionViews$SingletonViewFn2 from class org.apache.beam.runners.dataflow.worker.IsmSideInputReader at org.apache.beam.runners.dataflow.worker.IsmSideInputReader.<clinit>(IsmSideInputReader.java:112) at org.apache.beam.runners.dataflow.worker.BatchModeExecutionContext.lambda$getSideInputReader$0(BatchModeExecutionContext.java:290) at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Suppliers$NonSerializableMemoizingSupplier.get(Suppliers.java:167) at org.apache.beam.runners.dataflow.worker.LazilyInitializedSideInputReader.get(LazilyInitializedSideInputReader.java:50) at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.sideInput(SimpleDoFnRunner.java:267) at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.access$1200(SimpleDoFnRunner.java:84) at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.sideInput(SimpleDoFnRunner.java:395) at org.apache.beam.sdk.transforms.Reify$ReifyView$1.process(Reify.java:60) May 05, 2021 11:26:34 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process SEVERE: 2021-05-05T11:26:33.414Z: java.lang.NoClassDefFoundError: Could not initialize class org.apache.beam.runners.dataflow.worker.IsmSideInputReader at org.apache.beam.runners.dataflow.worker.BatchModeExecutionContext.lambda$getSideInputReader$0(BatchModeExecutionContext.java:290) at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Suppliers$NonSerializableMemoizingSupplier.get(Suppliers.java:167) at org.apache.beam.runners.dataflow.worker.LazilyInitializedSideInputReader.get(LazilyInitializedSideInputReader.java:50) at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.sideInput(SimpleDoFnRunner.java:267) at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.access$1200(SimpleDoFnRunner.java:84) at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.sideInput(SimpleDoFnRunner.java:395) at org.apache.beam.sdk.transforms.Reify$ReifyView$1.process(Reify.java:60) May 05, 2021 11:26:45 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process SEVERE: 2021-05-05T11:26:43.809Z: java.lang.NoClassDefFoundError: Could not initialize class org.apache.beam.runners.dataflow.worker.IsmSideInputReader at org.apache.beam.runners.dataflow.worker.BatchModeExecutionContext.lambda$getSideInputReader$0(BatchModeExecutionContext.java:290) at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Suppliers$NonSerializableMemoizingSupplier.get(Suppliers.java:167) at org.apache.beam.runners.dataflow.worker.LazilyInitializedSideInputReader.get(LazilyInitializedSideInputReader.java:50) at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.sideInput(SimpleDoFnRunner.java:267) at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.access$1200(SimpleDoFnRunner.java:84) at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.sideInput(SimpleDoFnRunner.java:395) at org.apache.beam.sdk.transforms.Reify$ReifyView$1.process(Reify.java:60) May 05, 2021 11:26:56 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process SEVERE: 2021-05-05T11:26:54.550Z: java.lang.NoClassDefFoundError: Could not initialize class org.apache.beam.runners.dataflow.worker.IsmSideInputReader at org.apache.beam.runners.dataflow.worker.BatchModeExecutionContext.lambda$getSideInputReader$0(BatchModeExecutionContext.java:290) at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Suppliers$NonSerializableMemoizingSupplier.get(Suppliers.java:167) at org.apache.beam.runners.dataflow.worker.LazilyInitializedSideInputReader.get(LazilyInitializedSideInputReader.java:50) at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.sideInput(SimpleDoFnRunner.java:267) at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.access$1200(SimpleDoFnRunner.java:84) at org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.sideInput(SimpleDoFnRunner.java:395) at org.apache.beam.sdk.transforms.Reify$ReifyView$1.process(Reify.java:60) May 05, 2021 11:26:56 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-05-05T11:26:54.626Z: Finished operation WriteUserScoreSums/TextIO.Write/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Create.Values/Read(CreateSource)+WriteUserScoreSums/TextIO.Write/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Reify.ReifyView/ParDo(Anonymous)+WriteUserScoreSums/TextIO.Write/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Values/Values/Map+WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Finalize+WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair with random key+WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign+WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify+WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write May 05, 2021 11:26:56 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process SEVERE: 2021-05-05T11:26:54.776Z: Workflow failed. Causes: S11:WriteUserScoreSums/TextIO.Write/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Create.Values/Read(CreateSource)+WriteUserScoreSums/TextIO.Write/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Reify.ReifyView/ParDo(Anonymous)+WriteUserScoreSums/TextIO.Write/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Values/Values/Map+WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Finalize+WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair with random key+WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign+WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify+WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write failed., The job failed because a work item has failed 4 times. Look in previous log entries for the cause of each one of the 4 failures. For more information, see https://cloud.google.com/dataflow/docs/guides/common-errors. The work item was attempted on these workers: userscore-jenkins-0505112-05050424-otyf-harness-gqnr Root cause: Work item failed., userscore-jenkins-0505112-05050424-otyf-harness-gqnr Root cause: Work item failed., userscore-jenkins-0505112-05050424-otyf-harness-gqnr Root cause: Work item failed., userscore-jenkins-0505112-05050424-otyf-harness-gqnr Root cause: Work item failed. May 05, 2021 11:26:56 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-05-05T11:26:54.902Z: Cleaning up. May 05, 2021 11:26:56 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-05-05T11:26:55.099Z: Stopping worker pool... May 05, 2021 11:27:40 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-05-05T11:27:40.296Z: Autoscaling: Resized worker pool from 1 to 0. May 05, 2021 11:27:40 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-05-05T11:27:40.337Z: Worker pool stopped. May 05, 2021 11:27:46 AM org.apache.beam.runners.dataflow.DataflowPipelineJob logTerminalState INFO: Job 2021-05-05_04_24_26-5692170606189967423 failed with status FAILED. gsutil cat gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/java-userscore-result-dataflow-runner.txt* | grep user19_BananaWallaby CommandException: No URLs matched: gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/java-userscore-result-dataflow-runner.txt* CommandException: No URLs matched: gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/java-userscore-result-dataflow-runner.txt* [ERROR] Failed command > Task :runners:google-cloud-dataflow-java:runMobileGamingJavaDataflow FAILED FAILURE: Build completed with 4 failures. 1: Task failed with an exception. ----------- * What went wrong: Execution failed for task ':runners:direct-java:runMobileGamingJavaDirect'. > Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with > non-zero exit value 1 * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. ============================================================================== 2: Task failed with an exception. ----------- * What went wrong: Execution failed for task ':runners:google-cloud-dataflow-java:runQuickstartJavaDataflow'. > Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with > non-zero exit value 1 * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. ============================================================================== 3: Task failed with an exception. ----------- * What went wrong: Execution failed for task ':runners:google-cloud-dataflow-java:runMobileGamingJavaDataflowBom'. > Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with > non-zero exit value 1 * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. ============================================================================== 4: Task failed with an exception. ----------- * What went wrong: Execution failed for task ':runners:google-cloud-dataflow-java:runMobileGamingJavaDataflow'. > Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with > non-zero exit value 1 * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. ============================================================================== * Get more help at https://help.gradle.org Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/6.8.3/userguide/command_line_interface.html#sec:command_line_warnings BUILD FAILED in 27m 27s 9 actionable tasks: 8 executed, 1 from cache Publishing build scan... https://gradle.com/s/e6ew4z5vtq4w2 Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
