See <https://ci-beam.apache.org/job/beam_PostRelease_NightlySnapshot/1576/display/redirect>
Changes: ------------------------------------------ [...truncated 2.92 MB...] INFO: Uploading 232 files from PipelineOptions.filesToStage to staging location to prepare for execution. Dec 05, 2021 12:42:31 PM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements INFO: Staging files complete: 232 files cached, 0 files newly uploaded in 0 seconds Dec 05, 2021 12:42:31 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Staging portable pipeline proto to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/ Dec 05, 2021 12:42:31 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <274045 bytes, hash e5e2c1a3a347824f2d7fc55024a9685cb848cae4f3327f64a20d1335a310cbc1> to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/pipeline-5eLBo6NHgk8tf8VQJKloXLhIyuTzMn9kog0TNaMQy8E.pb Dec 05, 2021 12:42:32 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding PubsubIO.Read/PubsubUnboundedSource as step s1 Dec 05, 2021 12:42:32 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding PubsubIO.Read/MapElements/Map as step s2 Dec 05, 2021 12:42:32 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding ParseGameEvent as step s3 Dec 05, 2021 12:42:32 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding CalculateTeamScores/LeaderboardTeamFixedWindows/Window.Assign as step s4 Dec 05, 2021 12:42:32 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding CalculateTeamScores/ExtractTeamScore/MapElements/Map as step s5 Dec 05, 2021 12:42:32 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding CalculateTeamScores/ExtractTeamScore/Combine.perKey(SumInteger)/GroupByKey as step s6 Dec 05, 2021 12:42:32 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding CalculateTeamScores/ExtractTeamScore/Combine.perKey(SumInteger)/Combine.GroupedValues as step s7 Dec 05, 2021 12:42:32 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteTeamScoreSums/ConvertToRow as step s8 Dec 05, 2021 12:42:32 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteTeamScoreSums/BigQueryIO.Write/PrepareWrite/ParDo(Anonymous) as step s9 Dec 05, 2021 12:42:32 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/CreateTables/ParDo(CreateTables) as step s10 Dec 05, 2021 12:42:32 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/ShardTableWrites as step s11 Dec 05, 2021 12:42:32 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/TagWithUniqueIds as step s12 Dec 05, 2021 12:42:32 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/Window.Into()/Window.Assign as step s13 Dec 05, 2021 12:42:32 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey as step s14 Dec 05, 2021 12:42:32 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/ExpandIterable as step s15 Dec 05, 2021 12:42:32 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/GlobalWindow/Window.Assign as step s16 Dec 05, 2021 12:42:32 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/StripShardId/Map as step s17 Dec 05, 2021 12:42:32 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/StreamingWrite/BatchedStreamingWrite.ViaBundleFinalization/ParMultiDo(BatchAndInsertElements) as step s18 Dec 05, 2021 12:42:32 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding CalculateUserScores/LeaderboardUserGlobalWindow/Window.Assign as step s19 Dec 05, 2021 12:42:32 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding CalculateUserScores/ExtractUserScore/MapElements/Map as step s20 Dec 05, 2021 12:42:32 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding CalculateUserScores/ExtractUserScore/Combine.perKey(SumInteger)/GroupByKey as step s21 Dec 05, 2021 12:42:32 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding CalculateUserScores/ExtractUserScore/Combine.perKey(SumInteger)/Combine.GroupedValues as step s22 Dec 05, 2021 12:42:32 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteUserScoreSums/ConvertToRow as step s23 Dec 05, 2021 12:42:32 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteUserScoreSums/BigQueryIO.Write/PrepareWrite/ParDo(Anonymous) as step s24 Dec 05, 2021 12:42:32 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/CreateTables/ParDo(CreateTables) as step s25 Dec 05, 2021 12:42:32 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/ShardTableWrites as step s26 Dec 05, 2021 12:42:33 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/TagWithUniqueIds as step s27 Dec 05, 2021 12:42:33 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/Window.Into()/Window.Assign as step s28 Dec 05, 2021 12:42:33 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey as step s29 Dec 05, 2021 12:42:33 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/ExpandIterable as step s30 Dec 05, 2021 12:42:33 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/GlobalWindow/Window.Assign as step s31 Dec 05, 2021 12:42:33 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/StripShardId/Map as step s32 Dec 05, 2021 12:42:33 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/StreamingWrite/BatchedStreamingWrite.ViaBundleFinalization/ParMultiDo(BatchAndInsertElements) as step s33 Dec 05, 2021 12:42:33 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Dataflow SDK version: 2.36.0-SNAPSHOT Dec 05, 2021 12:42:34 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2021-12-05_04_42_33-13296014152000631895?project=apache-beam-testing Dec 05, 2021 12:42:34 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Submitted job: 2021-12-05_04_42_33-13296014152000631895 Dec 05, 2021 12:42:34 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: To cancel the job using the 'gcloud' tool, run: > gcloud dataflow jobs --project=apache-beam-testing cancel > --region=us-central1 2021-12-05_04_42_33-13296014152000631895 Dec 05, 2021 12:42:38 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process WARNING: 2021-12-05T12:42:36.807Z: Autoscaling is enabled for Dataflow Streaming Engine. Workers will scale between 1 and 100 unless maxNumWorkers is specified. Dec 05, 2021 12:42:38 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-12-05T12:42:36.892Z: Autoscaling is enabled for job 2021-12-05_04_42_33-13296014152000631895. The number of workers will be between 1 and 100. Dec 05, 2021 12:42:38 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-12-05T12:42:36.923Z: Autoscaling was automatically enabled for job 2021-12-05_04_42_33-13296014152000631895. Dec 05, 2021 12:42:45 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-12-05T12:42:42.486Z: Worker configuration: e2-standard-2 in us-central1-b. Dec 05, 2021 12:42:45 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-12-05T12:42:44.118Z: Expanding CoGroupByKey operations into optimizable parts. Dec 05, 2021 12:42:45 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-12-05T12:42:44.284Z: Expanding SplittableProcessKeyed operations into optimizable parts. Dec 05, 2021 12:42:45 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-12-05T12:42:44.312Z: Expanding GroupByKey operations into streaming Read/Write steps Dec 05, 2021 12:42:45 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-12-05T12:42:44.476Z: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns Dec 05, 2021 12:42:45 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-12-05T12:42:44.669Z: Fusing adjacent ParDo, Read, Write, and Flatten operations Dec 05, 2021 12:42:45 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-12-05T12:42:44.698Z: Fusing consumer CalculateUserScores/LeaderboardUserGlobalWindow/Window.Assign into ParseGameEvent Dec 05, 2021 12:42:45 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-12-05T12:42:44.734Z: Fusing consumer CalculateTeamScores/LeaderboardTeamFixedWindows/Window.Assign into ParseGameEvent Dec 05, 2021 12:42:45 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-12-05T12:42:44.766Z: Fusing consumer PubsubIO.Read/MapElements/Map into PubsubIO.Read/PubsubUnboundedSource Dec 05, 2021 12:42:45 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-12-05T12:42:44.791Z: Fusing consumer ParseGameEvent into PubsubIO.Read/MapElements/Map Dec 05, 2021 12:42:45 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-12-05T12:42:44.825Z: Fusing consumer CalculateTeamScores/ExtractTeamScore/MapElements/Map into CalculateTeamScores/LeaderboardTeamFixedWindows/Window.Assign Dec 05, 2021 12:42:45 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-12-05T12:42:44.850Z: Fusing consumer CalculateTeamScores/ExtractTeamScore/Combine.perKey(SumInteger)/GroupByKey/WriteStream into CalculateTeamScores/ExtractTeamScore/MapElements/Map Dec 05, 2021 12:42:45 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-12-05T12:42:44.879Z: Fusing consumer CalculateTeamScores/ExtractTeamScore/Combine.perKey(SumInteger)/Combine.GroupedValues into CalculateTeamScores/ExtractTeamScore/Combine.perKey(SumInteger)/GroupByKey/ReadStream Dec 05, 2021 12:42:45 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-12-05T12:42:44.912Z: Fusing consumer WriteTeamScoreSums/ConvertToRow into CalculateTeamScores/ExtractTeamScore/Combine.perKey(SumInteger)/Combine.GroupedValues Dec 05, 2021 12:42:45 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-12-05T12:42:44.935Z: Fusing consumer WriteTeamScoreSums/BigQueryIO.Write/PrepareWrite/ParDo(Anonymous) into WriteTeamScoreSums/ConvertToRow Dec 05, 2021 12:42:45 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-12-05T12:42:44.976Z: Fusing consumer WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/CreateTables/ParDo(CreateTables) into WriteTeamScoreSums/BigQueryIO.Write/PrepareWrite/ParDo(Anonymous) Dec 05, 2021 12:42:45 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-12-05T12:42:45.011Z: Fusing consumer WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/ShardTableWrites into WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/CreateTables/ParDo(CreateTables) Dec 05, 2021 12:42:45 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-12-05T12:42:45.031Z: Fusing consumer WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/TagWithUniqueIds into WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/ShardTableWrites Dec 05, 2021 12:42:45 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-12-05T12:42:45.061Z: Fusing consumer WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/Window.Into()/Window.Assign into WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/TagWithUniqueIds Dec 05, 2021 12:42:45 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-12-05T12:42:45.093Z: Fusing consumer WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey/WriteStream into WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/Window.Into()/Window.Assign Dec 05, 2021 12:42:45 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-12-05T12:42:45.126Z: Fusing consumer WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey/MergeBuckets into WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey/ReadStream Dec 05, 2021 12:42:45 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-12-05T12:42:45.158Z: Fusing consumer WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/ExpandIterable into WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey/MergeBuckets Dec 05, 2021 12:42:45 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-12-05T12:42:45.192Z: Fusing consumer WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/GlobalWindow/Window.Assign into WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/ExpandIterable Dec 05, 2021 12:42:45 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-12-05T12:42:45.225Z: Fusing consumer WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/StripShardId/Map into WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/GlobalWindow/Window.Assign Dec 05, 2021 12:42:46 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-12-05T12:42:45.261Z: Fusing consumer WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/StreamingWrite/BatchedStreamingWrite.ViaBundleFinalization/ParMultiDo(BatchAndInsertElements) into WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/StripShardId/Map Dec 05, 2021 12:42:46 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-12-05T12:42:45.296Z: Fusing consumer CalculateUserScores/ExtractUserScore/MapElements/Map into CalculateUserScores/LeaderboardUserGlobalWindow/Window.Assign Dec 05, 2021 12:42:46 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-12-05T12:42:45.319Z: Fusing consumer CalculateUserScores/ExtractUserScore/Combine.perKey(SumInteger)/GroupByKey/WriteStream into CalculateUserScores/ExtractUserScore/MapElements/Map Dec 05, 2021 12:42:46 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-12-05T12:42:45.350Z: Fusing consumer CalculateUserScores/ExtractUserScore/Combine.perKey(SumInteger)/Combine.GroupedValues into CalculateUserScores/ExtractUserScore/Combine.perKey(SumInteger)/GroupByKey/ReadStream Dec 05, 2021 12:42:46 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-12-05T12:42:45.385Z: Fusing consumer WriteUserScoreSums/ConvertToRow into CalculateUserScores/ExtractUserScore/Combine.perKey(SumInteger)/Combine.GroupedValues Dec 05, 2021 12:42:46 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-12-05T12:42:45.429Z: Fusing consumer WriteUserScoreSums/BigQueryIO.Write/PrepareWrite/ParDo(Anonymous) into WriteUserScoreSums/ConvertToRow Dec 05, 2021 12:42:46 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-12-05T12:42:45.461Z: Fusing consumer WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/CreateTables/ParDo(CreateTables) into WriteUserScoreSums/BigQueryIO.Write/PrepareWrite/ParDo(Anonymous) Dec 05, 2021 12:42:46 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-12-05T12:42:45.514Z: Fusing consumer WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/ShardTableWrites into WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/CreateTables/ParDo(CreateTables) Dec 05, 2021 12:42:46 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-12-05T12:42:45.569Z: Fusing consumer WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/TagWithUniqueIds into WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/ShardTableWrites Dec 05, 2021 12:42:46 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-12-05T12:42:45.641Z: Fusing consumer WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/Window.Into()/Window.Assign into WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/TagWithUniqueIds Dec 05, 2021 12:42:46 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-12-05T12:42:45.685Z: Fusing consumer WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey/WriteStream into WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/Window.Into()/Window.Assign Dec 05, 2021 12:42:46 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-12-05T12:42:45.723Z: Fusing consumer WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey/MergeBuckets into WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey/ReadStream Dec 05, 2021 12:42:46 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-12-05T12:42:45.757Z: Fusing consumer WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/ExpandIterable into WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey/MergeBuckets Dec 05, 2021 12:42:46 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-12-05T12:42:45.792Z: Fusing consumer WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/GlobalWindow/Window.Assign into WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/ExpandIterable Dec 05, 2021 12:42:46 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-12-05T12:42:45.825Z: Fusing consumer WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/StripShardId/Map into WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/GlobalWindow/Window.Assign Dec 05, 2021 12:42:46 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-12-05T12:42:45.847Z: Fusing consumer WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/StreamingWrite/BatchedStreamingWrite.ViaBundleFinalization/ParMultiDo(BatchAndInsertElements) into WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/StripShardId/Map Dec 05, 2021 12:42:46 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-12-05T12:42:45.883Z: The pubsub read for: projects/apache-beam-testing/subscriptions/java_mobile_gaming_topic.subscription-12875879596016191134 is configured to compute input data watermarks based on custom timestamp attribute timestamp_ms. Cloud Dataflow has created an additional tracking subscription to do this, which will be cleaned up automatically. For details, see: https://cloud.google.com/dataflow/model/pubsub-io#timestamps-ids Dec 05, 2021 12:42:46 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-12-05T12:42:46.103Z: Starting 1 workers in us-central1-b... Dec 05, 2021 12:42:50 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-12-05T12:42:49.895Z: Pub/Sub resources set up for topic 'projects/apache-beam-testing/topics/java_mobile_gaming_topic'. Dec 05, 2021 12:43:01 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-12-05T12:43:00.746Z: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete bq query SELECT table_id FROM beam_postrelease_mobile_gaming.__TABLES_SUMMARY__ Waiting on bqjob_r3b6c30ce43cab354_0000017d8a9ed94d_1 ... (0s) Current status: RUNNING Waiting on bqjob_r3b6c30ce43cab354_0000017d8a9ed94d_1 ... (0s) Current status: DONE +-----------------------------------+ | table_id | +-----------------------------------+ | hourly_team_score_python_dataflow | | hourly_team_score_python_direct | | leaderboard_DataflowRunner_team | | leaderboard_DataflowRunner_user | | leaderboard_DirectRunner_team | +-----------------------------------+ bq query --batch "SELECT user FROM [apache-beam-testing:beam_postrelease_mobile_gaming.leaderboard_DataflowRunner_user] LIMIT 10" Waiting on bqjob_rdf4ed6089ceefab_0000017d8a9ee4e1_1 ... (0s) Current status: RUNNING Waiting on bqjob_rdf4ed6089ceefab_0000017d8a9ee4e1_1 ... (0s) Current status: DONE +---------------------------+ | user | +---------------------------+ | user8_AndroidGreenEmu | | user10_AquaKangaroo | | user12_AppleGreenCockatoo | | user6_BananaEmu | | user0_FuchsiaCassowary | | user12_AntiqueBrassBilby | | user9_AliceBlueKangaroo | | user17_BarnRedPossum | | user8_BeigeBilby | | user14_AliceBluePossum | +---------------------------+ Verified AliceBlue RUNNING_JOB=`gcloud dataflow jobs list | grep leaderboard-validation-1638708139995-422 | grep Running | cut -d' ' -f1` if [ ! -z "${RUNNING_JOB}" ] then gcloud dataflow jobs cancel ${RUNNING_JOB} else echo "Job 'leaderboard-validation-1638708139995-422' is not running." fi Cancelled job [2021-12-05_04_42_33-13296014152000631895] ************************************** * SUCCESS: LeaderBoard successfully run on DataflowRunner. with Streaming Engine ************************************** [SUCCESS] FAILURE: Build failed with an exception. * What went wrong: Execution failed for task ':runners:direct-java:runMobileGamingJavaDirect'. > Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with > non-zero exit value 1 * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. * Get more help at https://help.gradle.org Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/6.9.1/userguide/command_line_interface.html#sec:command_line_warnings BUILD FAILED in 1h 20m 16s 9 actionable tasks: 8 executed, 1 from cache Publishing build scan... https://gradle.com/s/6jwa2ywwluf4k Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
