See <https://ci-beam.apache.org/job/beam_PostRelease_NightlySnapshot/1428/display/redirect?page=changes>
Changes: [noreply] Fix broken package substitution in starter archetype [je.ik] [BEAM-12597] Add AppendingTransformer for reference.conf in shade [mrudary] Generalize S3FileSystem to support multiple URI schemes. [heejong] [BEAM-12604] Do not expose Zookeeper client port in Kafka k8s config [noreply] [BEAM-11434]Make SpannerAccessor public (#13641) [noreply] Update Beam Go row coder to more completely handle schema interfaces [noreply] [BEAM-12556] Enable Go Build Tests in Samza Runner (#15167) [noreply] [BEAM-8376] Google Cloud Firestore Connector - Add Firestore v1 Read ------------------------------------------ [...truncated 2.44 MB...] Jul 14, 2021 12:03:51 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteTeamScoreSums/BigQueryIO.Write/PrepareWrite/ParDo(Anonymous) as step s9 Jul 14, 2021 12:03:51 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/CreateTables/ParDo(CreateTables) as step s10 Jul 14, 2021 12:03:51 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/ShardTableWrites as step s11 Jul 14, 2021 12:03:51 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/TagWithUniqueIds as step s12 Jul 14, 2021 12:03:51 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/Window.Into()/Window.Assign as step s13 Jul 14, 2021 12:03:51 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey as step s14 Jul 14, 2021 12:03:51 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/ExpandIterable as step s15 Jul 14, 2021 12:03:51 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/GlobalWindow/Window.Assign as step s16 Jul 14, 2021 12:03:51 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/StripShardId/Map as step s17 Jul 14, 2021 12:03:51 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/StreamingWrite/BatchedStreamingWrite.ViaBundleFinalization/ParMultiDo(BatchAndInsertElements) as step s18 Jul 14, 2021 12:03:51 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding CalculateUserScores/LeaderboardUserGlobalWindow/Window.Assign as step s19 Jul 14, 2021 12:03:51 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding CalculateUserScores/ExtractUserScore/MapElements/Map as step s20 Jul 14, 2021 12:03:51 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding CalculateUserScores/ExtractUserScore/Combine.perKey(SumInteger)/GroupByKey as step s21 Jul 14, 2021 12:03:51 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding CalculateUserScores/ExtractUserScore/Combine.perKey(SumInteger)/Combine.GroupedValues as step s22 Jul 14, 2021 12:03:51 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteUserScoreSums/ConvertToRow as step s23 Jul 14, 2021 12:03:51 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteUserScoreSums/BigQueryIO.Write/PrepareWrite/ParDo(Anonymous) as step s24 Jul 14, 2021 12:03:51 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/CreateTables/ParDo(CreateTables) as step s25 Jul 14, 2021 12:03:51 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/ShardTableWrites as step s26 Jul 14, 2021 12:03:51 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/TagWithUniqueIds as step s27 Jul 14, 2021 12:03:51 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/Window.Into()/Window.Assign as step s28 Jul 14, 2021 12:03:51 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey as step s29 Jul 14, 2021 12:03:51 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/ExpandIterable as step s30 Jul 14, 2021 12:03:51 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/GlobalWindow/Window.Assign as step s31 Jul 14, 2021 12:03:51 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/StripShardId/Map as step s32 Jul 14, 2021 12:03:51 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/StreamingWrite/BatchedStreamingWrite.ViaBundleFinalization/ParMultiDo(BatchAndInsertElements) as step s33 Jul 14, 2021 12:03:51 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Dataflow SDK version: 2.32.0-SNAPSHOT Jul 14, 2021 12:03:52 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2021-07-14_05_03_51-2433397458484263790?project=apache-beam-testing Jul 14, 2021 12:03:52 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Submitted job: 2021-07-14_05_03_51-2433397458484263790 Jul 14, 2021 12:03:52 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: To cancel the job using the 'gcloud' tool, run: > gcloud dataflow jobs --project=apache-beam-testing cancel > --region=us-central1 2021-07-14_05_03_51-2433397458484263790 Jul 14, 2021 12:04:05 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-07-14T12:04:03.226Z: Worker configuration: n1-standard-4 in us-central1-a. Jul 14, 2021 12:04:05 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-07-14T12:04:04.003Z: Expanding CoGroupByKey operations into optimizable parts. Jul 14, 2021 12:04:05 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-07-14T12:04:04.202Z: Expanding SplittableProcessKeyed operations into optimizable parts. Jul 14, 2021 12:04:05 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-07-14T12:04:04.238Z: Expanding GroupByKey operations into streaming Read/Write steps Jul 14, 2021 12:04:05 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-07-14T12:04:04.395Z: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns Jul 14, 2021 12:04:05 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-07-14T12:04:04.594Z: Fusing adjacent ParDo, Read, Write, and Flatten operations Jul 14, 2021 12:04:05 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-07-14T12:04:04.627Z: Fusing consumer CalculateUserScores/LeaderboardUserGlobalWindow/Window.Assign into ParseGameEvent Jul 14, 2021 12:04:05 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-07-14T12:04:04.658Z: Fusing consumer CalculateTeamScores/LeaderboardTeamFixedWindows/Window.Assign into ParseGameEvent Jul 14, 2021 12:04:05 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-07-14T12:04:04.681Z: Fusing consumer PubsubIO.Read/MapElements/Map into PubsubIO.Read/PubsubUnboundedSource Jul 14, 2021 12:04:05 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-07-14T12:04:04.707Z: Fusing consumer ParseGameEvent into PubsubIO.Read/MapElements/Map Jul 14, 2021 12:04:05 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-07-14T12:04:04.732Z: Fusing consumer CalculateTeamScores/ExtractTeamScore/MapElements/Map into CalculateTeamScores/LeaderboardTeamFixedWindows/Window.Assign Jul 14, 2021 12:04:05 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-07-14T12:04:04.757Z: Fusing consumer CalculateTeamScores/ExtractTeamScore/Combine.perKey(SumInteger)/GroupByKey/WriteStream into CalculateTeamScores/ExtractTeamScore/MapElements/Map Jul 14, 2021 12:04:05 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-07-14T12:04:04.786Z: Fusing consumer CalculateTeamScores/ExtractTeamScore/Combine.perKey(SumInteger)/Combine.GroupedValues into CalculateTeamScores/ExtractTeamScore/Combine.perKey(SumInteger)/GroupByKey/ReadStream Jul 14, 2021 12:04:05 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-07-14T12:04:04.820Z: Fusing consumer WriteTeamScoreSums/ConvertToRow into CalculateTeamScores/ExtractTeamScore/Combine.perKey(SumInteger)/Combine.GroupedValues Jul 14, 2021 12:04:05 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-07-14T12:04:04.853Z: Fusing consumer WriteTeamScoreSums/BigQueryIO.Write/PrepareWrite/ParDo(Anonymous) into WriteTeamScoreSums/ConvertToRow Jul 14, 2021 12:04:05 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-07-14T12:04:04.889Z: Fusing consumer WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/CreateTables/ParDo(CreateTables) into WriteTeamScoreSums/BigQueryIO.Write/PrepareWrite/ParDo(Anonymous) Jul 14, 2021 12:04:05 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-07-14T12:04:04.921Z: Fusing consumer WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/ShardTableWrites into WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/CreateTables/ParDo(CreateTables) Jul 14, 2021 12:04:05 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-07-14T12:04:04.953Z: Fusing consumer WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/TagWithUniqueIds into WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/ShardTableWrites Jul 14, 2021 12:04:05 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-07-14T12:04:04.975Z: Fusing consumer WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/Window.Into()/Window.Assign into WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/TagWithUniqueIds Jul 14, 2021 12:04:05 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-07-14T12:04:05.009Z: Fusing consumer WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey/WriteStream into WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/Window.Into()/Window.Assign Jul 14, 2021 12:04:05 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-07-14T12:04:05.045Z: Fusing consumer WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey/MergeBuckets into WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey/ReadStream Jul 14, 2021 12:04:05 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-07-14T12:04:05.078Z: Fusing consumer WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/ExpandIterable into WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey/MergeBuckets Jul 14, 2021 12:04:05 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-07-14T12:04:05.113Z: Fusing consumer WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/GlobalWindow/Window.Assign into WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/ExpandIterable Jul 14, 2021 12:04:05 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-07-14T12:04:05.159Z: Fusing consumer WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/StripShardId/Map into WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/GlobalWindow/Window.Assign Jul 14, 2021 12:04:05 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-07-14T12:04:05.189Z: Fusing consumer WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/StreamingWrite/BatchedStreamingWrite.ViaBundleFinalization/ParMultiDo(BatchAndInsertElements) into WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/StripShardId/Map Jul 14, 2021 12:04:05 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-07-14T12:04:05.224Z: Fusing consumer CalculateUserScores/ExtractUserScore/MapElements/Map into CalculateUserScores/LeaderboardUserGlobalWindow/Window.Assign Jul 14, 2021 12:04:05 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-07-14T12:04:05.255Z: Fusing consumer CalculateUserScores/ExtractUserScore/Combine.perKey(SumInteger)/GroupByKey/WriteStream into CalculateUserScores/ExtractUserScore/MapElements/Map Jul 14, 2021 12:04:05 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-07-14T12:04:05.289Z: Fusing consumer CalculateUserScores/ExtractUserScore/Combine.perKey(SumInteger)/Combine.GroupedValues into CalculateUserScores/ExtractUserScore/Combine.perKey(SumInteger)/GroupByKey/ReadStream Jul 14, 2021 12:04:05 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-07-14T12:04:05.329Z: Fusing consumer WriteUserScoreSums/ConvertToRow into CalculateUserScores/ExtractUserScore/Combine.perKey(SumInteger)/Combine.GroupedValues Jul 14, 2021 12:04:05 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-07-14T12:04:05.362Z: Fusing consumer WriteUserScoreSums/BigQueryIO.Write/PrepareWrite/ParDo(Anonymous) into WriteUserScoreSums/ConvertToRow Jul 14, 2021 12:04:05 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-07-14T12:04:05.387Z: Fusing consumer WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/CreateTables/ParDo(CreateTables) into WriteUserScoreSums/BigQueryIO.Write/PrepareWrite/ParDo(Anonymous) Jul 14, 2021 12:04:05 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-07-14T12:04:05.420Z: Fusing consumer WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/ShardTableWrites into WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/CreateTables/ParDo(CreateTables) Jul 14, 2021 12:04:05 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-07-14T12:04:05.447Z: Fusing consumer WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/TagWithUniqueIds into WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/ShardTableWrites Jul 14, 2021 12:04:05 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-07-14T12:04:05.481Z: Fusing consumer WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/Window.Into()/Window.Assign into WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/TagWithUniqueIds Jul 14, 2021 12:04:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-07-14T12:04:05.505Z: Fusing consumer WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey/WriteStream into WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/Window.Into()/Window.Assign Jul 14, 2021 12:04:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-07-14T12:04:05.530Z: Fusing consumer WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey/MergeBuckets into WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey/ReadStream Jul 14, 2021 12:04:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-07-14T12:04:05.575Z: Fusing consumer WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/ExpandIterable into WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey/MergeBuckets Jul 14, 2021 12:04:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-07-14T12:04:05.602Z: Fusing consumer WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/GlobalWindow/Window.Assign into WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/ExpandIterable Jul 14, 2021 12:04:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-07-14T12:04:05.645Z: Fusing consumer WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/StripShardId/Map into WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/GlobalWindow/Window.Assign Jul 14, 2021 12:04:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-07-14T12:04:05.685Z: Fusing consumer WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/StreamingWrite/BatchedStreamingWrite.ViaBundleFinalization/ParMultiDo(BatchAndInsertElements) into WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/StripShardId/Map Jul 14, 2021 12:04:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-07-14T12:04:05.732Z: The pubsub read for: projects/apache-beam-testing/subscriptions/java_mobile_gaming_topic.subscription-1652280944522589264 is configured to compute input data watermarks based on custom timestamp attribute timestamp_ms. Cloud Dataflow has created an additional tracking subscription to do this, which will be cleaned up automatically. For details, see: https://cloud.google.com/dataflow/model/pubsub-io#timestamps-ids Jul 14, 2021 12:04:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-07-14T12:04:06.096Z: Executing operation WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey/ReadStream+WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey/MergeBuckets+WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/ExpandIterable+WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/GlobalWindow/Window.Assign+WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/StripShardId/Map+WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/StreamingWrite/BatchedStreamingWrite.ViaBundleFinalization/ParMultiDo(BatchAndInsertElements) Jul 14, 2021 12:04:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-07-14T12:04:06.126Z: Executing operation CalculateUserScores/ExtractUserScore/Combine.perKey(SumInteger)/GroupByKey/ReadStream+CalculateUserScores/ExtractUserScore/Combine.perKey(SumInteger)/Combine.GroupedValues+WriteUserScoreSums/ConvertToRow+WriteUserScoreSums/BigQueryIO.Write/PrepareWrite/ParDo(Anonymous)+WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/CreateTables/ParDo(CreateTables)+WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/ShardTableWrites+WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/TagWithUniqueIds+WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/Window.Into()/Window.Assign+WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey/WriteStream Jul 14, 2021 12:04:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-07-14T12:04:06.159Z: Executing operation WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey/ReadStream+WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey/MergeBuckets+WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/ExpandIterable+WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/GlobalWindow/Window.Assign+WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/StripShardId/Map+WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/StreamingWrite/BatchedStreamingWrite.ViaBundleFinalization/ParMultiDo(BatchAndInsertElements) Jul 14, 2021 12:04:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-07-14T12:04:06.170Z: Starting 1 workers in us-central1-a... Jul 14, 2021 12:04:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-07-14T12:04:06.182Z: Executing operation CalculateTeamScores/ExtractTeamScore/Combine.perKey(SumInteger)/GroupByKey/ReadStream+CalculateTeamScores/ExtractTeamScore/Combine.perKey(SumInteger)/Combine.GroupedValues+WriteTeamScoreSums/ConvertToRow+WriteTeamScoreSums/BigQueryIO.Write/PrepareWrite/ParDo(Anonymous)+WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/CreateTables/ParDo(CreateTables)+WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/ShardTableWrites+WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/TagWithUniqueIds+WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/Window.Into()/Window.Assign+WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey/WriteStream Jul 14, 2021 12:04:10 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-07-14T12:04:09.094Z: Pub/Sub resources set up for topic 'projects/apache-beam-testing/topics/java_mobile_gaming_topic'. Jul 14, 2021 12:04:10 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-07-14T12:04:09.164Z: Executing operation PubsubIO.Read/PubsubUnboundedSource+PubsubIO.Read/MapElements/Map+ParseGameEvent+CalculateUserScores/LeaderboardUserGlobalWindow/Window.Assign+CalculateTeamScores/LeaderboardTeamFixedWindows/Window.Assign+CalculateTeamScores/ExtractTeamScore/MapElements/Map+CalculateTeamScores/ExtractTeamScore/Combine.perKey(SumInteger)/GroupByKey/WriteStream+CalculateUserScores/ExtractUserScore/MapElements/Map+CalculateUserScores/ExtractUserScore/Combine.perKey(SumInteger)/GroupByKey/WriteStream Jul 14, 2021 12:04:21 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-07-14T12:04:19.911Z: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete bq query SELECT table_id FROM beam_postrelease_mobile_gaming.__TABLES_SUMMARY__ BigQuery error in query operation: Could not connect with BigQuery server. Http response status: 502 Http response content: <!DOCTYPE html> <html lang=en> <meta charset=utf-8> <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width"> <title>Error 502 (Server Error)!!1</title> <style> *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;pa dding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/ 1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/brandin g/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-bo rder-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_1 50x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{ background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_1 50x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px} </style> <a href=//www.google.com/><span id=logo aria-label=Google></span></a> <p><b>502.</b> <ins>That\ufffd\ufffd\ufffds an error.</ins> <p>The server encountered a temporary error and could not complete your request.<p>Please try again in 30 seconds. <ins>That\ufffd\ufffd\ufffds all we know.</ins> BigQuery error in query operation: Could not connect with BigQuery server. Http response status: 502 Http response content: <!DOCTYPE html> <html lang=en> <meta charset=utf-8> <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width"> <title>Error 502 (Server Error)!!1</title> <style> *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;pa dding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/ 1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/brandin g/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-bo rder-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_1 50x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{ background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_1 50x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px} </style> <a href=//www.google.com/><span id=logo aria-label=Google></span></a> <p><b>502.</b> <ins>That\ufffd\ufffd\ufffds an error.</ins> <p>The server encountered a temporary error and could not complete your request.<p>Please try again in 30 seconds. <ins>That\ufffd\ufffd\ufffds all we know.</ins> [ERROR] Failed command > Task :runners:google-cloud-dataflow-java:runMobileGamingJavaDataflowBom FAILED FAILURE: Build failed with an exception. * What went wrong: Execution failed for task ':runners:google-cloud-dataflow-java:runMobileGamingJavaDataflowBom'. > Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with > non-zero exit value 1 * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. * Get more help at https://help.gradle.org Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/6.8.3/userguide/command_line_interface.html#sec:command_line_warnings BUILD FAILED in 1h 4m 20s 9 actionable tasks: 8 executed, 1 from cache Publishing build scan... https://gradle.com/s/k6s5vjzm2bley Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
