See <https://ci-beam.apache.org/job/beam_PostRelease_NightlySnapshot/1324/display/redirect?page=changes>
Changes: [Sylvain Wallez] [BEAM-12044] JdbcIO read: always force autocommit to false ------------------------------------------ [...truncated 5.30 MB...] INFO: 2021-03-28T12:14:35.130Z: Fusing consumer WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/ShardTableWrites into WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/CreateTables/ParDo(CreateTables) Mar 28, 2021 12:14:37 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-03-28T12:14:35.161Z: Fusing consumer WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/TagWithUniqueIds into WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/ShardTableWrites Mar 28, 2021 12:14:37 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-03-28T12:14:35.185Z: Fusing consumer WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/Window.Into()/Window.Assign into WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/TagWithUniqueIds Mar 28, 2021 12:14:37 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-03-28T12:14:35.218Z: Fusing consumer WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey/WriteStream into WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/Window.Into()/Window.Assign Mar 28, 2021 12:14:37 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-03-28T12:14:35.255Z: Fusing consumer WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey/MergeBuckets into WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey/ReadStream Mar 28, 2021 12:14:37 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-03-28T12:14:35.287Z: Fusing consumer WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/ExpandIterable into WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey/MergeBuckets Mar 28, 2021 12:14:37 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-03-28T12:14:35.310Z: Fusing consumer WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/GlobalWindow/Window.Assign into WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/ExpandIterable Mar 28, 2021 12:14:37 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-03-28T12:14:35.336Z: Fusing consumer WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/StripShardId/Map into WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/GlobalWindow/Window.Assign Mar 28, 2021 12:14:37 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-03-28T12:14:35.365Z: Fusing consumer WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/StreamingWrite/BatchedStreamingWrite.ViaBundleFinalization/ParMultiDo(BatchAndInsertElements) into WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/StripShardId/Map Mar 28, 2021 12:14:37 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-03-28T12:14:35.414Z: The pubsub read for: projects/apache-beam-testing/subscriptions/java_mobile_gaming_topic.subscription-14765760150214997391 is configured to compute input data watermarks based on custom timestamp attribute timestamp_ms. Cloud Dataflow has created an additional tracking subscription to do this, which will be cleaned up automatically. For details, see: https://cloud.google.com/dataflow/model/pubsub-io#timestamps-ids Mar 28, 2021 12:14:39 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-03-28T12:14:38.028Z: Pub/Sub resources set up for topic 'projects/apache-beam-testing/topics/java_mobile_gaming_topic'. Mar 28, 2021 12:14:39 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-03-28T12:14:39.255Z: Starting 1 workers in us-central1-f... Mar 28, 2021 12:14:42 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-03-28T12:14:40.904Z: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete bq query SELECT table_id FROM beam_postrelease_mobile_gaming.__TABLES_SUMMARY__ Waiting on bqjob_r1f66fed86faf9f5b_0000017878c250e6_1 ... (0s) Current status: RUNNING Waiting on bqjob_r1f66fed86faf9f5b_0000017878c250e6_1 ... (0s) Current status: DONE +-----------------------------------+ | table_id | +-----------------------------------+ | hourly_team_score_python_dataflow | | hourly_team_score_python_direct | | leaderboard_DataflowRunner_user | | leaderboard_DirectRunner_team | | leaderboard_DirectRunner_user | +-----------------------------------+ Waiting for pipeline to produce more results... Mar 28, 2021 12:15:14 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-03-28T12:15:13.322Z: Autoscaling: Raised the number of workers to 1 so that the pipeline can catch up with its backlog and keep up with its input rate. Mar 28, 2021 12:16:03 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-03-28T12:16:03.033Z: Workers have started successfully. Mar 28, 2021 12:16:03 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-03-28T12:16:03.062Z: Workers have started successfully. bq query SELECT table_id FROM beam_postrelease_mobile_gaming.__TABLES_SUMMARY__ Waiting on bqjob_r56e03c2a3018a2b0_0000017878c34a27_1 ... (0s) Current status: RUNNING Waiting on bqjob_r56e03c2a3018a2b0_0000017878c34a27_1 ... (0s) Current status: DONE +-----------------------------------+ | table_id | +-----------------------------------+ | hourly_team_score_python_dataflow | | hourly_team_score_python_direct | | leaderboard_DataflowRunner_team | | leaderboard_DataflowRunner_user | | leaderboard_DirectRunner_team | | leaderboard_DirectRunner_user | +-----------------------------------+ bq query --batch "SELECT user FROM [apache-beam-testing:beam_postrelease_mobile_gaming.leaderboard_DataflowRunner_user] LIMIT 10" Waiting on bqjob_r2ae67f6114e672ab_0000017878c358b1_1 ... (0s) Current status: PENDING Waiting on bqjob_r2ae67f6114e672ab_0000017878c358b1_1 ... (1s) Current status: PENDING Waiting on bqjob_r2ae67f6114e672ab_0000017878c358b1_1 ... (2s) Current status: PENDING Waiting on bqjob_r2ae67f6114e672ab_0000017878c358b1_1 ... (3s) Current status: PENDING Waiting on bqjob_r2ae67f6114e672ab_0000017878c358b1_1 ... (4s) Current status: PENDING Waiting on bqjob_r2ae67f6114e672ab_0000017878c358b1_1 ... (5s) Current status: PENDING Waiting on bqjob_r2ae67f6114e672ab_0000017878c358b1_1 ... (6s) Current status: PENDING Waiting on bqjob_r2ae67f6114e672ab_0000017878c358b1_1 ... (7s) Current status: PENDING Waiting on bqjob_r2ae67f6114e672ab_0000017878c358b1_1 ... (8s) Current status: PENDING Waiting on bqjob_r2ae67f6114e672ab_0000017878c358b1_1 ... (9s) Current status: PENDING Waiting on bqjob_r2ae67f6114e672ab_0000017878c358b1_1 ... (10s) Current status: PENDING Waiting on bqjob_r2ae67f6114e672ab_0000017878c358b1_1 ... (11s) Current status: PENDING Waiting on bqjob_r2ae67f6114e672ab_0000017878c358b1_1 ... (12s) Current status: PENDING Waiting on bqjob_r2ae67f6114e672ab_0000017878c358b1_1 ... (13s) Current status: PENDING Waiting on bqjob_r2ae67f6114e672ab_0000017878c358b1_1 ... (14s) Current status: PENDING Waiting on bqjob_r2ae67f6114e672ab_0000017878c358b1_1 ... (15s) Current status: PENDING Waiting on bqjob_r2ae67f6114e672ab_0000017878c358b1_1 ... (16s) Current status: PENDING Waiting on bqjob_r2ae67f6114e672ab_0000017878c358b1_1 ... (17s) Current status: PENDING Waiting on bqjob_r2ae67f6114e672ab_0000017878c358b1_1 ... (18s) Current status: PENDING Waiting on bqjob_r2ae67f6114e672ab_0000017878c358b1_1 ... (19s) Current status: PENDING Waiting on bqjob_r2ae67f6114e672ab_0000017878c358b1_1 ... (20s) Current status: PENDING Waiting on bqjob_r2ae67f6114e672ab_0000017878c358b1_1 ... (21s) Current status: PENDING Waiting on bqjob_r2ae67f6114e672ab_0000017878c358b1_1 ... (22s) Current status: PENDING Waiting on bqjob_r2ae67f6114e672ab_0000017878c358b1_1 ... (23s) Current status: PENDING Waiting on bqjob_r2ae67f6114e672ab_0000017878c358b1_1 ... (24s) Current status: PENDING Waiting on bqjob_r2ae67f6114e672ab_0000017878c358b1_1 ... (25s) Current status: PENDING Waiting on bqjob_r2ae67f6114e672ab_0000017878c358b1_1 ... (26s) Current status: PENDING Waiting on bqjob_r2ae67f6114e672ab_0000017878c358b1_1 ... (27s) Current status: PENDING Waiting on bqjob_r2ae67f6114e672ab_0000017878c358b1_1 ... (28s) Current status: PENDING Waiting on bqjob_r2ae67f6114e672ab_0000017878c358b1_1 ... (29s) Current status: PENDING Waiting on bqjob_r2ae67f6114e672ab_0000017878c358b1_1 ... (30s) Current status: PENDING Waiting on bqjob_r2ae67f6114e672ab_0000017878c358b1_1 ... (31s) Current status: PENDING Waiting on bqjob_r2ae67f6114e672ab_0000017878c358b1_1 ... (32s) Current status: PENDING Waiting on bqjob_r2ae67f6114e672ab_0000017878c358b1_1 ... (33s) Current status: PENDING Waiting on bqjob_r2ae67f6114e672ab_0000017878c358b1_1 ... (34s) Current status: PENDING Waiting on bqjob_r2ae67f6114e672ab_0000017878c358b1_1 ... (35s) Current status: PENDING Waiting on bqjob_r2ae67f6114e672ab_0000017878c358b1_1 ... (36s) Current status: PENDING Waiting on bqjob_r2ae67f6114e672ab_0000017878c358b1_1 ... (37s) Current status: PENDING Waiting on bqjob_r2ae67f6114e672ab_0000017878c358b1_1 ... (38s) Current status: PENDING Waiting on bqjob_r2ae67f6114e672ab_0000017878c358b1_1 ... (39s) Current status: PENDING Waiting on bqjob_r2ae67f6114e672ab_0000017878c358b1_1 ... (40s) Current status: PENDING Waiting on bqjob_r2ae67f6114e672ab_0000017878c358b1_1 ... (41s) Current status: PENDING Waiting on bqjob_r2ae67f6114e672ab_0000017878c358b1_1 ... (42s) Current status: PENDING Waiting on bqjob_r2ae67f6114e672ab_0000017878c358b1_1 ... (43s) Current status: PENDING Waiting on bqjob_r2ae67f6114e672ab_0000017878c358b1_1 ... (44s) Current status: PENDING Waiting on bqjob_r2ae67f6114e672ab_0000017878c358b1_1 ... (45s) Current status: PENDING Waiting on bqjob_r2ae67f6114e672ab_0000017878c358b1_1 ... (46s) Current status: PENDING Waiting on bqjob_r2ae67f6114e672ab_0000017878c358b1_1 ... (47s) Current status: PENDING Waiting on bqjob_r2ae67f6114e672ab_0000017878c358b1_1 ... (48s) Current status: PENDING Waiting on bqjob_r2ae67f6114e672ab_0000017878c358b1_1 ... (49s) Current status: PENDING Waiting on bqjob_r2ae67f6114e672ab_0000017878c358b1_1 ... (50s) Current status: PENDING Waiting on bqjob_r2ae67f6114e672ab_0000017878c358b1_1 ... (51s) Current status: PENDING Waiting on bqjob_r2ae67f6114e672ab_0000017878c358b1_1 ... (52s) Current status: PENDING Waiting on bqjob_r2ae67f6114e672ab_0000017878c358b1_1 ... (53s) Current status: PENDING Waiting on bqjob_r2ae67f6114e672ab_0000017878c358b1_1 ... (54s) Current status: PENDING Waiting on bqjob_r2ae67f6114e672ab_0000017878c358b1_1 ... (55s) Current status: PENDING ...............................................................................................................................................................................................................................................................................................................................Introducing a parse error. Waiting on bqjob_r2ae67f6114e672ab_0000017878c358b1_1 ... (56s) Current status: PENDING Waiting on bqjob_r2ae67f6114e672ab_0000017878c358b1_1 ... (57s) Current status: PENDING Waiting on bqjob_r2ae67f6114e672ab_0000017878c358b1_1 ... (58s) Current status: PENDING Waiting on bqjob_r2ae67f6114e672ab_0000017878c358b1_1 ... (59s) Current status: PENDING Waiting on bqjob_r2ae67f6114e672ab_0000017878c358b1_1 ... (60s) Current status: PENDING Waiting on bqjob_r2ae67f6114e672ab_0000017878c358b1_1 ... (61s) Current status: PENDING Waiting on bqjob_r2ae67f6114e672ab_0000017878c358b1_1 ... (62s) Current status: PENDING Waiting on bqjob_r2ae67f6114e672ab_0000017878c358b1_1 ... (63s) Current status: PENDING Waiting on bqjob_r2ae67f6114e672ab_0000017878c358b1_1 ... (64s) Current status: PENDING Waiting on bqjob_r2ae67f6114e672ab_0000017878c358b1_1 ... (64s) Current status: DONE +------------------------------+ | user | +------------------------------+ | user16_AndroidGreenCassowary | | user1_BisqueCockatoo | | user4_AliceBlueCockatoo | | user1_AndroidGreenBandicoot | | user3_AlmondMarmot | | user13_FuchsiaKangaroo | | user0_BisquePlatypus | | user8_AndroidGreenBandicoot | | user1_AlmondMarmot | | user2_AliceBlueCockatoo | +------------------------------+ Verified AliceBlue RUNNING_JOB=`gcloud dataflow jobs list | grep leaderboard-validation-1616933634845-118 | grep Running | cut -d' ' -f1` if [ ! -z "${RUNNING_JOB}" ] then gcloud dataflow jobs cancel ${RUNNING_JOB} else echo "Job 'leaderboard-validation-1616933634845-118' is not running." fi Cancelled job [2021-03-28_05_14_23-13216340680240304156] ************************************** * SUCCESS: LeaderBoard successfully run on DataflowRunner. with Streaming Engine ************************************** [SUCCESS] FAILURE: Build failed with an exception. * What went wrong: Execution failed for task ':runners:google-cloud-dataflow-java:runMobileGamingJavaDataflow'. > Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with > non-zero exit value 1 * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. * Get more help at https://help.gradle.org Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/6.8/userguide/command_line_interface.html#sec:command_line_warnings BUILD FAILED in 1h 17m 5s 9 actionable tasks: 8 executed, 1 from cache Publishing build scan... https://gradle.com/s/lzcfzedv6kcga Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
