See <https://ci-beam.apache.org/job/beam_PostRelease_NightlySnapshot/1355/display/redirect?page=changes>
Changes: [Sylvain Wallez] [BEAM-12226] JdbcIO DefaultRetryStrategy: retry on PostgreSQL-specific [Kenneth Knowles] [BEAM-12231] Update Dataflow container to adjust for API changes [noreply] Clarify when to use `@DefaultCoder` (#14326) [noreply] [BEAM-12232] Expose public factory methods in SparkJobServerDriver ------------------------------------------ [...truncated 3.98 MB...] INFO: 2021-04-27T20:12:51.372Z: Fusing consumer WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/StreamingWrite/BatchedStreamingWrite.ViaBundleFinalization/ParMultiDo(BatchAndInsertElements) into WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/StripShardId/Map Apr 27, 2021 8:12:52 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-04-27T20:12:51.426Z: The pubsub read for: projects/apache-beam-testing/subscriptions/java_mobile_gaming_topic.subscription-8620046984745911465 is configured to compute input data watermarks based on custom timestamp attribute timestamp_ms. Cloud Dataflow has created an additional tracking subscription to do this, which will be cleaned up automatically. For details, see: https://cloud.google.com/dataflow/model/pubsub-io#timestamps-ids Apr 27, 2021 8:12:52 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-04-27T20:12:51.716Z: Starting 1 workers in us-central1-f... Apr 27, 2021 8:12:55 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-04-27T20:12:54.500Z: Pub/Sub resources set up for topic 'projects/apache-beam-testing/topics/java_mobile_gaming_topic'. Apr 27, 2021 8:13:08 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-04-27T20:13:06.068Z: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete bq query SELECT table_id FROM beam_postrelease_mobile_gaming.__TABLES_SUMMARY__ Waiting on bqjob_r4339b9a0c45f5b96_0000017914f72101_1 ... (0s) Current status: RUNNING Waiting on bqjob_r4339b9a0c45f5b96_0000017914f72101_1 ... (0s) Current status: DONE +-----------------------------------+ | table_id | +-----------------------------------+ | hourly_team_score_python_dataflow | | hourly_team_score_python_direct | | leaderboard_DirectRunner_team | +-----------------------------------+ Waiting for pipeline to produce more results... Apr 27, 2021 8:13:40 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-04-27T20:13:37.945Z: Autoscaling: Raised the number of workers to 1 so that the pipeline can catch up with its backlog and keep up with its input rate. Apr 27, 2021 8:14:09 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-04-27T20:14:07.950Z: Workers have started successfully. Apr 27, 2021 8:14:09 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-04-27T20:14:07.986Z: Workers have started successfully. ..............................................................................................................................................................................................................Introducing a parse error. bq query SELECT table_id FROM beam_postrelease_mobile_gaming.__TABLES_SUMMARY__ Waiting on bqjob_r66567297883299f2_0000017914f8192e_1 ... (0s) Current status: RUNNING Waiting on bqjob_r66567297883299f2_0000017914f8192e_1 ... (0s) Current status: DONE +-----------------------------------+ | table_id | +-----------------------------------+ | hourly_team_score_python_dataflow | | hourly_team_score_python_direct | | leaderboard_DirectRunner_team | +-----------------------------------+ Waiting for pipeline to produce more results... bq query SELECT table_id FROM beam_postrelease_mobile_gaming.__TABLES_SUMMARY__ Waiting on bqjob_r4eb6a67b6add036_0000017914f91168_1 ... (0s) Current status: RUNNING Waiting on bqjob_r4eb6a67b6add036_0000017914f91168_1 ... (0s) Current status: DONE +-----------------------------------+ | table_id | +-----------------------------------+ | hourly_team_score_python_dataflow | | hourly_team_score_python_direct | | leaderboard_DirectRunner_team | +-----------------------------------+ Waiting for pipeline to produce more results... bq query SELECT table_id FROM beam_postrelease_mobile_gaming.__TABLES_SUMMARY__ Waiting on bqjob_r317d9cdf4ca0cea6_0000017914fa0995_1 ... (0s) Current status: RUNNING Waiting on bqjob_r317d9cdf4ca0cea6_0000017914fa0995_1 ... (0s) Current status: DONE +-----------------------------------+ | table_id | +-----------------------------------+ | hourly_team_score_python_dataflow | | hourly_team_score_python_direct | | leaderboard_DataflowRunner_team | | leaderboard_DirectRunner_team | +-----------------------------------+ Waiting for pipeline to produce more results... .........................................................................................................................................................................................................................................................................................................................................................................................................DELAY(490944, 1) {timestamp_ms=1619554158000} late data for: user17_AzureKookaburra,AzureKookaburra,6,1619554158000,2021-04-27 13:17:29.859 bq query SELECT table_id FROM beam_postrelease_mobile_gaming.__TABLES_SUMMARY__ Waiting on bqjob_r53e5e0fd4b1829e_0000017914fb0090_1 ... (0s) Current status: RUNNING Waiting on bqjob_r53e5e0fd4b1829e_0000017914fb0090_1 ... (1s) Current status: RUNNING Waiting on bqjob_r53e5e0fd4b1829e_0000017914fb0090_1 ... (1s) Current status: DONE +-----------------------------------+ | table_id | +-----------------------------------+ | hourly_team_score_python_dataflow | | hourly_team_score_python_direct | | leaderboard_DataflowRunner_team | | leaderboard_DirectRunner_team | +-----------------------------------+ Waiting for pipeline to produce more results... bq query SELECT table_id FROM beam_postrelease_mobile_gaming.__TABLES_SUMMARY__ Waiting on bqjob_r20b2e250bd532555_0000017914fbfc89_1 ... (0s) Current status: RUNNING Waiting on bqjob_r20b2e250bd532555_0000017914fbfc89_1 ... (0s) Current status: DONE +-----------------------------------+ | table_id | +-----------------------------------+ | hourly_team_score_python_dataflow | | hourly_team_score_python_direct | | leaderboard_DataflowRunner_team | | leaderboard_DirectRunner_team | +-----------------------------------+ Waiting for pipeline to produce more results... bq query SELECT table_id FROM beam_postrelease_mobile_gaming.__TABLES_SUMMARY__ Waiting on bqjob_r4edfd412fbfafcb0_0000017914fcf457_1 ... (0s) Current status: RUNNING Waiting on bqjob_r4edfd412fbfafcb0_0000017914fcf457_1 ... (0s) Current status: DONE +-----------------------------------+ | table_id | +-----------------------------------+ | hourly_team_score_python_dataflow | | hourly_team_score_python_direct | | leaderboard_DataflowRunner_team | | leaderboard_DirectRunner_team | +-----------------------------------+ Waiting for pipeline to produce more results... .........................................................................................................................................................................................................................................................................................................................Introducing a parse error. bq query SELECT table_id FROM beam_postrelease_mobile_gaming.__TABLES_SUMMARY__ Waiting on bqjob_r27394325eed02807_0000017914fdeb8f_1 ... (0s) Current status: RUNNING Waiting on bqjob_r27394325eed02807_0000017914fdeb8f_1 ... (0s) Current status: DONE +-----------------------------------+ | table_id | +-----------------------------------+ | hourly_team_score_python_dataflow | | hourly_team_score_python_direct | | leaderboard_DataflowRunner_team | | leaderboard_DirectRunner_team | +-----------------------------------+ Waiting for pipeline to produce more results... bq query SELECT table_id FROM beam_postrelease_mobile_gaming.__TABLES_SUMMARY__ Waiting on bqjob_r3ccb8b6637cf8b9c_0000017914fee348_1 ... (0s) Current status: RUNNING Waiting on bqjob_r3ccb8b6637cf8b9c_0000017914fee348_1 ... (0s) Current status: DONE +-----------------------------------+ | table_id | +-----------------------------------+ | hourly_team_score_python_dataflow | | hourly_team_score_python_direct | | leaderboard_DataflowRunner_team | | leaderboard_DirectRunner_team | +-----------------------------------+ Waiting for pipeline to produce more results... ..............................................................................................................................................................................................................................................................................................DELAY(346122, 1) {timestamp_ms=1619554604000} late data for: user3_AmethystWallaby,AmethystWallaby,5,1619554604000,2021-04-27 13:22:30.468 bq query SELECT table_id FROM beam_postrelease_mobile_gaming.__TABLES_SUMMARY__ Waiting on bqjob_r52fa049f110abbb1_0000017914ffdc04_1 ... (0s) Current status: RUNNING Waiting on bqjob_r52fa049f110abbb1_0000017914ffdc04_1 ... (0s) Current status: DONE +-----------------------------------+ | table_id | +-----------------------------------+ | hourly_team_score_python_dataflow | | hourly_team_score_python_direct | | leaderboard_DataflowRunner_team | | leaderboard_DirectRunner_team | +-----------------------------------+ Waiting for pipeline to produce more results... bq query SELECT table_id FROM beam_postrelease_mobile_gaming.__TABLES_SUMMARY__ Waiting on bqjob_r3544b20e1fbd0100_000001791500d71a_1 ... (0s) Current status: RUNNING Waiting on bqjob_r3544b20e1fbd0100_000001791500d71a_1 ... (0s) Current status: DONE +-----------------------------------+ | table_id | +-----------------------------------+ | hourly_team_score_python_dataflow | | hourly_team_score_python_direct | | leaderboard_DataflowRunner_team | | leaderboard_DirectRunner_team | +-----------------------------------+ Waiting for pipeline to produce more results... bq query SELECT table_id FROM beam_postrelease_mobile_gaming.__TABLES_SUMMARY__ Waiting on bqjob_r758b7793fff132a2_000001791501ce6e_1 ... (0s) Current status: RUNNING Waiting on bqjob_r758b7793fff132a2_000001791501ce6e_1 ... (0s) Current status: DONE +-----------------------------------+ | table_id | +-----------------------------------+ | hourly_team_score_python_dataflow | | hourly_team_score_python_direct | | leaderboard_DataflowRunner_team | | leaderboard_DataflowRunner_user | | leaderboard_DirectRunner_team | +-----------------------------------+ bq query --batch "SELECT user FROM [apache-beam-testing:beam_postrelease_mobile_gaming.leaderboard_DataflowRunner_user] LIMIT 10" Waiting on bqjob_r40d4e689ac3027b7_000001791501dcee_1 ... (0s) Current status: RUNNING Waiting on bqjob_r40d4e689ac3027b7_000001791501dcee_1 ... (0s) Current status: DONE +------------------------------+ | user | +------------------------------+ | user2_BattleshipGreyPlatypus | | user6_BananaEmu | | user5_AuburnWombat | | user4_AzureKookaburra | | user3_RubyBandicoot | | user1_AlmondQuokka | | user7_AuburnPlatypus | | user0_AzureEchidna | | user0_AuburnNumbat | | user5_BattleshipGreyDingo | +------------------------------+ Verified Almond RUNNING_JOB=`gcloud dataflow jobs list | grep leaderboard-validation-1619554344230-469 | grep Running | cut -d' ' -f1` if [ ! -z "${RUNNING_JOB}" ] then gcloud dataflow jobs cancel ${RUNNING_JOB} else echo "Job 'leaderboard-validation-1619554344230-469' is not running." fi Cancelled job [2021-04-27_13_12_38-10244767523801253557] ************************************** * SUCCESS: LeaderBoard successfully run on DataflowRunner. with Streaming Engine ************************************** [SUCCESS] FAILURE: Build failed with an exception. * What went wrong: Execution failed for task ':runners:direct-java:runMobileGamingJavaDirect'. > Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with > non-zero exit value 1 * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. * Get more help at https://help.gradle.org Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/6.8.3/userguide/command_line_interface.html#sec:command_line_warnings BUILD FAILED in 1h 35m 17s 9 actionable tasks: 8 executed, 1 from cache Publishing build scan... https://gradle.com/s/ghgqftmelejt6 Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
