See
<https://ci-beam.apache.org/job/beam_PostRelease_NightlySnapshot/1450/display/redirect?page=changes>
Changes:
[Etienne Chauchot] [BEAM-12591] Put Spark Structured Streaming runner sources
back to main
[Etienne Chauchot] [BEAM-12629] As spark DataSourceV2 is only available for
spark 2,
[Etienne Chauchot] [BEAM-12627] Deal with spark Encoders braking change between
spark 2 and
[Etienne Chauchot] [BEAM-12591] move SchemaHelpers to correct package
[Etienne Chauchot] [BEAM-8470] Disable wait for termination in a streaming
pipeline because
[Etienne Chauchot] [BEAM-12630] Deal with breaking change in streaming
pipelines start by
[Etienne Chauchot] [BEAM-12629] Make source tests spark version agnostic and
move them back
[Etienne Chauchot] [BEAM-12629] Make a spark 3 source impl
[ajamato] [BEAM-12670] Relocate bq client exception imports to try block and
[Etienne Chauchot] [BEAM-12591] Fix checkstyle and spotless
[Etienne Chauchot] [BEAM-12629] Reduce serializable to only needed classes and
Fix schema
[Etienne Chauchot] [BEAM-12591] Add checkstyle exceptions for version specific
classes
[Etienne Chauchot] [BEAM-12629] Fix sources javadocs and improve impl
[dpires] [BEAM-12715] Use shard number specified by user in SnowflakeIO batch
[Etienne Chauchot] [BEAM-12591] Add spark 3 to structured streaming validates
runner tests
[relax] fix GroupIntoBatches
[noreply] [BEAM-12703] Fix universal metrics. (#15260)
[noreply] [BEAM-12702] Pull step unique names from pipeline for metrics.
(#15261)
[noreply] [BEAM-12678] Add dependency of java jars when running go VR on
portable
[noreply] [BEAM-12671] Mark known composite transforms native (#15236)
------------------------------------------
[...truncated 2.83 MB...]
Waiting on bqjob_rf0a0d3dc17015ac_0000017b16564880_1 ... (0s) Current status:
RUNNING
Waiting on bqjob_rf0a0d3dc17015ac_0000017b16564880_1 ... (0s) Current status:
DONE
+-----------------------------------+
| table_id |
+-----------------------------------+
| hourly_team_score_python_dataflow |
| hourly_team_score_python_direct |
| leaderboard_DirectRunner_team |
| leaderboard_DirectRunner_user |
+-----------------------------------+
Waiting for pipeline to produce more results...
Aug 05, 2021 12:42:42 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-08-05T12:42:40.739Z: Pub/Sub resources set up for topic
'projects/apache-beam-testing/topics/java_mobile_gaming_topic'.
Aug 05, 2021 12:43:06 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-08-05T12:43:05.415Z: Pub/Sub resources set up for topic
'projects/apache-beam-testing/topics/java_mobile_gaming_topic'.
Aug 05, 2021 12:43:30 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-08-05T12:43:28.175Z: Pub/Sub resources set up for topic
'projects/apache-beam-testing/topics/java_mobile_gaming_topic'.
Aug 05, 2021 12:43:40 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-08-05T12:43:39.986Z: Pub/Sub resources set up for topic
'projects/apache-beam-testing/topics/java_mobile_gaming_topic'.
bq query SELECT table_id FROM beam_postrelease_mobile_gaming.__TABLES_SUMMARY__
Waiting on bqjob_r74f8778b27d2f996_0000017b16573ebf_1 ... (0s) Current status:
RUNNING
Waiting on bqjob_r74f8778b27d2f996_0000017b16573ebf_1 ... (0s) Current status:
DONE
+-----------------------------------+
| table_id |
+-----------------------------------+
| hourly_team_score_python_dataflow |
| hourly_team_score_python_direct |
| leaderboard_DirectRunner_team |
| leaderboard_DirectRunner_user |
+-----------------------------------+
Waiting for pipeline to produce more results...
Aug 05, 2021 12:44:00 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-08-05T12:43:59.592Z: Pub/Sub resources set up for topic
'projects/apache-beam-testing/topics/java_mobile_gaming_topic'.
Aug 05, 2021 12:44:05 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-08-05T12:44:03.584Z: Pub/Sub resources set up for topic
'projects/apache-beam-testing/topics/java_mobile_gaming_topic'.
Aug 05, 2021 12:44:29 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-08-05T12:44:27.264Z: Pub/Sub resources set up for topic
'projects/apache-beam-testing/topics/java_mobile_gaming_topic'.
bq query SELECT table_id FROM beam_postrelease_mobile_gaming.__TABLES_SUMMARY__
Waiting on bqjob_r37cf3a2dfc7ca4dc_0000017b165834f2_1 ... (0s) Current status:
RUNNING
Waiting on bqjob_r37cf3a2dfc7ca4dc_0000017b165834f2_1 ... (0s) Current status:
DONE
+-----------------------------------+
| table_id |
+-----------------------------------+
| hourly_team_score_python_dataflow |
| hourly_team_score_python_direct |
| leaderboard_DirectRunner_team |
| leaderboard_DirectRunner_user |
+-----------------------------------+
Waiting for pipeline to produce more results...
Aug 05, 2021 12:44:52 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-08-05T12:44:51.791Z: Pub/Sub resources set up for topic
'projects/apache-beam-testing/topics/java_mobile_gaming_topic'.
Aug 05, 2021 12:45:14 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-08-05T12:45:13.131Z: Pub/Sub resources set up for topic
'projects/apache-beam-testing/topics/java_mobile_gaming_topic'.
Aug 05, 2021 12:45:31 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-08-05T12:45:31.122Z: Pub/Sub resources set up for topic
'projects/apache-beam-testing/topics/java_mobile_gaming_topic'.
bq query SELECT table_id FROM beam_postrelease_mobile_gaming.__TABLES_SUMMARY__
Waiting on bqjob_rf09210569ddbbc2_0000017b16592b4d_1 ... (0s) Current status:
RUNNING
Waiting on bqjob_rf09210569ddbbc2_0000017b16592b4d_1 ... (0s) Current status:
DONE
+-----------------------------------+
| table_id |
+-----------------------------------+
| hourly_team_score_python_dataflow |
| hourly_team_score_python_direct |
| leaderboard_DirectRunner_team |
| leaderboard_DirectRunner_user |
+-----------------------------------+
Waiting for pipeline to produce more results...
Aug 05, 2021 12:45:58 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-08-05T12:45:56.828Z: Pub/Sub resources set up for topic
'projects/apache-beam-testing/topics/java_mobile_gaming_topic'.
Aug 05, 2021 12:46:21 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-08-05T12:46:20.550Z: Pub/Sub resources set up for topic
'projects/apache-beam-testing/topics/java_mobile_gaming_topic'.
Aug 05, 2021 12:46:27 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-08-05T12:46:26.193Z: Pub/Sub resources set up for topic
'projects/apache-beam-testing/topics/java_mobile_gaming_topic'.
bq query SELECT table_id FROM beam_postrelease_mobile_gaming.__TABLES_SUMMARY__
Waiting on bqjob_r7a9fa74bf17ace75_0000017b165a21e1_1 ... (0s) Current status:
RUNNING
Waiting on bqjob_r7a9fa74bf17ace75_0000017b165a21e1_1 ... (0s) Current status:
DONE
+-----------------------------------+
| table_id |
+-----------------------------------+
| hourly_team_score_python_dataflow |
| hourly_team_score_python_direct |
| leaderboard_DirectRunner_team |
| leaderboard_DirectRunner_user |
+-----------------------------------+
Waiting for pipeline to produce more results...
Aug 05, 2021 12:46:58 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-08-05T12:46:56.637Z: Pub/Sub resources set up for topic
'projects/apache-beam-testing/topics/java_mobile_gaming_topic'.
Aug 05, 2021 12:47:17 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-08-05T12:47:16.471Z: Pub/Sub resources set up for topic
'projects/apache-beam-testing/topics/java_mobile_gaming_topic'.
.......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................DELAY(483780,
1)
{timestamp_ms=1628167154000}
late data for: Robot-13,AquaKangaroo,0,1628167154000,2021-08-05 05:47:18.388
Aug 05, 2021 12:47:30 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-08-05T12:47:28.981Z: Pub/Sub resources set up for topic
'projects/apache-beam-testing/topics/java_mobile_gaming_topic'.
Aug 05, 2021 12:47:32 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-08-05T12:47:32.202Z: Pub/Sub resources set up for topic
'projects/apache-beam-testing/topics/java_mobile_gaming_topic'.
Aug 05, 2021 12:47:45 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-08-05T12:47:45.156Z: Pub/Sub resources set up for topic
'projects/apache-beam-testing/topics/java_mobile_gaming_topic'.
bq query SELECT table_id FROM beam_postrelease_mobile_gaming.__TABLES_SUMMARY__
Waiting on bqjob_rb303a7dcf49ec75_0000017b165b1866_1 ... (0s) Current status:
RUNNING
Waiting on bqjob_rb303a7dcf49ec75_0000017b165b1866_1 ... (0s) Current status:
DONE
+-----------------------------------+
| table_id |
+-----------------------------------+
| hourly_team_score_python_dataflow |
| hourly_team_score_python_direct |
| leaderboard_DirectRunner_team |
| leaderboard_DirectRunner_user |
+-----------------------------------+
Waiting for pipeline to produce more results...
Aug 05, 2021 12:48:03 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-08-05T12:48:01.872Z: Pub/Sub resources set up for topic
'projects/apache-beam-testing/topics/java_mobile_gaming_topic'.
Aug 05, 2021 12:48:13 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-08-05T12:48:12.343Z: Pub/Sub resources set up for topic
'projects/apache-beam-testing/topics/java_mobile_gaming_topic'.
Aug 05, 2021 12:48:35 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-08-05T12:48:33.748Z: Pub/Sub resources set up for topic
'projects/apache-beam-testing/topics/java_mobile_gaming_topic'.
Aug 05, 2021 12:48:40 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-08-05T12:48:38.861Z: Pub/Sub resources set up for topic
'projects/apache-beam-testing/topics/java_mobile_gaming_topic'.
bq query SELECT table_id FROM beam_postrelease_mobile_gaming.__TABLES_SUMMARY__
Waiting on bqjob_r556fb87f1fd96251_0000017b165c0ee9_1 ... (0s) Current status:
RUNNING
Waiting on bqjob_r556fb87f1fd96251_0000017b165c0ee9_1 ... (0s) Current status:
DONE
+-----------------------------------+
| table_id |
+-----------------------------------+
| hourly_team_score_python_dataflow |
| hourly_team_score_python_direct |
| leaderboard_DirectRunner_team |
| leaderboard_DirectRunner_user |
+-----------------------------------+
Waiting for pipeline to produce more results...
Aug 05, 2021 12:49:01 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-08-05T12:49:00.009Z: Pub/Sub resources set up for topic
'projects/apache-beam-testing/topics/java_mobile_gaming_topic'.
Aug 05, 2021 12:49:27 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-08-05T12:49:25.020Z: Pub/Sub resources set up for topic
'projects/apache-beam-testing/topics/java_mobile_gaming_topic'.
Aug 05, 2021 12:49:30 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-08-05T12:49:30.158Z: Pub/Sub resources set up for topic
'projects/apache-beam-testing/topics/java_mobile_gaming_topic'.
Aug 05, 2021 12:49:54 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-08-05T12:49:54.210Z: Pub/Sub resources set up for topic
'projects/apache-beam-testing/topics/java_mobile_gaming_topic'.
bq query SELECT table_id FROM beam_postrelease_mobile_gaming.__TABLES_SUMMARY__
Waiting on bqjob_r7b598c8a5786bf3c_0000017b165d0542_1 ... (0s) Current status:
RUNNING
Waiting on bqjob_r7b598c8a5786bf3c_0000017b165d0542_1 ... (0s) Current status:
DONE
+-----------------------------------+
| table_id |
+-----------------------------------+
| hourly_team_score_python_dataflow |
| hourly_team_score_python_direct |
| leaderboard_DirectRunner_team |
| leaderboard_DirectRunner_user |
+-----------------------------------+
Waiting for pipeline to produce more results...
Aug 05, 2021 12:50:12 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-08-05T12:50:10.846Z: Pub/Sub resources set up for topic
'projects/apache-beam-testing/topics/java_mobile_gaming_topic'.
Aug 05, 2021 12:50:31 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-08-05T12:50:30.623Z: Pub/Sub resources set up for topic
'projects/apache-beam-testing/topics/java_mobile_gaming_topic'.
Aug 05, 2021 12:50:35 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-08-05T12:50:35.036Z: Pub/Sub resources set up for topic
'projects/apache-beam-testing/topics/java_mobile_gaming_topic'.
Aug 05, 2021 12:50:55 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-08-05T12:50:54.264Z: Pub/Sub resources set up for topic
'projects/apache-beam-testing/topics/java_mobile_gaming_topic'.
Aug 05, 2021 12:50:58 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-08-05T12:50:58.561Z: Pub/Sub resources set up for topic
'projects/apache-beam-testing/topics/java_mobile_gaming_topic'.
bq query SELECT table_id FROM beam_postrelease_mobile_gaming.__TABLES_SUMMARY__
Waiting on bqjob_r3cd8333f53ddf860_0000017b165dfc2b_1 ... (0s) Current status:
RUNNING
Waiting on bqjob_r3cd8333f53ddf860_0000017b165dfc2b_1 ... (0s) Current status:
DONE
+-----------------------------------+
| table_id |
+-----------------------------------+
| hourly_team_score_python_dataflow |
| hourly_team_score_python_direct |
| leaderboard_DataflowRunner_team |
| leaderboard_DataflowRunner_user |
| leaderboard_DirectRunner_team |
| leaderboard_DirectRunner_user |
+-----------------------------------+
bq query --batch "SELECT user FROM
[apache-beam-testing:beam_postrelease_mobile_gaming.leaderboard_DataflowRunner_user]
LIMIT 10"
Waiting on bqjob_r4feec19d4c72f908_0000017b165e086a_1 ... (0s) Current status:
RUNNING
Waiting on bqjob_r4feec19d4c72f908_0000017b165e086a_1 ... (0s) Current status:
DONE
+--------------------------+
| user |
+--------------------------+
| user13_AsparagusCaneToad |
| user4_BeigeKangaroo |
| user16_AsparagusCaneToad |
| user13_FuchsiaCassowary |
| user4_AntiqueBrassMarmot |
| user0_AuburnKoala |
| user2_AuburnKoala |
| user4_AuburnKoala |
| user12_AmethystQuokka |
| user5_AzureBandicoot |
+--------------------------+
Verified Amethyst
RUNNING_JOB=`gcloud dataflow jobs list | grep
leaderboard-validation-1628166720501-771 | grep Running | cut -d' ' -f1`
if [ ! -z "${RUNNING_JOB}" ]
then
gcloud dataflow jobs cancel ${RUNNING_JOB}
else
echo "Job 'leaderboard-validation-1628166720501-771' is not running."
fi
Job 'leaderboard-validation-1628166720501-771' is not running.
**************************************
* SUCCESS: LeaderBoard successfully run on DataflowRunner. with Streaming Engine
**************************************
[SUCCESS]
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task
':runners:google-cloud-dataflow-java:runMobileGamingJavaDataflowBom'.
> Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with
> non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug
option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with
Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See
https://docs.gradle.org/6.8.3/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 1h 50m 53s
9 actionable tasks: 8 executed, 1 from cache
Publishing build scan...
https://gradle.com/s/6ctvaibtire7m
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
