See
<https://ci-beam.apache.org/job/beam_PostRelease_NightlySnapshot/1514/display/redirect?page=changes>
Changes:
[Robert Bradshaw] Restore "Default to Runner v2 for Python Streaming jobs.
(#15140)"
[stefan.istrate] Fix "too many pings" errors.
[stefan.istrate] Increase keepalive timeout to 5 minutes.
[david.prieto] [BEAM-12950] Not delete orphaned files to avoid missing events
[david.prieto] [BEAM-12950] Add Bug fix description to CHANGES.md
[david.prieto] [BEAM-12950] fix linter issues
[Robert Bradshaw] Dead letter option.
[Robert Bradshaw] Guard setup.py logic with __main__ condition.
[stefan.istrate] Fix yapf complaints.
[Robert Bradshaw] Avoid incompatible setting.
[aydar.zaynutdinov] [BEAM-12969] [Playground]
[david.prieto] [BEAN-12950] Skip unit test
[noreply] [BEAM-12909][BEAM-12849] Add support for running spark3 nexmark
queries
[Robert Bradshaw] Add the ability to use subprocesses with the dead letter
queue.
[Robert Bradshaw] Support multi-output DoFns.
[noreply] Merge pull request #15510 from [BEAM-12883] Add coder for
[Robert Bradshaw] multi-output fix
[Robert Bradshaw] Add thresholding to dead letter pattern.
[Robert Bradshaw] treshold test fixes
[Robert Bradshaw] Better naming, documentation.
[noreply] [BEAM-12482] Ensure that we ignore schema update options when loading
[Kyle Weaver] Moving to 2.35.0-SNAPSHOT on master branch.
[Kyle Weaver] Add 2.35.0 section to changelog.
[noreply] Fix email links in the contact page
------------------------------------------
[...truncated 2.74 MB...]
late data for: Robot-17,AmethystWombat,8,1633610281000,2021-10-07 05:43:50.072
Oct 07, 2021 12:43:58 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-10-07T12:43:57.850Z: Pub/Sub resources set up for topic
'projects/apache-beam-testing/topics/java_mobile_gaming_topic'.
bq query SELECT table_id FROM beam_postrelease_mobile_gaming.__TABLES_SUMMARY__
Waiting on bqjob_r5c5aeecb68b57aaf_0000017c5ac875b6_1 ... (0s) Current status:
RUNNING
Waiting on bqjob_r5c5aeecb68b57aaf_0000017c5ac875b6_1 ... (0s) Current status:
DONE
+-----------------------------------+
| table_id |
+-----------------------------------+
| hourly_team_score_python_dataflow |
| hourly_team_score_python_direct |
| leaderboard_DirectRunner_team |
+-----------------------------------+
Waiting for pipeline to produce more results...
Oct 07, 2021 12:44:30 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-10-07T12:44:28.933Z: Pub/Sub resources set up for topic
'projects/apache-beam-testing/topics/java_mobile_gaming_topic'.
Oct 07, 2021 12:44:36 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-10-07T12:44:36.381Z: Pub/Sub resources set up for topic
'projects/apache-beam-testing/topics/java_mobile_gaming_topic'.
Oct 07, 2021 12:45:01 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-10-07T12:45:00.337Z: Pub/Sub resources set up for topic
'projects/apache-beam-testing/topics/java_mobile_gaming_topic'.
Oct 07, 2021 12:45:18 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-10-07T12:45:17.606Z: Pub/Sub resources set up for topic
'projects/apache-beam-testing/topics/java_mobile_gaming_topic'.
bq query SELECT table_id FROM beam_postrelease_mobile_gaming.__TABLES_SUMMARY__
Waiting on bqjob_r153fa7a018fbaa51_0000017c5ac96bfb_1 ... (0s) Current status:
RUNNING
Waiting on bqjob_r153fa7a018fbaa51_0000017c5ac96bfb_1 ... (0s) Current status:
DONE
+-----------------------------------+
| table_id |
+-----------------------------------+
| hourly_team_score_python_dataflow |
| hourly_team_score_python_direct |
| leaderboard_DirectRunner_team |
+-----------------------------------+
Waiting for pipeline to produce more results...
Oct 07, 2021 12:45:35 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-10-07T12:45:33.669Z: Pub/Sub resources set up for topic
'projects/apache-beam-testing/topics/java_mobile_gaming_topic'.
Oct 07, 2021 12:45:57 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-10-07T12:45:56.904Z: Pub/Sub resources set up for topic
'projects/apache-beam-testing/topics/java_mobile_gaming_topic'.
bq query SELECT table_id FROM beam_postrelease_mobile_gaming.__TABLES_SUMMARY__
Waiting on bqjob_r16837fb09e02ce4f_0000017c5aca626b_1 ... (0s) Current status:
RUNNING
Oct 07, 2021 12:46:27 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-10-07T12:46:27.055Z: Pub/Sub resources set up for topic
'projects/apache-beam-testing/topics/java_mobile_gaming_topic'.
Waiting on bqjob_r16837fb09e02ce4f_0000017c5aca626b_1 ... (0s) Current status:
DONE
+-----------------------------------+
| table_id |
+-----------------------------------+
| hourly_team_score_python_dataflow |
| hourly_team_score_python_direct |
| leaderboard_DirectRunner_team |
+-----------------------------------+
Waiting for pipeline to produce more results...
Oct 07, 2021 12:46:46 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-10-07T12:46:44.895Z: Pub/Sub resources set up for topic
'projects/apache-beam-testing/topics/java_mobile_gaming_topic'.
Oct 07, 2021 12:46:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-10-07T12:46:50.490Z: Pub/Sub resources set up for topic
'projects/apache-beam-testing/topics/java_mobile_gaming_topic'.
Oct 07, 2021 12:47:10 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-10-07T12:47:09.408Z: Pub/Sub resources set up for topic
'projects/apache-beam-testing/topics/java_mobile_gaming_topic'.
bq query SELECT table_id FROM beam_postrelease_mobile_gaming.__TABLES_SUMMARY__
Waiting on bqjob_r3d2abdf1aa93a003_0000017c5acb585f_1 ... (0s) Current status:
RUNNING
Waiting on bqjob_r3d2abdf1aa93a003_0000017c5acb585f_1 ... (0s) Current status:
DONE
+-----------------------------------+
| table_id |
+-----------------------------------+
| hourly_team_score_python_dataflow |
| hourly_team_score_python_direct |
| leaderboard_DirectRunner_team |
+-----------------------------------+
Waiting for pipeline to produce more results...
Oct 07, 2021 12:47:32 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-10-07T12:47:31.438Z: Pub/Sub resources set up for topic
'projects/apache-beam-testing/topics/java_mobile_gaming_topic'.
Oct 07, 2021 12:47:51 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-10-07T12:47:49.954Z: Pub/Sub resources set up for topic
'projects/apache-beam-testing/topics/java_mobile_gaming_topic'.
....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................Introducing
a parse error.
Oct 07, 2021 12:48:18 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-10-07T12:48:17.078Z: Pub/Sub resources set up for topic
'projects/apache-beam-testing/topics/java_mobile_gaming_topic'.
Oct 07, 2021 12:48:23 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-10-07T12:48:22.950Z: Pub/Sub resources set up for topic
'projects/apache-beam-testing/topics/java_mobile_gaming_topic'.
Oct 07, 2021 12:48:30 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-10-07T12:48:29.839Z: Pub/Sub resources set up for topic
'projects/apache-beam-testing/topics/java_mobile_gaming_topic'.
bq query SELECT table_id FROM beam_postrelease_mobile_gaming.__TABLES_SUMMARY__
Waiting on bqjob_r54598882dd2789f4_0000017c5acc4e69_1 ... (0s) Current status:
RUNNING
Waiting on bqjob_r54598882dd2789f4_0000017c5acc4e69_1 ... (0s) Current status:
DONE
+-----------------------------------+
| table_id |
+-----------------------------------+
| hourly_team_score_python_dataflow |
| hourly_team_score_python_direct |
| leaderboard_DirectRunner_team |
+-----------------------------------+
Waiting for pipeline to produce more results...
Oct 07, 2021 12:48:44 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-10-07T12:48:43.366Z: Pub/Sub resources set up for topic
'projects/apache-beam-testing/topics/java_mobile_gaming_topic'.
...................................................................DELAY(589219,
1)
{timestamp_ms=1633610341000}
late data for: user1_AquaNumbat,AquaNumbat,17,1633610341000,2021-10-07
05:48:50.662
Oct 07, 2021 12:48:51 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-10-07T12:48:49.038Z: Pub/Sub resources set up for topic
'projects/apache-beam-testing/topics/java_mobile_gaming_topic'.
Oct 07, 2021 12:49:03 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-10-07T12:49:01.554Z: Pub/Sub resources set up for topic
'projects/apache-beam-testing/topics/java_mobile_gaming_topic'.
Oct 07, 2021 12:49:33 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-10-07T12:49:32.523Z: Pub/Sub resources set up for topic
'projects/apache-beam-testing/topics/java_mobile_gaming_topic'.
bq query SELECT table_id FROM beam_postrelease_mobile_gaming.__TABLES_SUMMARY__
Waiting on bqjob_r23432a414bc35941_0000017c5acd5196_1 ... (0s) Current status:
RUNNING
Waiting on bqjob_r23432a414bc35941_0000017c5acd5196_1 ... (0s) Current status:
DONE
+-----------------------------------+
| table_id |
+-----------------------------------+
| hourly_team_score_python_dataflow |
| hourly_team_score_python_direct |
| leaderboard_DirectRunner_team |
+-----------------------------------+
Waiting for pipeline to produce more results...
Oct 07, 2021 12:49:53 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-10-07T12:49:52.730Z: Pub/Sub resources set up for topic
'projects/apache-beam-testing/topics/java_mobile_gaming_topic'.
Oct 07, 2021 12:50:14 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-10-07T12:50:13.307Z: Pub/Sub resources set up for topic
'projects/apache-beam-testing/topics/java_mobile_gaming_topic'.
bq query SELECT table_id FROM beam_postrelease_mobile_gaming.__TABLES_SUMMARY__
Oct 07, 2021 12:50:42 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-10-07T12:50:41.846Z: Pub/Sub resources set up for topic
'projects/apache-beam-testing/topics/java_mobile_gaming_topic'.
Waiting on bqjob_r33d085f2d91a9cfe_0000017c5ace47a7_1 ... (0s) Current status:
RUNNING
Waiting on bqjob_r33d085f2d91a9cfe_0000017c5ace47a7_1 ... (0s) Current status:
DONE
+-----------------------------------+
| table_id |
+-----------------------------------+
| hourly_team_score_python_dataflow |
| hourly_team_score_python_direct |
| leaderboard_DirectRunner_team |
+-----------------------------------+
Waiting for pipeline to produce more results...
Oct 07, 2021 12:50:49 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-10-07T12:50:47.580Z: Pub/Sub resources set up for topic
'projects/apache-beam-testing/topics/java_mobile_gaming_topic'.
Oct 07, 2021 12:50:53 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-10-07T12:50:51.419Z: Pub/Sub resources set up for topic
'projects/apache-beam-testing/topics/java_mobile_gaming_topic'.
Oct 07, 2021 12:50:54 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-10-07T12:50:54.557Z: Pub/Sub resources set up for topic
'projects/apache-beam-testing/topics/java_mobile_gaming_topic'.
Oct 07, 2021 12:51:09 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-10-07T12:51:06.903Z: Pub/Sub resources set up for topic
'projects/apache-beam-testing/topics/java_mobile_gaming_topic'.
Oct 07, 2021 12:51:15 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-10-07T12:51:14.656Z: Pub/Sub resources set up for topic
'projects/apache-beam-testing/topics/java_mobile_gaming_topic'.
bq query SELECT table_id FROM beam_postrelease_mobile_gaming.__TABLES_SUMMARY__
Oct 07, 2021 12:51:46 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-10-07T12:51:45.666Z: Pub/Sub resources set up for topic
'projects/apache-beam-testing/topics/java_mobile_gaming_topic'.
Waiting on bqjob_r25d17e91a57aa79a_0000017c5acf3da8_1 ... (0s) Current status:
RUNNING
Waiting on bqjob_r25d17e91a57aa79a_0000017c5acf3da8_1 ... (0s) Current status:
DONE
+-----------------------------------+
| table_id |
+-----------------------------------+
| hourly_team_score_python_dataflow |
| hourly_team_score_python_direct |
| leaderboard_DirectRunner_team |
+-----------------------------------+
Waiting for pipeline to produce more results...
Oct 07, 2021 12:52:01 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-10-07T12:52:00.943Z: Pub/Sub resources set up for topic
'projects/apache-beam-testing/topics/java_mobile_gaming_topic'.
Oct 07, 2021 12:52:10 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-10-07T12:52:10.228Z: Pub/Sub resources set up for topic
'projects/apache-beam-testing/topics/java_mobile_gaming_topic'.
Oct 07, 2021 12:52:43 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-10-07T12:52:41.416Z: Pub/Sub resources set up for topic
'projects/apache-beam-testing/topics/java_mobile_gaming_topic'.
bq query SELECT table_id FROM beam_postrelease_mobile_gaming.__TABLES_SUMMARY__
Waiting on bqjob_r2987bc502effde96_0000017c5ad033b1_1 ... (0s) Current status:
RUNNING
Waiting on bqjob_r2987bc502effde96_0000017c5ad033b1_1 ... (0s) Current status:
DONE
+-----------------------------------+
| table_id |
+-----------------------------------+
| hourly_team_score_python_dataflow |
| hourly_team_score_python_direct |
| leaderboard_DirectRunner_team |
+-----------------------------------+
Waiting for pipeline to produce more results...
Oct 07, 2021 12:53:03 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-10-07T12:53:02.053Z: Pub/Sub resources set up for topic
'projects/apache-beam-testing/topics/java_mobile_gaming_topic'.
Oct 07, 2021 12:53:22 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-10-07T12:53:22.470Z: Pub/Sub resources set up for topic
'projects/apache-beam-testing/topics/java_mobile_gaming_topic'.
RUNNING_JOB=`gcloud dataflow jobs list | grep
leaderboard-validation-1633610024095-730 | grep Running | cut -d' ' -f1`
if [ ! -z "${RUNNING_JOB}" ]
then
gcloud dataflow jobs cancel ${RUNNING_JOB}
else
echo "Job 'leaderboard-validation-1633610024095-730' is not running."
fi
Job 'leaderboard-validation-1633610024095-730' is not running.
[ERROR] FAILED: Failed running LeaderBoard on DataflowRunner with Streaming
Engine
> Task :runners:google-cloud-dataflow-java:runMobileGamingJavaDataflowBom FAILED
FAILURE: Build completed with 3 failures.
1: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':runners:direct-java:runMobileGamingJavaDirect'.
> Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with
> non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug
option to get more log output. Run with --scan to get full insights.
==============================================================================
2: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task
':runners:google-cloud-dataflow-java:runMobileGamingJavaDataflow'.
> Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with
> non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug
option to get more log output. Run with --scan to get full insights.
==============================================================================
3: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task
':runners:google-cloud-dataflow-java:runMobileGamingJavaDataflowBom'.
> Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with
> non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug
option to get more log output. Run with --scan to get full insights.
==============================================================================
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with
Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See
https://docs.gradle.org/6.9.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 1h 53m 33s
9 actionable tasks: 8 executed, 1 from cache
Publishing build scan...
https://gradle.com/s/ov52w7lhvr2ha
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
