See
<https://ci-beam.apache.org/job/beam_PostRelease_NightlySnapshot/1556/display/redirect?page=changes>
Changes:
[brachipa] [BEAM-12737] error handling for sql calc error
[brachipa] [BEAM-12737] error handling for sql calc error
[brachipa] [BEAM-12737] error handling for sql calc error
[brachipa] [BEAM-12737] add test
[brachipa] [BEAM-12737] fix build
[brachipa] [BEAM-12737] fix build
[brachipa] revert
[brachipa] [beam-12737] cr fix
[brachipa] [beam-12737] cr fixes
[brachipa] [beam-12737] send errorTransformer to `buildPTransform`
[brachipa] [beam-12737] send errorTransformer to `buildPTransform`
[brachipa] [beam-12737] fix type `? extends POutput`
[Robert Bradshaw] Fully resolve type variables after PTransform type inference.
[brachipa] [beam-12737] add complex test, change error row type
[noreply] [BEAM-13226] Batch metrics parsing errors (#15972)
[noreply] [BEAM-13239] pass through io.EOF when decoding row (#15971)
[noreply] [BEAM-5378] Register types & funcs for examples. (#15970)
[Robert Bradshaw] Key-inferable type hints for CombinePerKey.
[Robert Bradshaw] Implement type variable binding for union types.
[Robert Bradshaw] Add comment about wildcard.
[noreply] [BEAM-3293] Add initial MultiMap side input backing type (#15975)
[Robert Bradshaw] Type inference deterministic coder fix for old Python
versions.
[Robert Bradshaw] Also implement type matching for union types.
[noreply] [BEAM-13238] Bump all Dataflow Java VR timeouts (#15979)
[noreply] Merge pull request #15978: [BEAM-8691] Upgrading Java deps compatible
[noreply] [BEAM-13221] Don't drop last data. (#15939)
------------------------------------------
[...truncated 3.10 MB...]
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
java.lang.RuntimeException:
com.google.api.client.googleapis.json.GoogleJsonResponseException: 404 Not Found
POST
https://bigquery.googleapis.com/bigquery/v2/projects/apache-beam-testing/datasets/beam_postrelease_mobile_gaming/tables/leaderboard_DataflowRunner_user/insertAll?prettyPrint=false
{
"code" : 404,
"errors" : [ {
"domain" : "global",
"message" : "Not found: table Table is deleted:
844138762903:beam_postrelease_mobile_gaming.leaderboard_DataflowRunner_user",
"reason" : "notFound"
} ],
"message" : "Not found: table Table is deleted:
844138762903:beam_postrelease_mobile_gaming.leaderboard_DataflowRunner_user",
"status" : "NOT_FOUND"
}
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.insertAll(BigQueryServicesImpl.java:1073)
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.insertAll(BigQueryServicesImpl.java:1126)
org.apache.beam.sdk.io.gcp.bigquery.BatchedStreamingWrite.flushRows(BatchedStreamingWrite.java:421)
org.apache.beam.sdk.io.gcp.bigquery.BatchedStreamingWrite.access$900(BatchedStreamingWrite.java:72)
org.apache.beam.sdk.io.gcp.bigquery.BatchedStreamingWrite$BatchAndInsertElements.finishBundle(BatchedStreamingWrite.java:267)
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException:
404 Not Found
POST
https://bigquery.googleapis.com/bigquery/v2/projects/apache-beam-testing/datasets/beam_postrelease_mobile_gaming/tables/leaderboard_DataflowRunner_user/insertAll?prettyPrint=false
{
"code" : 404,
"errors" : [ {
"domain" : "global",
"message" : "Not found: table Table is deleted:
844138762903:beam_postrelease_mobile_gaming.leaderboard_DataflowRunner_user",
"reason" : "notFound"
} ],
"message" : "Not found: table Table is deleted:
844138762903:beam_postrelease_mobile_gaming.leaderboard_DataflowRunner_user",
"status" : "NOT_FOUND"
}
com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:118)
com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:37)
com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:428)
com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1111)
com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:514)
com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:455)
com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:565)
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl$InsertBatchofRowsCallable.call(BigQueryServicesImpl.java:864)
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl$InsertBatchofRowsCallable.call(BigQueryServicesImpl.java:815)
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$BoundedExecutorService$SemaphoreCallable.call(BigQueryServicesImpl.java:1666)
java.util.concurrent.FutureTask.run(FutureTask.java:266)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
java.lang.RuntimeException:
com.google.api.client.googleapis.json.GoogleJsonResponseException: 404 Not Found
POST
https://bigquery.googleapis.com/bigquery/v2/projects/apache-beam-testing/datasets/beam_postrelease_mobile_gaming/tables/leaderboard_DataflowRunner_user/insertAll?prettyPrint=false
{
"code" : 404,
"errors" : [ {
"domain" : "global",
"message" : "Not found: table Table is deleted:
844138762903:beam_postrelease_mobile_gaming.leaderboard_DataflowRunner_user",
"reason" : "notFound"
} ],
"message" : "Not found: table Table is deleted:
844138762903:beam_postrelease_mobile_gaming.leaderboard_DataflowRunner_user",
"status" : "NOT_FOUND"
}
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.insertAll(BigQueryServicesImpl.java:1073)
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.insertAll(BigQueryServicesImpl.java:1126)
org.apache.beam.sdk.io.gcp.bigquery.BatchedStreamingWrite.flushRows(BatchedStreamingWrite.java:421)
org.apache.beam.sdk.io.gcp.bigquery.BatchedStreamingWrite.access$900(BatchedStreamingWrite.java:72)
org.apache.beam.sdk.io.gcp.bigquery.BatchedStreamingWrite$BatchAndInsertElements.finishBundle(BatchedStreamingWrite.java:267)
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException:
404 Not Found
POST
https://bigquery.googleapis.com/bigquery/v2/projects/apache-beam-testing/datasets/beam_postrelease_mobile_gaming/tables/leaderboard_DataflowRunner_user/insertAll?prettyPrint=false
{
"code" : 404,
"errors" : [ {
"domain" : "global",
"message" : "Not found: table Table is deleted:
844138762903:beam_postrelease_mobile_gaming.leaderboard_DataflowRunner_user",
"reason" : "notFound"
} ],
"message" : "Not found: table Table is deleted:
844138762903:beam_postrelease_mobile_gaming.leaderboard_DataflowRunner_user",
"status" : "NOT_FOUND"
}
com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:118)
com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:37)
com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:428)
com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1111)
com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:514)
com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:455)
com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:565)
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl$InsertBatchofRowsCallable.call(BigQueryServicesImpl.java:864)
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl$InsertBatchofRowsCallable.call(BigQueryServicesImpl.java:815)
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$BoundedExecutorService$SemaphoreCallable.call(BigQueryServicesImpl.java:1666)
java.util.concurrent.FutureTask.run(FutureTask.java:266)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
java.lang.RuntimeException:
com.google.api.client.googleapis.json.GoogleJsonResponseException: 404 Not Found
POST
https://bigquery.googleapis.com/bigquery/v2/projects/apache-beam-testing/datasets/beam_postrelease_mobile_gaming/tables/leaderboard_DataflowRunner_user/insertAll?prettyPrint=false
{
"code" : 404,
"errors" : [ {
"domain" : "global",
"message" : "Not found: table Table is deleted:
844138762903:beam_postrelease_mobile_gaming.leaderboard_DataflowRunner_user",
"reason" : "notFound"
} ],
"message" : "Not found: table Table is deleted:
844138762903:beam_postrelease_mobile_gaming.leaderboard_DataflowRunner_user",
"status" : "NOT_FOUND"
}
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.insertAll(BigQueryServicesImpl.java:1073)
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.insertAll(BigQueryServicesImpl.java:1126)
org.apache.beam.sdk.io.gcp.bigquery.BatchedStreamingWrite.flushRows(BatchedStreamingWrite.java:421)
org.apache.beam.sdk.io.gcp.bigquery.BatchedStreamingWrite.access$900(BatchedStreamingWrite.java:72)
org.apache.beam.sdk.io.gcp.bigquery.BatchedStreamingWrite$BatchAndInsertElements.finishBundle(BatchedStreamingWrite.java:267)
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException:
404 Not Found
POST
https://bigquery.googleapis.com/bigquery/v2/projects/apache-beam-testing/datasets/beam_postrelease_mobile_gaming/tables/leaderboard_DataflowRunner_user/insertAll?prettyPrint=false
{
"code" : 404,
"errors" : [ {
"domain" : "global",
"message" : "Not found: table Table is deleted:
844138762903:beam_postrelease_mobile_gaming.leaderboard_DataflowRunner_user",
"reason" : "notFound"
} ],
"message" : "Not found: table Table is deleted:
844138762903:beam_postrelease_mobile_gaming.leaderboard_DataflowRunner_user",
"status" : "NOT_FOUND"
}
com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:118)
com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:37)
com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:428)
com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1111)
com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:514)
com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:455)
com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:565)
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl$InsertBatchofRowsCallable.call(BigQueryServicesImpl.java:864)
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl$InsertBatchofRowsCallable.call(BigQueryServicesImpl.java:815)
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$BoundedExecutorService$SemaphoreCallable.call(BigQueryServicesImpl.java:1666)
java.util.concurrent.FutureTask.run(FutureTask.java:266)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
java.lang.RuntimeException:
com.google.api.client.googleapis.json.GoogleJsonResponseException: 404 Not Found
POST
https://bigquery.googleapis.com/bigquery/v2/projects/apache-beam-testing/datasets/beam_postrelease_mobile_gaming/tables/leaderboard_DataflowRunner_user/insertAll?prettyPrint=false
{
"code" : 404,
"errors" : [ {
"domain" : "global",
"message" : "Not found: table Table is deleted:
844138762903:beam_postrelease_mobile_gaming.leaderboard_DataflowRunner_user",
"reason" : "notFound"
} ],
"message" : "Not found: table Table is deleted:
844138762903:beam_postrelease_mobile_gaming.leaderboard_DataflowRunner_user",
"status" : "NOT_FOUND"
}
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.insertAll(BigQueryServicesImpl.java:1073)
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.insertAll(BigQueryServicesImpl.java:1126)
org.apache.beam.sdk.io.gcp.bigquery.BatchedStreamingWrite.flushRows(BatchedStreamingWrite.java:421)
org.apache.beam.sdk.io.gcp.bigquery.BatchedStreamingWrite.access$900(BatchedStreamingWrite.java:72)
org.apache.beam.sdk.io.gcp.bigquery.BatchedStreamingWrite$BatchAndInsertElements.finishBundle(BatchedStreamingWrite.java:267)
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException:
404 Not Found
POST
https://bigquery.googleapis.com/bigquery/v2/projects/apache-beam-testing/datasets/beam_postrelease_mobile_gaming/tables/leaderboard_DataflowRunner_user/insertAll?prettyPrint=false
{
"code" : 404,
"errors" : [ {
"domain" : "global",
"message" : "Not found: table Table is deleted:
844138762903:beam_postrelease_mobile_gaming.leaderboard_DataflowRunner_user",
"reason" : "notFound"
} ],
"message" : "Not found: table Table is deleted:
844138762903:beam_postrelease_mobile_gaming.leaderboard_DataflowRunner_user",
"status" : "NOT_FOUND"
}
com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:118)
com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:37)
com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:428)
com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1111)
com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:514)
com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:455)
com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:565)
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl$InsertBatchofRowsCallable.call(BigQueryServicesImpl.java:864)
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl$InsertBatchofRowsCallable.call(BigQueryServicesImpl.java:815)
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$BoundedExecutorService$SemaphoreCallable.call(BigQueryServicesImpl.java:1666)
jav
bq query SELECT table_id FROM beam_postrelease_mobile_gaming.__TABLES_SUMMARY__
Waiting on bqjob_r26a38b81aa0d1bee_0000017d28c1e895_1 ... (0s) Current status:
RUNNING
Waiting on bqjob_r26a38b81aa0d1bee_0000017d28c1e895_1 ... (0s) Current status:
DONE
+-----------------------------------+
| table_id |
+-----------------------------------+
| hourly_team_score_python_dataflow |
| hourly_team_score_python_direct |
| leaderboard_DataflowRunner_team |
| leaderboard_DataflowRunner_user |
| leaderboard_DirectRunner_team |
+-----------------------------------+
bq query --batch "SELECT user FROM
[apache-beam-testing:beam_postrelease_mobile_gaming.leaderboard_DataflowRunner_user]
LIMIT 10"
Waiting on bqjob_r63d8b6c7460940e2_0000017d28c1f41e_1 ... (0s) Current status:
RUNNING
Waiting on bqjob_r63d8b6c7460940e2_0000017d28c1f41e_1 ... (0s) Current status:
DONE
+-------------------------+
| user |
+-------------------------+
| user6_BananaMarmot |
| user6_AliceBlueEchidna |
| user5_AliceBlueEchidna |
| user2_BarnRedKangaroo |
| user5_BeigeDingo |
| user5_ArmyGreenCockatoo |
| user2_AzureEmu |
| user5_RubyCassowary |
| user1_AliceBlueEchidna |
| user2_RubyKangaroo |
+-------------------------+
Verified AliceBlue
RUNNING_JOB=`gcloud dataflow jobs list | grep
leaderboard-validation-1637065541785-346 | grep Running | cut -d' ' -f1`
if [ ! -z "${RUNNING_JOB}" ]
then
gcloud dataflow jobs cancel ${RUNNING_JOB}
else
echo "Job 'leaderboard-validation-1637065541785-346' is not running."
fi
Cancelled job [2021-11-16_04_26_01-10397521328464370477]
**************************************
* SUCCESS: LeaderBoard successfully run on DataflowRunner. with Streaming Engine
**************************************
[SUCCESS]
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':runners:direct-java:runMobileGamingJavaDirect'.
> Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with
> non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug
option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with
Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See
https://docs.gradle.org/6.9.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 1h 38m 49s
9 actionable tasks: 8 executed, 1 from cache
Publishing build scan...
https://gradle.com/s/trciuvt67af4s
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
