See
<https://ci-beam.apache.org/job/beam_PostRelease_NightlySnapshot/1488/display/redirect?page=changes>
Changes:
[noreply] Register MapCoder, some comments/cleanup. (#15471)
[noreply] [BEAM-12588] Multimap user state proto changes (#15473)
[noreply] Added type annotations to some combiners missing it. (#15414)
[noreply] [BEAM-12634] JmsIO auto scaling feature (#15464)
[noreply] [BEAM-12662] Get Flink version from cluster. (#15223)
[noreply] Port changes from Pub/Sub Lite to beam (#15418)
[heejong] [BEAM-12805] Fix XLang CombinePerKey test by explicitly assigning the
[BenWhitehead] [BEAM-8376] Google Cloud Firestore Connector - Add handling for
[noreply] Decreasing peak memory usage for beam.TupleCombineFn (#15494)
[noreply] [BEAM-12802] Add support for prefetch through data layers down through
[noreply] [BEAM-11097] Add implementation of side input cache (#15483)
------------------------------------------
[...truncated 2.64 MB...]
**************************************
* SUCCESS: UserScore successfully run on DataflowRunner.
**************************************
gsutil rm
gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/java-userscore-result-dataflow-runner.txt*
Removing
gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/java-userscore-result-dataflow-runner.txt-00000-of-00001...
/ [1 objects]
Operation completed over 1 objects.
**************************************
* Test: Running: HourlyTeamScore example on DataflowRunner
**************************************
mvn compile exec:java -q -Dmaven.wagon.http.retryHandler.class=default
-Dmaven.wagon.http.retryHandler.count=5 -Dmaven.wagon.http.pool=false
-Dmaven.wagon.httpconnectionManager.ttlSeconds=120
-Dhttp.keepAlive=false
-Dexec.mainClass=org.apache.beam.examples.complete.game.HourlyTeamScore
-Dexec.args="--tempLocation=gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp
--runner=DataflowRunner
--input=gs://dataflow-samples/game/5000_gaming_data.csv
--project=apache-beam-testing --region=us-central1
--output=gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/java-hourlyteamscore-result-dataflow-runner.txt
" -Pdataflow-runner
Using maven /home/jenkins/tools/maven/apache-maven-3.5.4
Sep 11, 2021 12:05:33 PM
org.apache.beam.runners.dataflow.options.DataflowPipelineOptions$StagingLocationFactory
create
INFO: No stagingLocation provided, falling back to gcpTempLocation
Sep 11, 2021 12:05:34 PM org.apache.beam.runners.dataflow.DataflowRunner
fromOptions
INFO: PipelineOptions.filesToStage was not specified. Defaulting to files from
the classpath: will stage 228 files. Enable logging at DEBUG level to see which
files will be staged.
Sep 11, 2021 12:05:35 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Executing pipeline on the Dataflow Service, which will have billing
implications related to Google Compute Engine usage and other Google Cloud
Services.
Sep 11, 2021 12:05:38 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging portable pipeline proto to
gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/
Sep 11, 2021 12:05:38 PM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage
INFO: Uploading <235367 bytes, hash
449f6cacdc3aa5d5901514dd4b9f97ef58694693ae1d4342073cbb62ea40f7f5> to
gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/pipeline-RJ9srNw6pdWQFRTdS5-X71hpRpOuHUNCBzy7YupA9_U.pb
Sep 11, 2021 12:05:40 PM org.apache.beam.runners.dataflow.util.PackageUtil
stageClasspathElements
INFO: Uploading 228 files from PipelineOptions.filesToStage to staging location
to prepare for execution.
Sep 11, 2021 12:05:41 PM org.apache.beam.runners.dataflow.util.PackageUtil
stageClasspathElements
INFO: Staging files complete: 228 files cached, 0 files newly uploaded in 1
seconds
Sep 11, 2021 12:05:41 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding TextIO.Read/Read as step s1
Sep 11, 2021 12:05:41 PM org.apache.beam.sdk.io.FileBasedSource
getEstimatedSizeBytes
INFO: Filepattern gs://dataflow-samples/game/5000_gaming_data.csv matched 1
files with total size 386400
Sep 11, 2021 12:05:41 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding ParseGameEvent as step s2
Sep 11, 2021 12:05:42 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding FilterStartTime/ParDo(Anonymous) as step s3
Sep 11, 2021 12:05:42 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding FilterEndTime/ParDo(Anonymous) as step s4
Sep 11, 2021 12:05:42 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding AddEventTimestamps/AddTimestamps as step s5
Sep 11, 2021 12:05:42 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding FixedWindowsTeam/Window.Assign as step s6
Sep 11, 2021 12:05:42 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding ExtractTeamScore/MapElements/Map as step s7
Sep 11, 2021 12:05:42 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding ExtractTeamScore/Combine.perKey(SumInteger)/GroupByKey as step s8
Sep 11, 2021 12:05:42 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding ExtractTeamScore/Combine.perKey(SumInteger)/Combine.GroupedValues
as step s9
Sep 11, 2021 12:05:42 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteTeamScoreSums/ConvertToRow as step s10
Sep 11, 2021 12:05:42 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteShardedBundlesToTempFiles/ApplyShardingKey
as step s11
Sep 11, 2021 12:05:42 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteShardedBundlesToTempFiles/GroupIntoShards
as step s12
Sep 11, 2021 12:05:42 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteShardedBundlesToTempFiles/WriteShardsIntoTempFiles
as step s13
Sep 11, 2021 12:05:42 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Add
void key/AddKeys/Map as step s14
Sep 11, 2021 12:05:42 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/Window.Into()/Window.Assign
as step s15
Sep 11, 2021 12:05:42 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey
as step s16
Sep 11, 2021 12:05:42 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/ExpandIterable
as step s17
Sep 11, 2021 12:05:42 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Drop
key/Values/Map as step s18
Sep 11, 2021 12:05:42 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Gather
bundles as step s19
Sep 11, 2021 12:05:42 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Pair
with random key as step s20
Sep 11, 2021 12:05:42 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign
as step s21
Sep 11, 2021 12:05:42 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey
as step s22
Sep 11, 2021 12:05:42 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable
as step s23
Sep 11, 2021 12:05:42 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Values/Values/Map
as step s24
Sep 11, 2021 12:05:42 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Finalize
as step s25
Sep 11, 2021 12:05:42 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair
with random key as step s26
Sep 11, 2021 12:05:42 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign
as step s27
Sep 11, 2021 12:05:42 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey
as step s28
Sep 11, 2021 12:05:42 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable
as step s29
Sep 11, 2021 12:05:42 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Values/Values/Map
as step s30
Sep 11, 2021 12:05:43 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Dataflow SDK version: 2.34.0-SNAPSHOT
Sep 11, 2021 12:05:44 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-09-11_05_05_43-4775761313722959780?project=apache-beam-testing
Sep 11, 2021 12:05:44 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Submitted job: 2021-09-11_05_05_43-4775761313722959780
Sep 11, 2021 12:05:44 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel
> --region=us-central1 2021-09-11_05_05_43-4775761313722959780
Sep 11, 2021 12:05:49 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:46.659Z: Autoscaling is enabled for job
2021-09-11_05_05_43-4775761313722959780. The number of workers will be between
1 and 1000.
Sep 11, 2021 12:05:49 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:46.763Z: Autoscaling was automatically enabled for job
2021-09-11_05_05_43-4775761313722959780.
Sep 11, 2021 12:05:53 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:53.179Z: Worker configuration: e2-standard-2 in
us-central1-c.
Sep 11, 2021 12:05:56 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:54.085Z: Expanding CoGroupByKey operations into
optimizable parts.
Sep 11, 2021 12:05:56 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:54.311Z: Expanding GroupByKey operations into
optimizable parts.
Sep 11, 2021 12:05:56 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:54.350Z: Lifting ValueCombiningMappingFns into
MergeBucketsMappingFns
Sep 11, 2021 12:05:56 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:54.587Z: Fusing adjacent ParDo, Read, Write, and Flatten
operations
Sep 11, 2021 12:05:56 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:54.637Z: Fusing consumer ParseGameEvent into
TextIO.Read/Read
Sep 11, 2021 12:05:56 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:54.668Z: Fusing consumer
FilterStartTime/ParDo(Anonymous) into ParseGameEvent
Sep 11, 2021 12:05:56 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:54.703Z: Fusing consumer FilterEndTime/ParDo(Anonymous)
into FilterStartTime/ParDo(Anonymous)
Sep 11, 2021 12:05:56 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:54.751Z: Fusing consumer
AddEventTimestamps/AddTimestamps into FilterEndTime/ParDo(Anonymous)
Sep 11, 2021 12:05:56 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:54.790Z: Fusing consumer FixedWindowsTeam/Window.Assign
into AddEventTimestamps/AddTimestamps
Sep 11, 2021 12:05:56 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:54.825Z: Fusing consumer
ExtractTeamScore/MapElements/Map into FixedWindowsTeam/Window.Assign
Sep 11, 2021 12:05:56 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:54.855Z: Fusing consumer
ExtractTeamScore/Combine.perKey(SumInteger)/GroupByKey+ExtractTeamScore/Combine.perKey(SumInteger)/Combine.GroupedValues/Partial
into ExtractTeamScore/MapElements/Map
Sep 11, 2021 12:05:56 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:54.890Z: Fusing consumer
ExtractTeamScore/Combine.perKey(SumInteger)/GroupByKey/Reify into
ExtractTeamScore/Combine.perKey(SumInteger)/GroupByKey+ExtractTeamScore/Combine.perKey(SumInteger)/Combine.GroupedValues/Partial
Sep 11, 2021 12:05:56 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:54.913Z: Fusing consumer
ExtractTeamScore/Combine.perKey(SumInteger)/GroupByKey/Write into
ExtractTeamScore/Combine.perKey(SumInteger)/GroupByKey/Reify
Sep 11, 2021 12:05:56 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:54.945Z: Fusing consumer
ExtractTeamScore/Combine.perKey(SumInteger)/Combine.GroupedValues into
ExtractTeamScore/Combine.perKey(SumInteger)/GroupByKey/Read
Sep 11, 2021 12:05:56 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:54.978Z: Fusing consumer
ExtractTeamScore/Combine.perKey(SumInteger)/Combine.GroupedValues/Extract into
ExtractTeamScore/Combine.perKey(SumInteger)/Combine.GroupedValues
Sep 11, 2021 12:05:56 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:55.015Z: Fusing consumer WriteTeamScoreSums/ConvertToRow
into ExtractTeamScore/Combine.perKey(SumInteger)/Combine.GroupedValues/Extract
Sep 11, 2021 12:05:56 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:55.046Z: Fusing consumer
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteShardedBundlesToTempFiles/ApplyShardingKey
into WriteTeamScoreSums/ConvertToRow
Sep 11, 2021 12:05:56 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:55.083Z: Fusing consumer
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteShardedBundlesToTempFiles/GroupIntoShards/Reify
into
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteShardedBundlesToTempFiles/ApplyShardingKey
Sep 11, 2021 12:05:56 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:55.114Z: Fusing consumer
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteShardedBundlesToTempFiles/GroupIntoShards/Write
into
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteShardedBundlesToTempFiles/GroupIntoShards/Reify
Sep 11, 2021 12:05:56 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:55.147Z: Fusing consumer
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteShardedBundlesToTempFiles/GroupIntoShards/GroupByWindow
into
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteShardedBundlesToTempFiles/GroupIntoShards/Read
Sep 11, 2021 12:05:56 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:55.180Z: Fusing consumer
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteShardedBundlesToTempFiles/WriteShardsIntoTempFiles
into
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteShardedBundlesToTempFiles/GroupIntoShards/GroupByWindow
Sep 11, 2021 12:05:56 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:55.202Z: Fusing consumer
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Add
void key/AddKeys/Map into
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteShardedBundlesToTempFiles/WriteShardsIntoTempFiles
Sep 11, 2021 12:05:56 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:55.234Z: Fusing consumer
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/Window.Into()/Window.Assign
into
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Add
void key/AddKeys/Map
Sep 11, 2021 12:05:56 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:55.276Z: Fusing consumer
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/Reify
into
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/Window.Into()/Window.Assign
Sep 11, 2021 12:05:56 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:55.314Z: Fusing consumer
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/Write
into
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/Reify
Sep 11, 2021 12:05:56 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:55.345Z: Fusing consumer
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/GroupByWindow
into
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/Read
Sep 11, 2021 12:05:56 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:55.368Z: Fusing consumer
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/ExpandIterable
into
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/GroupByWindow
Sep 11, 2021 12:05:56 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:55.400Z: Fusing consumer
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Drop
key/Values/Map into
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/ExpandIterable
Sep 11, 2021 12:05:56 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:55.427Z: Fusing consumer
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Gather
bundles into
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Drop
key/Values/Map
Sep 11, 2021 12:05:56 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:55.450Z: Fusing consumer
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Pair
with random key into
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Gather
bundles
Sep 11, 2021 12:05:56 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:55.483Z: Fusing consumer
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign
into
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Pair
with random key
Sep 11, 2021 12:05:56 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:55.516Z: Fusing consumer
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify
into
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign
Sep 11, 2021 12:05:56 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:55.553Z: Fusing consumer
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write
into
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify
Sep 11, 2021 12:05:56 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:55.591Z: Fusing consumer
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow
into
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read
Sep 11, 2021 12:05:56 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:55.617Z: Fusing consumer
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable
into
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow
Sep 11, 2021 12:05:56 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:55.653Z: Fusing consumer
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Values/Values/Map
into
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable
Sep 11, 2021 12:05:56 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:55.686Z: Fusing consumer
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Finalize
into
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Values/Values/Map
Sep 11, 2021 12:05:56 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:55.722Z: Fusing consumer
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair
with random key into
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Finalize
Sep 11, 2021 12:05:56 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:55.756Z: Fusing consumer
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign
into
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair
with random key
Sep 11, 2021 12:05:56 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:55.789Z: Fusing consumer
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify
into
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign
Sep 11, 2021 12:05:56 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:55.824Z: Fusing consumer
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write
into
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify
Sep 11, 2021 12:05:56 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:55.850Z: Fusing consumer
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow
into
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read
Sep 11, 2021 12:05:56 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:55.874Z: Fusing consumer
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable
into
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow
Sep 11, 2021 12:05:56 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:55.929Z: Fusing consumer
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Values/Values/Map
into
WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable
Sep 11, 2021 12:05:56 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:56.325Z: Executing operation
ExtractTeamScore/Combine.perKey(SumInteger)/GroupByKey/Create
Sep 11, 2021 12:05:56 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:56.406Z: Starting 1 workers in us-central1-c...
Sep 11, 2021 12:05:59 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:56.852Z: Finished operation
ExtractTeamScore/Combine.perKey(SumInteger)/GroupByKey/Create
Sep 11, 2021 12:05:59 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:57.065Z: Executing operation
TextIO.Read/Read+ParseGameEvent+FilterStartTime/ParDo(Anonymous)+FilterEndTime/ParDo(Anonymous)+AddEventTimestamps/AddTimestamps+FixedWindowsTeam/Window.Assign+ExtractTeamScore/MapElements/Map+ExtractTeamScore/Combine.perKey(SumInteger)/GroupByKey+ExtractTeamScore/Combine.perKey(SumInteger)/Combine.GroupedValues/Partial+ExtractTeamScore/Combine.perKey(SumInteger)/GroupByKey/Reify+ExtractTeamScore/Combine.perKey(SumInteger)/GroupByKey/Write
Sep 11, 2021 12:06:25 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:06:25.126Z: Your project already contains 100
Dataflow-created metric descriptors, so new user metrics of the form
custom.googleapis.com/* will not be created. However, all user metrics are also
available in the metric dataflow.googleapis.com/job/user_counter. If you rely
on the custom metrics, you can delete old / unused metric descriptors. See
https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list
and
https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
Sep 11, 2021 12:06:31 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
SEVERE: 2021-09-11T12:06:30.676Z: Startup of the worker pool in zone
us-central1-c failed to bring up any of the desired 1 workers. QUOTA_EXCEEDED:
Quota 'CPUS' exceeded. Limit: 2500.0 in region us-central1.
Sep 11, 2021 12:06:31 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
SEVERE: 2021-09-11T12:06:30.707Z: Workflow failed.
Sep 11, 2021 12:06:31 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:06:30.796Z: Finished operation
TextIO.Read/Read+ParseGameEvent+FilterStartTime/ParDo(Anonymous)+FilterEndTime/ParDo(Anonymous)+AddEventTimestamps/AddTimestamps+FixedWindowsTeam/Window.Assign+ExtractTeamScore/MapElements/Map+ExtractTeamScore/Combine.perKey(SumInteger)/GroupByKey+ExtractTeamScore/Combine.perKey(SumInteger)/Combine.GroupedValues/Partial+ExtractTeamScore/Combine.perKey(SumInteger)/GroupByKey/Reify+ExtractTeamScore/Combine.perKey(SumInteger)/GroupByKey/Write
Sep 11, 2021 12:06:31 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:06:30.895Z: Cleaning up.
Sep 11, 2021 12:06:31 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:06:31.022Z: Stopping worker pool...
Sep 11, 2021 12:06:48 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:06:47.596Z: Worker pool stopped.
Sep 11, 2021 12:07:25 PM org.apache.beam.runners.dataflow.DataflowPipelineJob
logTerminalState
INFO: Job 2021-09-11_05_05_43-4775761313722959780 failed with status FAILED.
gsutil cat
gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/java-hourlyteamscore-result-dataflow-runner.txt*
| grep AzureBilby
CommandException: No URLs matched:
gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/java-hourlyteamscore-result-dataflow-runner.txt*
CommandException: No URLs matched:
gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/java-hourlyteamscore-result-dataflow-runner.txt*
[ERROR] Failed command
> Task :runners:google-cloud-dataflow-java:runMobileGamingJavaDataflow FAILED
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task
':runners:google-cloud-dataflow-java:runMobileGamingJavaDataflow'.
> Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with
> non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug
option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with
Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See
https://docs.gradle.org/6.8.3/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 1h 7m 16s
9 actionable tasks: 8 executed, 1 from cache
Publishing build scan...
https://gradle.com/s/iz4swn6iosdn6
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
