See <https://ci-beam.apache.org/job/beam_PostRelease_NightlySnapshot/1486/display/redirect?page=changes>
Changes: [suztomo] [BEAM-11205] Upgrading the Libraries BOM to v22 [randomstep] [BEAM-12708] Bump arrow-memory-netty [Etienne Chauchot] [BEAM-12153] implement GroupByKey with CombinePerKey with Concatenate [Etienne Chauchot] [BEAM-11023] Increase memory in SS Validates runner tests to avoid OOM [vincent.marquez] [BEAM-9008] adds CassandraIO.readAll [Etienne Chauchot] [BEAM-12727] extract Concatenate CombineFn to runner-core module to [noreply] Add display data for JdbcIO.write (#15460) [noreply] Merge pull request #15480: [BEAM-12356] Make sure DatasetService is [noreply] [BEAM-11981] Java Bigtable - Implement IO Request Count metrics (#15342) [noreply] [BEAM-12834] Improve Go SDK cross-language documentation and API. ------------------------------------------ [...truncated 2.67 MB...] Sep 09, 2021 12:01:50 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-09T12:01:48.466Z: Fusing consumer WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles into WriteUserScoreSums/TextIO.Write/WriteFiles/RewindowIntoGlobal/Window.Assign Sep 09, 2021 12:01:50 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-09T12:01:48.499Z: Fusing consumer WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Reify into WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles Sep 09, 2021 12:01:50 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-09T12:01:48.534Z: Fusing consumer WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Write into WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Reify Sep 09, 2021 12:01:50 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-09T12:01:48.584Z: Fusing consumer WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/GroupByWindow into WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Read Sep 09, 2021 12:01:50 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-09T12:01:48.636Z: Fusing consumer WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnwritten into WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/GroupByWindow Sep 09, 2021 12:01:50 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-09T12:01:48.674Z: Fusing consumer WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/DropShardNum into WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnwritten Sep 09, 2021 12:01:50 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-09T12:01:49.102Z: Executing operation ExtractUserScore/Combine.perKey(SumInteger)/GroupByKey/Create Sep 09, 2021 12:01:50 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-09T12:01:49.167Z: Starting 1 workers in us-central1-c... Sep 09, 2021 12:01:50 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-09T12:01:49.496Z: Finished operation ExtractUserScore/Combine.perKey(SumInteger)/GroupByKey/Create Sep 09, 2021 12:01:50 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-09T12:01:49.706Z: Executing operation TextIO.Read/Read+ParseGameEvent+ExtractUserScore/MapElements/Map+ExtractUserScore/Combine.perKey(SumInteger)/GroupByKey+ExtractUserScore/Combine.perKey(SumInteger)/Combine.GroupedValues/Partial+ExtractUserScore/Combine.perKey(SumInteger)/GroupByKey/Reify+ExtractUserScore/Combine.perKey(SumInteger)/GroupByKey/Write Sep 09, 2021 12:02:08 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-09T12:02:07.919Z: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete Sep 09, 2021 12:02:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-09T12:02:25.677Z: Autoscaling: Raised the number of workers to 1 based on the rate of progress in the currently running stage(s). Sep 09, 2021 12:03:03 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-09T12:03:02.846Z: Workers have started successfully. Sep 09, 2021 12:03:03 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-09T12:03:02.884Z: Workers have started successfully. Sep 09, 2021 12:03:23 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-09T12:03:22.989Z: Finished operation TextIO.Read/Read+ParseGameEvent+ExtractUserScore/MapElements/Map+ExtractUserScore/Combine.perKey(SumInteger)/GroupByKey+ExtractUserScore/Combine.perKey(SumInteger)/Combine.GroupedValues/Partial+ExtractUserScore/Combine.perKey(SumInteger)/GroupByKey/Reify+ExtractUserScore/Combine.perKey(SumInteger)/GroupByKey/Write Sep 09, 2021 12:03:23 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-09T12:03:23.059Z: Executing operation ExtractUserScore/Combine.perKey(SumInteger)/GroupByKey/Close Sep 09, 2021 12:03:23 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-09T12:03:23.127Z: Finished operation ExtractUserScore/Combine.perKey(SumInteger)/GroupByKey/Close Sep 09, 2021 12:03:23 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-09T12:03:23.214Z: Executing operation WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Create Sep 09, 2021 12:03:25 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-09T12:03:24.499Z: Finished operation WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Create Sep 09, 2021 12:03:25 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-09T12:03:24.677Z: Executing operation ExtractUserScore/Combine.perKey(SumInteger)/GroupByKey/Read+ExtractUserScore/Combine.perKey(SumInteger)/Combine.GroupedValues+ExtractUserScore/Combine.perKey(SumInteger)/Combine.GroupedValues/Extract+WriteUserScoreSums/ConvertToRow+WriteUserScoreSums/TextIO.Write/WriteFiles/RewindowIntoGlobal/Window.Assign+WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles+WriteUserScoreSums/TextIO.Write/WriteFiles/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow)+WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Reify+WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Write Sep 09, 2021 12:03:30 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-09T12:03:28.371Z: Finished operation ExtractUserScore/Combine.perKey(SumInteger)/GroupByKey/Read+ExtractUserScore/Combine.perKey(SumInteger)/Combine.GroupedValues+ExtractUserScore/Combine.perKey(SumInteger)/Combine.GroupedValues/Extract+WriteUserScoreSums/ConvertToRow+WriteUserScoreSums/TextIO.Write/WriteFiles/RewindowIntoGlobal/Window.Assign+WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles+WriteUserScoreSums/TextIO.Write/WriteFiles/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow)+WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Reify+WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Write Sep 09, 2021 12:03:30 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-09T12:03:28.543Z: Executing operation WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Close Sep 09, 2021 12:03:30 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-09T12:03:28.641Z: Finished operation WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Close Sep 09, 2021 12:03:30 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-09T12:03:28.713Z: Executing operation WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Read+WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/GroupByWindow+WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnwritten+WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/DropShardNum+WriteUserScoreSums/TextIO.Write/WriteFiles/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow) Sep 09, 2021 12:03:33 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-09T12:03:31.720Z: Finished operation WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Read+WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/GroupByWindow+WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnwritten+WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/DropShardNum+WriteUserScoreSums/TextIO.Write/WriteFiles/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow) Sep 09, 2021 12:03:33 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-09T12:03:31.905Z: Executing operation WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/Flatten.PCollections/Unzipped-1 Sep 09, 2021 12:03:33 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-09T12:03:32.004Z: Finished operation WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/Flatten.PCollections/Unzipped-1 Sep 09, 2021 12:03:33 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-09T12:03:32.165Z: Executing operation WriteUserScoreSums/TextIO.Write/WriteFiles/GatherTempFileResults/View.AsList/CreateDataflowView Sep 09, 2021 12:03:33 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-09T12:03:32.222Z: Finished operation WriteUserScoreSums/TextIO.Write/WriteFiles/GatherTempFileResults/View.AsList/CreateDataflowView Sep 09, 2021 12:03:33 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-09T12:03:32.372Z: Executing operation WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Create Sep 09, 2021 12:03:33 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-09T12:03:32.562Z: Finished operation WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Create Sep 09, 2021 12:03:33 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-09T12:03:32.746Z: Executing operation WriteUserScoreSums/TextIO.Write/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Create.Values/Read(CreateSource)+WriteUserScoreSums/TextIO.Write/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Reify.ReifyView/ParDo(Anonymous)+WriteUserScoreSums/TextIO.Write/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Values/Values/Map+WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Finalize+WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair with random key+WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign+WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify+WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write Sep 09, 2021 12:03:36 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-09T12:03:35.860Z: Finished operation WriteUserScoreSums/TextIO.Write/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Create.Values/Read(CreateSource)+WriteUserScoreSums/TextIO.Write/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Reify.ReifyView/ParDo(Anonymous)+WriteUserScoreSums/TextIO.Write/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Values/Values/Map+WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Finalize+WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair with random key+WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign+WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify+WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write Sep 09, 2021 12:03:36 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-09T12:03:35.925Z: Executing operation WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Close Sep 09, 2021 12:03:36 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-09T12:03:35.979Z: Finished operation WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Close Sep 09, 2021 12:03:36 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-09T12:03:36.051Z: Executing operation WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read+WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow+WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable+WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Values/Values/Map Sep 09, 2021 12:03:36 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-09T12:03:36.333Z: Finished operation WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read+WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow+WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable+WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Values/Values/Map Sep 09, 2021 12:03:36 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-09T12:03:36.484Z: Cleaning up. Sep 09, 2021 12:03:38 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-09T12:03:36.590Z: Stopping worker pool... Sep 09, 2021 12:05:42 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-09T12:05:42.708Z: Autoscaling: Resized worker pool from 1 to 0. Sep 09, 2021 12:05:42 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-09T12:05:42.745Z: Worker pool stopped. Sep 09, 2021 12:06:18 PM org.apache.beam.runners.dataflow.DataflowPipelineJob logTerminalState INFO: Job 2021-09-09_05_01_09-10897150730585566451 finished with status DONE. gsutil cat gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/java-userscore-result-dataflow-runner.txt* | grep user19_BananaWallaby total_score: 231, user: user19_BananaWallaby Verified total_score: 231, user: user19_BananaWallaby ************************************** * SUCCESS: UserScore successfully run on DataflowRunner. ************************************** gsutil rm gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/java-userscore-result-dataflow-runner.txt* Removing gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/java-userscore-result-dataflow-runner.txt-00000-of-00001... / [1 objects] Operation completed over 1 objects. ************************************** * Test: Running: HourlyTeamScore example on DataflowRunner ************************************** mvn compile exec:java -q -Dmaven.wagon.http.retryHandler.class=default -Dmaven.wagon.http.retryHandler.count=5 -Dmaven.wagon.http.pool=false -Dmaven.wagon.httpconnectionManager.ttlSeconds=120 -Dhttp.keepAlive=false -Dexec.mainClass=org.apache.beam.examples.complete.game.HourlyTeamScore -Dexec.args="--tempLocation=gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp --runner=DataflowRunner --input=gs://dataflow-samples/game/5000_gaming_data.csv --project=apache-beam-testing --region=us-central1 --output=gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/java-hourlyteamscore-result-dataflow-runner.txt " -Pdataflow-runner Using maven /home/jenkins/tools/maven/apache-maven-3.5.4 Sep 09, 2021 12:06:25 PM org.apache.beam.runners.dataflow.options.DataflowPipelineOptions$StagingLocationFactory create INFO: No stagingLocation provided, falling back to gcpTempLocation Sep 09, 2021 12:06:26 PM org.apache.beam.runners.dataflow.DataflowRunner fromOptions INFO: PipelineOptions.filesToStage was not specified. Defaulting to files from the classpath: will stage 228 files. Enable logging at DEBUG level to see which files will be staged. Sep 09, 2021 12:06:26 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Executing pipeline on the Dataflow Service, which will have billing implications related to Google Compute Engine usage and other Google Cloud Services. Sep 09, 2021 12:06:28 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Staging portable pipeline proto to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/ Sep 09, 2021 12:06:28 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <235364 bytes, hash b0aea769493c0e3ef1ef0c2ab533ac2ad952c1c273e9f5c3b008f4b857bb9fed> to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/pipeline-sK6naUk8Dj7x7wwqtTOsKtlSwcJz6fXDsAj0uFe7n-0.pb Sep 09, 2021 12:06:30 PM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements INFO: Uploading 228 files from PipelineOptions.filesToStage to staging location to prepare for execution. Sep 09, 2021 12:06:30 PM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements INFO: Staging files complete: 228 files cached, 0 files newly uploaded in 0 seconds Sep 09, 2021 12:06:31 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding TextIO.Read/Read as step s1 Sep 09, 2021 12:06:31 PM org.apache.beam.sdk.io.FileBasedSource getEstimatedSizeBytes INFO: Filepattern gs://dataflow-samples/game/5000_gaming_data.csv matched 1 files with total size 386400 Sep 09, 2021 12:06:31 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding ParseGameEvent as step s2 Sep 09, 2021 12:06:31 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding FilterStartTime/ParDo(Anonymous) as step s3 Sep 09, 2021 12:06:31 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding FilterEndTime/ParDo(Anonymous) as step s4 Sep 09, 2021 12:06:31 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding AddEventTimestamps/AddTimestamps as step s5 Sep 09, 2021 12:06:31 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding FixedWindowsTeam/Window.Assign as step s6 Sep 09, 2021 12:06:31 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding ExtractTeamScore/MapElements/Map as step s7 Sep 09, 2021 12:06:31 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding ExtractTeamScore/Combine.perKey(SumInteger)/GroupByKey as step s8 Sep 09, 2021 12:06:31 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding ExtractTeamScore/Combine.perKey(SumInteger)/Combine.GroupedValues as step s9 Sep 09, 2021 12:06:31 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteTeamScoreSums/ConvertToRow as step s10 Sep 09, 2021 12:06:31 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteShardedBundlesToTempFiles/ApplyShardingKey as step s11 Sep 09, 2021 12:06:31 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteShardedBundlesToTempFiles/GroupIntoShards as step s12 Sep 09, 2021 12:06:31 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteShardedBundlesToTempFiles/WriteShardsIntoTempFiles as step s13 Sep 09, 2021 12:06:31 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Add void key/AddKeys/Map as step s14 Sep 09, 2021 12:06:31 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/Window.Into()/Window.Assign as step s15 Sep 09, 2021 12:06:31 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey as step s16 Sep 09, 2021 12:06:31 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/ExpandIterable as step s17 Sep 09, 2021 12:06:31 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Drop key/Values/Map as step s18 Sep 09, 2021 12:06:31 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Gather bundles as step s19 Sep 09, 2021 12:06:31 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Pair with random key as step s20 Sep 09, 2021 12:06:31 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign as step s21 Sep 09, 2021 12:06:31 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey as step s22 Sep 09, 2021 12:06:31 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable as step s23 Sep 09, 2021 12:06:31 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Values/Values/Map as step s24 Sep 09, 2021 12:06:31 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Finalize as step s25 Sep 09, 2021 12:06:31 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair with random key as step s26 Sep 09, 2021 12:06:31 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign as step s27 Sep 09, 2021 12:06:31 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey as step s28 Sep 09, 2021 12:06:31 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable as step s29 Sep 09, 2021 12:06:31 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Values/Values/Map as step s30 Sep 09, 2021 12:06:31 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Dataflow SDK version: 2.34.0-SNAPSHOT Sep 09, 2021 12:06:32 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2021-09-09_05_06_31-3524899216147049663?project=apache-beam-testing Sep 09, 2021 12:06:32 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Submitted job: 2021-09-09_05_06_31-3524899216147049663 Sep 09, 2021 12:06:32 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: To cancel the job using the 'gcloud' tool, run: > gcloud dataflow jobs --project=apache-beam-testing cancel > --region=us-central1 2021-09-09_05_06_31-3524899216147049663 Sep 09, 2021 12:06:37 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-09T12:06:35.161Z: Autoscaling is enabled for job 2021-09-09_05_06_31-3524899216147049663. The number of workers will be between 1 and 1000. Sep 09, 2021 12:06:37 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-09T12:06:35.286Z: Autoscaling was automatically enabled for job 2021-09-09_05_06_31-3524899216147049663. Sep 09, 2021 12:06:51 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-09T12:06:50.309Z: Worker configuration: e2-standard-2 in us-central1-c. Sep 09, 2021 12:06:51 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process SEVERE: 2021-09-09T12:06:51.119Z: Workflow failed. Causes: Project apache-beam-testing has insufficient quota(s) to execute this workflow with 1 instances in region us-central1. Quota summary (required/available): 1/19451 instances, 2/0 CPUs, 25/260276 disk GB, 0/2397 SSD disk GB, 1/222 instance groups, 1/226 managed instance groups, 1/447 instance templates, 1/680 in-use IP addresses. Please see https://cloud.google.com/compute/docs/resource-quotas about requesting more quota. Sep 09, 2021 12:06:51 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-09T12:06:51.169Z: Cleaning up. Sep 09, 2021 12:06:51 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-09T12:06:51.238Z: Worker pool stopped. Sep 09, 2021 12:06:52 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-09T12:06:52.554Z: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete Sep 09, 2021 12:06:56 PM org.apache.beam.runners.dataflow.DataflowPipelineJob logTerminalState INFO: Job 2021-09-09_05_06_31-3524899216147049663 failed with status FAILED. gsutil cat gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/java-hourlyteamscore-result-dataflow-runner.txt* | grep AzureBilby CommandException: No URLs matched: gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/java-hourlyteamscore-result-dataflow-runner.txt* CommandException: No URLs matched: gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/java-hourlyteamscore-result-dataflow-runner.txt* [ERROR] Failed command > Task :runners:google-cloud-dataflow-java:runMobileGamingJavaDataflow FAILED FAILURE: Build completed with 2 failures. 1: Task failed with an exception. ----------- * What went wrong: Execution failed for task ':runners:direct-java:runMobileGamingJavaDirect'. > Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with > non-zero exit value 1 * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. ============================================================================== 2: Task failed with an exception. ----------- * What went wrong: Execution failed for task ':runners:google-cloud-dataflow-java:runMobileGamingJavaDataflow'. > Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with > non-zero exit value 1 * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. ============================================================================== * Get more help at https://help.gradle.org Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/6.8.3/userguide/command_line_interface.html#sec:command_line_warnings BUILD FAILED in 1h 6m 38s 9 actionable tasks: 8 executed, 1 from cache Publishing build scan... https://gradle.com/s/j6xgujcv5qvde Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
