See <https://ci-beam.apache.org/job/beam_PostRelease_NightlySnapshot/1198/display/redirect?page=changes>
Changes: [dennisylyung] [BEAM-11331] Change region flag to awsRegion in io.aws2 [zhaohaizhou940527] [BEAM-11365] Direct Runner: Handle pipeline option string quotes under [ajamato] Add start_times to MonitoringInfos and populate them in the python SDK [piotr.szuberski] [BEAM-11375] Downgrade Kafka clients dependency to fix Spark postcommit [ludav] [BEAM-10911] Grafana post-commit status board: job_name cells link to [noreply] [BEAM-9980] use ALL_SUPPORTED_VERSIONS for constant [noreply] [BEAM-11376] Ignoring DicomIOReadIT (#13450) [noreply] [BEAM-10986] Update shadow jar plugin. (#13453) ------------------------------------------ [...truncated 3.84 MB...] Dec 03, 2020 11:07:19 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/Flatten.PCollections as step s12 Dec 03, 2020 11:07:19 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow) as step s13 Dec 03, 2020 11:07:19 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/GatherTempFileResults/View.AsList/CreateDataflowView as step s14 Dec 03, 2020 11:07:19 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Create.Values/Read(CreateSource) as step s15 Dec 03, 2020 11:07:19 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Reify.ReifyView/ParDo(Anonymous) as step s16 Dec 03, 2020 11:07:19 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Values/Values/Map as step s17 Dec 03, 2020 11:07:19 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/FinalizeTempFileBundles/Finalize as step s18 Dec 03, 2020 11:07:19 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair with random key as step s19 Dec 03, 2020 11:07:19 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign as step s20 Dec 03, 2020 11:07:19 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey as step s21 Dec 03, 2020 11:07:19 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable as step s22 Dec 03, 2020 11:07:19 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Values/Values/Map as step s23 Dec 03, 2020 11:07:19 AM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Staging pipeline description to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/ > Task :runners:twister2:runQuickstartJavaTwister2 SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/tmp/groovy-generated-530931720714449936-tmpdir/.m2/repository/org/slf4j/slf4j-jdk14/1.7.30/slf4j-jdk14-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/tmp/groovy-generated-530931720714449936-tmpdir/.m2/repository/org/slf4j/slf4j-log4j12/1.7.25/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.JDK14LoggerFactory] Dec 03, 2020 11:07:20 AM org.apache.beam.runners.twister2.Twister2Runner fromOptions INFO: PipelineOptions.filesToStage was not specified. Defaulting to files from the classpath: will stage {} files. Enable logging at DEBUG level to see which files will be staged377 Dec 03, 2020 11:07:21 AM org.apache.beam.runners.twister2.Twister2Runner run INFO: Translating pipeline to Twister2 program. Dec 03, 2020 11:07:34 AM org.apache.beam.runners.twister2.Twister2Runner run WARNING: Twister2 Local Mode currently only supports single worker Dec 03, 2020 11:07:34 AM edu.iu.dsc.tws.rsched.core.ResourceAllocator loadConfig INFO: Loaded configuration with twister2_home: /tmp and configuration: /tmp/conf/ and cluster: standalone grep Foundation counts* Foundation: 1 Verified Foundation: 1 [SUCCESS] > Task :runners:google-cloud-dataflow-java:runQuickstartJavaDataflow Dec 03, 2020 11:07:19 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <143465 bytes, hash f8941dc72a66ad096dae00d0799a2b8a9e942b5d580f25c26ad6c46dcf28071b> to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/pipeline--JQdxypmrQltrgDQeZorip6UK11YDyXCatbEbc8oBxs.pb Dec 03, 2020 11:07:19 AM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Dataflow SDK version: 2.27.0-SNAPSHOT Dec 03, 2020 11:07:21 AM org.apache.beam.runners.dataflow.DataflowRunner run INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-12-03_03_07_19-9099186933947465056?project=apache-beam-testing Dec 03, 2020 11:07:21 AM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Submitted job: 2020-12-03_03_07_19-9099186933947465056 Dec 03, 2020 11:07:21 AM org.apache.beam.runners.dataflow.DataflowRunner run INFO: To cancel the job using the 'gcloud' tool, run: > gcloud dataflow jobs --project=apache-beam-testing cancel > --region=us-central1 2020-12-03_03_07_19-9099186933947465056 Dec 03, 2020 11:07:22 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-12-03T11:07:19.815Z: Autoscaling was automatically enabled for job 2020-12-03_03_07_19-9099186933947465056. Dec 03, 2020 11:07:22 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-12-03T11:07:19.815Z: Autoscaling is enabled for job 2020-12-03_03_07_19-9099186933947465056. The number of workers will be between 1 and 1000. Dec 03, 2020 11:07:28 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-12-03T11:07:28.446Z: Worker configuration: n1-standard-1 in us-central1-f. Dec 03, 2020 11:07:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-12-03T11:07:29.304Z: Expanding CoGroupByKey operations into optimizable parts. Dec 03, 2020 11:07:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-12-03T11:07:29.519Z: Expanding GroupByKey operations into optimizable parts. Dec 03, 2020 11:07:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-12-03T11:07:29.560Z: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns Dec 03, 2020 11:07:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-12-03T11:07:29.800Z: Fusing adjacent ParDo, Read, Write, and Flatten operations Dec 03, 2020 11:07:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-12-03T11:07:29.846Z: Fusing consumer WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Reify.ReifyView/ParDo(Anonymous) into WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Create.Values/Read(CreateSource) Dec 03, 2020 11:07:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-12-03T11:07:29.892Z: Fusing consumer WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Values/Values/Map into WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Reify.ReifyView/ParDo(Anonymous) Dec 03, 2020 11:07:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-12-03T11:07:29.941Z: Fusing consumer WriteCounts/WriteFiles/FinalizeTempFileBundles/Finalize into WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Values/Values/Map Dec 03, 2020 11:07:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-12-03T11:07:29.990Z: Fusing consumer WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair with random key into WriteCounts/WriteFiles/FinalizeTempFileBundles/Finalize Dec 03, 2020 11:07:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-12-03T11:07:30.038Z: Fusing consumer WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign into WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair with random key Dec 03, 2020 11:07:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-12-03T11:07:30.071Z: Fusing consumer WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify into WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign Dec 03, 2020 11:07:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-12-03T11:07:30.099Z: Fusing consumer WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write into WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify Dec 03, 2020 11:07:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-12-03T11:07:30.137Z: Fusing consumer WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow into WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read Dec 03, 2020 11:07:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-12-03T11:07:30.172Z: Fusing consumer WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable into WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow Dec 03, 2020 11:07:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-12-03T11:07:30.207Z: Fusing consumer WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Values/Values/Map into WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable Dec 03, 2020 11:07:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-12-03T11:07:30.262Z: Unzipping flatten s12 for input s11.org.apache.beam.sdk.values.PCollection.<init>:402#5f2ef1f005ae0b4 Dec 03, 2020 11:07:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-12-03T11:07:30.295Z: Fusing unzipped copy of WriteCounts/WriteFiles/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow), through flatten WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/Flatten.PCollections, into producer WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/DropShardNum Dec 03, 2020 11:07:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-12-03T11:07:30.362Z: Fusing consumer WriteCounts/WriteFiles/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow) into WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles Dec 03, 2020 11:07:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-12-03T11:07:30.395Z: Fusing consumer WordCount.CountWords/ParDo(ExtractWords) into ReadLines/Read Dec 03, 2020 11:07:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-12-03T11:07:30.418Z: Fusing consumer WordCount.CountWords/Count.PerElement/Init/Map into WordCount.CountWords/ParDo(ExtractWords) Dec 03, 2020 11:07:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-12-03T11:07:30.467Z: Fusing consumer WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Partial into WordCount.CountWords/Count.PerElement/Init/Map Dec 03, 2020 11:07:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-12-03T11:07:30.503Z: Fusing consumer WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Reify into WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Partial Dec 03, 2020 11:07:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-12-03T11:07:30.532Z: Fusing consumer WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Write into WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Reify Dec 03, 2020 11:07:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-12-03T11:07:30.558Z: Fusing consumer WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues into WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Read Dec 03, 2020 11:07:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-12-03T11:07:30.582Z: Fusing consumer WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Extract into WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues Dec 03, 2020 11:07:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-12-03T11:07:30.621Z: Fusing consumer MapElements/Map into WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Extract Dec 03, 2020 11:07:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-12-03T11:07:30.650Z: Fusing consumer WriteCounts/WriteFiles/RewindowIntoGlobal/Window.Assign into MapElements/Map Dec 03, 2020 11:07:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-12-03T11:07:30.838Z: Fusing consumer WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles into WriteCounts/WriteFiles/RewindowIntoGlobal/Window.Assign Dec 03, 2020 11:07:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-12-03T11:07:30.886Z: Fusing consumer WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Reify into WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles Dec 03, 2020 11:07:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-12-03T11:07:30.926Z: Fusing consumer WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Write into WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Reify Dec 03, 2020 11:07:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-12-03T11:07:30.957Z: Fusing consumer WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/GroupByWindow into WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Read Dec 03, 2020 11:07:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-12-03T11:07:30.993Z: Fusing consumer WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnwritten into WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/GroupByWindow Dec 03, 2020 11:07:33 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-12-03T11:07:31.026Z: Fusing consumer WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/DropShardNum into WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnwritten Dec 03, 2020 11:07:33 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-12-03T11:07:31.447Z: Executing operation WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Create Dec 03, 2020 11:07:33 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-12-03T11:07:31.543Z: Executing operation WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Create Dec 03, 2020 11:07:33 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-12-03T11:07:31.616Z: Executing operation WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Create Dec 03, 2020 11:07:33 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-12-03T11:07:31.623Z: Starting 1 workers in us-central1-f... Dec 03, 2020 11:07:33 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-12-03T11:07:31.688Z: Finished operation WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Create Dec 03, 2020 11:07:33 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-12-03T11:07:31.703Z: Finished operation WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Create Dec 03, 2020 11:07:33 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-12-03T11:07:31.703Z: Finished operation WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Create Dec 03, 2020 11:07:33 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-12-03T11:07:32.303Z: Executing operation ReadLines/Read+WordCount.CountWords/ParDo(ExtractWords)+WordCount.CountWords/Count.PerElement/Init/Map+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Partial+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Reify+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Write Dec 03, 2020 11:07:57 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-12-03T11:07:57.321Z: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete Dec 03, 2020 11:08:13 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-12-03T11:08:12.011Z: Autoscaling: Raised the number of workers to 1 based on the rate of progress in the currently running stage(s). Dec 03, 2020 11:08:36 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-12-03T11:08:36.561Z: Workers have started successfully. Dec 03, 2020 11:08:36 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-12-03T11:08:36.602Z: Workers have started successfully. Dec 03, 2020 11:10:28 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-12-03T11:10:27.281Z: Finished operation ReadLines/Read+WordCount.CountWords/ParDo(ExtractWords)+WordCount.CountWords/Count.PerElement/Init/Map+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Partial+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Reify+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Write Dec 03, 2020 11:10:28 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-12-03T11:10:27.349Z: Executing operation WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Close Dec 03, 2020 11:10:28 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-12-03T11:10:27.438Z: Finished operation WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Close Dec 03, 2020 11:10:28 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-12-03T11:10:27.497Z: Executing operation WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Read+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Extract+MapElements/Map+WriteCounts/WriteFiles/RewindowIntoGlobal/Window.Assign+WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles+WriteCounts/WriteFiles/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow)+WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Reify+WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Write Dec 03, 2020 11:10:36 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-12-03T11:10:34.564Z: Finished operation WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Read+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Extract+MapElements/Map+WriteCounts/WriteFiles/RewindowIntoGlobal/Window.Assign+WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles+WriteCounts/WriteFiles/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow)+WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Reify+WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Write Dec 03, 2020 11:10:36 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-12-03T11:10:34.686Z: Executing operation WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Close Dec 03, 2020 11:10:36 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-12-03T11:10:34.746Z: Finished operation WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Close Dec 03, 2020 11:10:36 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-12-03T11:10:34.817Z: Executing operation WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Read+WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/GroupByWindow+WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnwritten+WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/DropShardNum+WriteCounts/WriteFiles/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow) Dec 03, 2020 11:10:38 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-12-03T11:10:37.147Z: Finished operation WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Read+WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/GroupByWindow+WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnwritten+WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/DropShardNum+WriteCounts/WriteFiles/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow) Dec 03, 2020 11:10:38 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-12-03T11:10:37.297Z: Executing operation WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/Flatten.PCollections/Unzipped-1 Dec 03, 2020 11:10:38 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-12-03T11:10:37.366Z: Finished operation WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/Flatten.PCollections/Unzipped-1 Dec 03, 2020 11:10:38 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-12-03T11:10:37.503Z: Executing operation WriteCounts/WriteFiles/GatherTempFileResults/View.AsList/CreateDataflowView Dec 03, 2020 11:10:38 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-12-03T11:10:37.561Z: Finished operation WriteCounts/WriteFiles/GatherTempFileResults/View.AsList/CreateDataflowView Dec 03, 2020 11:10:38 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-12-03T11:10:37.718Z: Executing operation WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Create.Values/Read(CreateSource)+WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Reify.ReifyView/ParDo(Anonymous)+WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Values/Values/Map+WriteCounts/WriteFiles/FinalizeTempFileBundles/Finalize+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair with random key+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write Dec 03, 2020 11:10:41 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-12-03T11:10:40.978Z: Finished operation WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Create.Values/Read(CreateSource)+WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Reify.ReifyView/ParDo(Anonymous)+WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Values/Values/Map+WriteCounts/WriteFiles/FinalizeTempFileBundles/Finalize+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair with random key+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write Dec 03, 2020 11:10:41 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-12-03T11:10:41.046Z: Executing operation WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Close Dec 03, 2020 11:10:41 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-12-03T11:10:41.087Z: Finished operation WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Close Dec 03, 2020 11:10:41 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-12-03T11:10:41.149Z: Executing operation WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Values/Values/Map Dec 03, 2020 11:10:45 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-12-03T11:10:42.432Z: Finished operation WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Values/Values/Map Dec 03, 2020 11:10:45 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-12-03T11:10:42.591Z: Cleaning up. Dec 03, 2020 11:10:45 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-12-03T11:10:42.658Z: Stopping worker pool... Dec 03, 2020 11:11:33 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-12-03T11:11:33.010Z: Autoscaling: Resized worker pool from 1 to 0. Dec 03, 2020 11:11:33 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-12-03T11:11:33.064Z: Worker pool stopped. Dec 03, 2020 11:11:45 AM org.apache.beam.runners.dataflow.DataflowPipelineJob logTerminalState INFO: Job 2020-12-03_03_07_19-9099186933947465056 finished with status DONE. gsutil cat gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/count* | grep Montague: Montague: 47 Verified Montague: 47 gsutil rm gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/count* Removing gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/counts-00000-of-00003... / [1 objects] Removing gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/counts-00001-of-00003... / [2 objects] Removing gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/counts-00002-of-00003... / [3 objects] Operation completed over 3 objects. [SUCCESS] FAILURE: Build completed with 2 failures. 1: Task failed with an exception. ----------- * What went wrong: Execution failed for task ':runners:google-cloud-dataflow-java:runMobileGamingJavaDataflow'. > Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with > non-zero exit value 1 * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. ============================================================================== 2: Task failed with an exception. ----------- * What went wrong: Execution failed for task ':runners:direct-java:runMobileGamingJavaDirect'. > Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with > non-zero exit value 1 * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. ============================================================================== * Get more help at https://help.gradle.org Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/6.7/userguide/command_line_interface.html#sec:command_line_warnings BUILD FAILED in 11m 29s 8 actionable tasks: 7 executed, 1 from cache Publishing build scan... https://gradle.com/s/orog3572aobkw Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
