See <https://builds.apache.org/job/beam_PostRelease_NightlySnapshot/943/display/redirect?page=changes>
Changes: [kcweaver] [BEAM-9509] Improve error message for bad job server URL. [mxm] [BEAM-8201] Cleanup FnServices from DockerEnvironmentFactory and [iemejia] [website] Update information about Beam's LTS policy ------------------------------------------ [...truncated 3.52 MB...] Apr 06, 2020 11:24:51 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/GatherTempFileResults/ToList/Combine.perKey(ToList)/Combine.GroupedValues as step s15 Apr 06, 2020 11:24:52 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/GatherTempFileResults/ToList/Values/Values/Map as step s16 Apr 06, 2020 11:24:52 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/GatherTempFileResults/ToList/View.AsIterable/ParDo(ToIsmRecordForGlobalWindow) as step s17 Apr 06, 2020 11:24:52 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/GatherTempFileResults/ToList/View.AsIterable/CreateDataflowView as step s18 Apr 06, 2020 11:24:52 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/GatherTempFileResults/ToList/CreateVoid/Read(CreateSource) as step s19 Apr 06, 2020 11:24:52 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/GatherTempFileResults/ToList/ProduceDefault as step s20 Apr 06, 2020 11:24:52 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/GatherTempFileResults/ToList/Flatten.PCollections as step s21 Apr 06, 2020 11:24:52 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/FinalizeTempFileBundles/Finalize as step s22 Apr 06, 2020 11:24:52 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair with random key as step s23 Apr 06, 2020 11:24:52 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign as step s24 Apr 06, 2020 11:24:52 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey as step s25 Apr 06, 2020 11:24:52 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable as step s26 Apr 06, 2020 11:24:52 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Values/Values/Map as step s27 Apr 06, 2020 11:24:52 AM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Staging pipeline description to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/ Apr 06, 2020 11:24:52 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <74760 bytes, hash BhH-yR93lUJqAvfTitH05A> to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/pipeline-BhH-yR93lUJqAvfTitH05A.pb Apr 06, 2020 11:24:52 AM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Dataflow SDK version: 2.21.0-SNAPSHOT Apr 06, 2020 11:24:53 AM org.apache.beam.runners.dataflow.options.DataflowPipelineOptions$DefaultGcpRegionFactory create WARNING: Region will default to us-central1. Future releases of Beam will require the user to set the region explicitly. https://cloud.google.com/compute/docs/regions-zones/regions-zones Apr 06, 2020 11:24:55 AM org.apache.beam.runners.dataflow.DataflowRunner run INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-06_04_24_53-4107355629350440775?project=apache-beam-testing Apr 06, 2020 11:24:55 AM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Submitted job: 2020-04-06_04_24_53-4107355629350440775 Apr 06, 2020 11:24:55 AM org.apache.beam.runners.dataflow.DataflowRunner run INFO: To cancel the job using the 'gcloud' tool, run: > gcloud dataflow jobs --project=apache-beam-testing cancel > --region=us-central1 2020-04-06_04_24_53-4107355629350440775 Apr 06, 2020 11:24:55 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:24:53.746Z: Autoscaling was automatically enabled for job 2020-04-06_04_24_53-4107355629350440775. Apr 06, 2020 11:24:55 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:24:53.746Z: Autoscaling is enabled for job 2020-04-06_04_24_53-4107355629350440775. The number of workers will be between 1 and 1000. Apr 06, 2020 11:24:58 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:24:57.434Z: Checking permissions granted to controller Service Account. Apr 06, 2020 11:25:04 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:25:03.367Z: Worker configuration: n1-standard-1 in us-central1-a. Apr 06, 2020 11:25:04 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:25:03.975Z: Expanding CoGroupByKey operations into optimizable parts. Apr 06, 2020 11:25:04 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:25:04.136Z: Expanding GroupByKey operations into optimizable parts. Apr 06, 2020 11:25:04 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:25:04.160Z: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns Apr 06, 2020 11:25:04 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:25:04.383Z: Fusing adjacent ParDo, Read, Write, and Flatten operations Apr 06, 2020 11:25:04 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:25:04.422Z: Fusing consumer WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair with random key into WriteCounts/WriteFiles/FinalizeTempFileBundles/Finalize Apr 06, 2020 11:25:04 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:25:04.447Z: Fusing consumer WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign into WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair with random key Apr 06, 2020 11:25:04 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:25:04.480Z: Fusing consumer WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify into WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign Apr 06, 2020 11:25:04 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:25:04.507Z: Fusing consumer WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write into WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify Apr 06, 2020 11:25:04 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:25:04.529Z: Fusing consumer WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow into WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read Apr 06, 2020 11:25:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:25:04.565Z: Fusing consumer WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable into WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow Apr 06, 2020 11:25:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:25:04.590Z: Fusing consumer WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Values/Values/Map into WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable Apr 06, 2020 11:25:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:25:04.622Z: Unzipping flatten s21 for input s16.org.apache.beam.sdk.values.PCollection.<init>:400#1abfe6cb40bcf2e7 Apr 06, 2020 11:25:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:25:04.660Z: Fusing unzipped copy of WriteCounts/WriteFiles/FinalizeTempFileBundles/Finalize, through flatten WriteCounts/WriteFiles/GatherTempFileResults/ToList/Flatten.PCollections, into producer WriteCounts/WriteFiles/GatherTempFileResults/ToList/Values/Values/Map Apr 06, 2020 11:25:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:25:04.699Z: Fusing consumer WriteCounts/WriteFiles/FinalizeTempFileBundles/Finalize into WriteCounts/WriteFiles/GatherTempFileResults/ToList/ProduceDefault Apr 06, 2020 11:25:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:25:04.734Z: Fusing consumer WriteCounts/WriteFiles/GatherTempFileResults/ToList/ProduceDefault into WriteCounts/WriteFiles/GatherTempFileResults/ToList/CreateVoid/Read(CreateSource) Apr 06, 2020 11:25:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:25:04.769Z: Fusing consumer WriteCounts/WriteFiles/GatherTempFileResults/ToList/View.AsIterable/ParDo(ToIsmRecordForGlobalWindow) into WriteCounts/WriteFiles/GatherTempFileResults/ToList/Values/Values/Map Apr 06, 2020 11:25:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:25:04.805Z: Unzipping flatten s12 for input s11.org.apache.beam.sdk.values.PCollection.<init>:400#20ff67585e33a8f6 Apr 06, 2020 11:25:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:25:04.843Z: Fusing unzipped copy of WriteCounts/WriteFiles/GatherTempFileResults/ToList/WithKeys/AddKeys/Map, through flatten WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/Flatten.PCollections, into producer WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/DropShardNum Apr 06, 2020 11:25:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:25:04.876Z: Fusing consumer WriteCounts/WriteFiles/GatherTempFileResults/ToList/Combine.perKey(ToList)/GroupByKey/Reify into WriteCounts/WriteFiles/GatherTempFileResults/ToList/Combine.perKey(ToList)/GroupByKey+WriteCounts/WriteFiles/GatherTempFileResults/ToList/Combine.perKey(ToList)/Combine.GroupedValues/Partial Apr 06, 2020 11:25:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:25:04.910Z: Fusing consumer WriteCounts/WriteFiles/GatherTempFileResults/ToList/Combine.perKey(ToList)/GroupByKey/Write into WriteCounts/WriteFiles/GatherTempFileResults/ToList/Combine.perKey(ToList)/GroupByKey/Reify Apr 06, 2020 11:25:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:25:04.950Z: Fusing consumer WriteCounts/WriteFiles/GatherTempFileResults/ToList/Combine.perKey(ToList)/Combine.GroupedValues into WriteCounts/WriteFiles/GatherTempFileResults/ToList/Combine.perKey(ToList)/GroupByKey/Read Apr 06, 2020 11:25:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:25:04.989Z: Fusing consumer WriteCounts/WriteFiles/GatherTempFileResults/ToList/Combine.perKey(ToList)/Combine.GroupedValues/Extract into WriteCounts/WriteFiles/GatherTempFileResults/ToList/Combine.perKey(ToList)/Combine.GroupedValues Apr 06, 2020 11:25:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:25:05.023Z: Fusing consumer WriteCounts/WriteFiles/GatherTempFileResults/ToList/Values/Values/Map into WriteCounts/WriteFiles/GatherTempFileResults/ToList/Combine.perKey(ToList)/Combine.GroupedValues/Extract Apr 06, 2020 11:25:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:25:05.058Z: Unzipping flatten s12-u68 for input s13.org.apache.beam.sdk.values.PCollection.<init>:400#5f2ef1f005ae0b4-c66 Apr 06, 2020 11:25:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:25:05.094Z: Fusing unzipped copy of WriteCounts/WriteFiles/GatherTempFileResults/ToList/Combine.perKey(ToList)/GroupByKey+WriteCounts/WriteFiles/GatherTempFileResults/ToList/Combine.perKey(ToList)/Combine.GroupedValues/Partial, through flatten WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/Flatten.PCollections/Unzipped-1, into producer WriteCounts/WriteFiles/GatherTempFileResults/ToList/WithKeys/AddKeys/Map Apr 06, 2020 11:25:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:25:05.120Z: Fusing consumer WriteCounts/WriteFiles/GatherTempFileResults/ToList/WithKeys/AddKeys/Map into WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles Apr 06, 2020 11:25:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:25:05.143Z: Fusing consumer WriteCounts/WriteFiles/GatherTempFileResults/ToList/Combine.perKey(ToList)/GroupByKey+WriteCounts/WriteFiles/GatherTempFileResults/ToList/Combine.perKey(ToList)/Combine.GroupedValues/Partial into WriteCounts/WriteFiles/GatherTempFileResults/ToList/WithKeys/AddKeys/Map Apr 06, 2020 11:25:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:25:05.165Z: Fusing consumer WordCount.CountWords/ParDo(ExtractWords) into ReadLines/Read Apr 06, 2020 11:25:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:25:05.199Z: Fusing consumer WordCount.CountWords/Count.PerElement/Init/Map into WordCount.CountWords/ParDo(ExtractWords) Apr 06, 2020 11:25:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:25:05.231Z: Fusing consumer WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Partial into WordCount.CountWords/Count.PerElement/Init/Map Apr 06, 2020 11:25:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:25:05.277Z: Fusing consumer WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Reify into WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Partial Apr 06, 2020 11:25:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:25:05.313Z: Fusing consumer WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Write into WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Reify Apr 06, 2020 11:25:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:25:05.349Z: Fusing consumer WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues into WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Read Apr 06, 2020 11:25:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:25:05.384Z: Fusing consumer WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Extract into WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues Apr 06, 2020 11:25:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:25:05.420Z: Fusing consumer MapElements/Map into WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Extract Apr 06, 2020 11:25:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:25:05.455Z: Fusing consumer WriteCounts/WriteFiles/RewindowIntoGlobal/Window.Assign into MapElements/Map Apr 06, 2020 11:25:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:25:05.488Z: Fusing consumer WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles into WriteCounts/WriteFiles/RewindowIntoGlobal/Window.Assign Apr 06, 2020 11:25:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:25:05.525Z: Fusing consumer WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Reify into WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles Apr 06, 2020 11:25:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:25:05.563Z: Fusing consumer WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Write into WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Reify Apr 06, 2020 11:25:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:25:05.596Z: Fusing consumer WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/GroupByWindow into WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Read Apr 06, 2020 11:25:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:25:05.632Z: Fusing consumer WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnwritten into WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/GroupByWindow Apr 06, 2020 11:25:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:25:05.668Z: Fusing consumer WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/DropShardNum into WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnwritten Apr 06, 2020 11:25:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:25:06.046Z: Executing operation WriteCounts/WriteFiles/GatherTempFileResults/ToList/Combine.perKey(ToList)/GroupByKey/Create Apr 06, 2020 11:25:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:25:06.076Z: Executing operation WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Create Apr 06, 2020 11:25:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:25:06.108Z: Executing operation WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Create Apr 06, 2020 11:25:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:25:06.118Z: Starting 1 workers in us-central1-a... Apr 06, 2020 11:25:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:25:06.141Z: Executing operation WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Create Apr 06, 2020 11:25:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:25:06.163Z: Finished operation WriteCounts/WriteFiles/GatherTempFileResults/ToList/Combine.perKey(ToList)/GroupByKey/Create Apr 06, 2020 11:25:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:25:06.180Z: Finished operation WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Create Apr 06, 2020 11:25:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:25:06.180Z: Finished operation WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Create Apr 06, 2020 11:25:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:25:06.200Z: Finished operation WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Create Apr 06, 2020 11:25:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:25:06.343Z: Executing operation ReadLines/Read+WordCount.CountWords/ParDo(ExtractWords)+WordCount.CountWords/Count.PerElement/Init/Map+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Partial+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Reify+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Write Apr 06, 2020 11:25:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process WARNING: 2020-04-06T11:25:30.212Z: Your project already contains 100 Dataflow-created metric descriptors and Stackdriver will not create new Dataflow custom metrics for this job. Each unique user-defined metric name (independent of the DoFn in which it is defined) produces a new metric descriptor. To delete old / unused metric descriptors see https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete Apr 06, 2020 11:25:33 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:25:32.185Z: Autoscaling: Raised the number of workers to 1 based on the rate of progress in the currently running stage(s). Apr 06, 2020 11:26:02 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:26:00.914Z: Workers have started successfully. Apr 06, 2020 11:26:02 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:26:00.945Z: Workers have started successfully. Apr 06, 2020 11:27:14 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:27:14.011Z: Finished operation ReadLines/Read+WordCount.CountWords/ParDo(ExtractWords)+WordCount.CountWords/Count.PerElement/Init/Map+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Partial+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Reify+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Write Apr 06, 2020 11:27:14 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:27:14.067Z: Executing operation WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Close Apr 06, 2020 11:27:14 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:27:14.123Z: Finished operation WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Close Apr 06, 2020 11:27:14 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:27:14.188Z: Executing operation WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Read+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Extract+MapElements/Map+WriteCounts/WriteFiles/RewindowIntoGlobal/Window.Assign+WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles+WriteCounts/WriteFiles/GatherTempFileResults/ToList/WithKeys/AddKeys/Map+WriteCounts/WriteFiles/GatherTempFileResults/ToList/Combine.perKey(ToList)/GroupByKey+WriteCounts/WriteFiles/GatherTempFileResults/ToList/Combine.perKey(ToList)/Combine.GroupedValues/Partial+WriteCounts/WriteFiles/GatherTempFileResults/ToList/Combine.perKey(ToList)/GroupByKey/Reify+WriteCounts/WriteFiles/GatherTempFileResults/ToList/Combine.perKey(ToList)/GroupByKey/Write+WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Reify+WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Write Apr 06, 2020 11:27:20 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:27:18.998Z: Finished operation WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Read+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Extract+MapElements/Map+WriteCounts/WriteFiles/RewindowIntoGlobal/Window.Assign+WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles+WriteCounts/WriteFiles/GatherTempFileResults/ToList/WithKeys/AddKeys/Map+WriteCounts/WriteFiles/GatherTempFileResults/ToList/Combine.perKey(ToList)/GroupByKey+WriteCounts/WriteFiles/GatherTempFileResults/ToList/Combine.perKey(ToList)/Combine.GroupedValues/Partial+WriteCounts/WriteFiles/GatherTempFileResults/ToList/Combine.perKey(ToList)/GroupByKey/Reify+WriteCounts/WriteFiles/GatherTempFileResults/ToList/Combine.perKey(ToList)/GroupByKey/Write+WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Reify+WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Write Apr 06, 2020 11:27:20 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:27:19.069Z: Executing operation WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Close Apr 06, 2020 11:27:20 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:27:19.129Z: Finished operation WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Close Apr 06, 2020 11:27:20 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:27:19.206Z: Executing operation WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Read+WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/GroupByWindow+WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnwritten+WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/DropShardNum+WriteCounts/WriteFiles/GatherTempFileResults/ToList/WithKeys/AddKeys/Map+WriteCounts/WriteFiles/GatherTempFileResults/ToList/Combine.perKey(ToList)/GroupByKey+WriteCounts/WriteFiles/GatherTempFileResults/ToList/Combine.perKey(ToList)/Combine.GroupedValues/Partial+WriteCounts/WriteFiles/GatherTempFileResults/ToList/Combine.perKey(ToList)/GroupByKey/Reify+WriteCounts/WriteFiles/GatherTempFileResults/ToList/Combine.perKey(ToList)/GroupByKey/Write Apr 06, 2020 11:27:20 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:27:19.444Z: Finished operation WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Read+WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/GroupByWindow+WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnwritten+WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/DropShardNum+WriteCounts/WriteFiles/GatherTempFileResults/ToList/WithKeys/AddKeys/Map+WriteCounts/WriteFiles/GatherTempFileResults/ToList/Combine.perKey(ToList)/GroupByKey+WriteCounts/WriteFiles/GatherTempFileResults/ToList/Combine.perKey(ToList)/Combine.GroupedValues/Partial+WriteCounts/WriteFiles/GatherTempFileResults/ToList/Combine.perKey(ToList)/GroupByKey/Reify+WriteCounts/WriteFiles/GatherTempFileResults/ToList/Combine.perKey(ToList)/GroupByKey/Write Apr 06, 2020 11:27:20 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:27:19.524Z: Executing operation WriteCounts/WriteFiles/GatherTempFileResults/ToList/Combine.perKey(ToList)/GroupByKey/Close Apr 06, 2020 11:27:20 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:27:19.578Z: Finished operation WriteCounts/WriteFiles/GatherTempFileResults/ToList/Combine.perKey(ToList)/GroupByKey/Close Apr 06, 2020 11:27:20 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:27:19.647Z: Executing operation WriteCounts/WriteFiles/GatherTempFileResults/ToList/Combine.perKey(ToList)/GroupByKey/Read+WriteCounts/WriteFiles/GatherTempFileResults/ToList/Combine.perKey(ToList)/Combine.GroupedValues+WriteCounts/WriteFiles/GatherTempFileResults/ToList/Combine.perKey(ToList)/Combine.GroupedValues/Extract+WriteCounts/WriteFiles/GatherTempFileResults/ToList/Values/Values/Map+WriteCounts/WriteFiles/FinalizeTempFileBundles/Finalize+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair with random key+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write+WriteCounts/WriteFiles/GatherTempFileResults/ToList/View.AsIterable/ParDo(ToIsmRecordForGlobalWindow) Apr 06, 2020 11:27:26 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:27:26.606Z: Finished operation WriteCounts/WriteFiles/GatherTempFileResults/ToList/Combine.perKey(ToList)/GroupByKey/Read+WriteCounts/WriteFiles/GatherTempFileResults/ToList/Combine.perKey(ToList)/Combine.GroupedValues+WriteCounts/WriteFiles/GatherTempFileResults/ToList/Combine.perKey(ToList)/Combine.GroupedValues/Extract+WriteCounts/WriteFiles/GatherTempFileResults/ToList/Values/Values/Map+WriteCounts/WriteFiles/FinalizeTempFileBundles/Finalize+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair with random key+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write+WriteCounts/WriteFiles/GatherTempFileResults/ToList/View.AsIterable/ParDo(ToIsmRecordForGlobalWindow) Apr 06, 2020 11:27:26 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:27:26.762Z: Executing operation WriteCounts/WriteFiles/GatherTempFileResults/ToList/View.AsIterable/CreateDataflowView Apr 06, 2020 11:27:26 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:27:26.811Z: Finished operation WriteCounts/WriteFiles/GatherTempFileResults/ToList/View.AsIterable/CreateDataflowView Apr 06, 2020 11:27:28 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:27:26.973Z: Executing operation WriteCounts/WriteFiles/GatherTempFileResults/ToList/CreateVoid/Read(CreateSource)+WriteCounts/WriteFiles/GatherTempFileResults/ToList/ProduceDefault+WriteCounts/WriteFiles/FinalizeTempFileBundles/Finalize+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair with random key+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write Apr 06, 2020 11:27:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:27:29.433Z: Finished operation WriteCounts/WriteFiles/GatherTempFileResults/ToList/CreateVoid/Read(CreateSource)+WriteCounts/WriteFiles/GatherTempFileResults/ToList/ProduceDefault+WriteCounts/WriteFiles/FinalizeTempFileBundles/Finalize+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair with random key+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write Apr 06, 2020 11:27:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:27:29.505Z: Executing operation WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Close Apr 06, 2020 11:27:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:27:29.558Z: Finished operation WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Close Apr 06, 2020 11:27:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:27:29.638Z: Executing operation WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Values/Values/Map Apr 06, 2020 11:27:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:27:30.924Z: Finished operation WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Values/Values/Map Apr 06, 2020 11:27:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:27:31.138Z: Cleaning up. Apr 06, 2020 11:27:32 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:27:31.202Z: Stopping worker pool... Apr 06, 2020 11:29:05 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:29:04.807Z: Autoscaling: Resized worker pool from 1 to 0. Apr 06, 2020 11:29:05 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-04-06T11:29:04.850Z: Worker pool stopped. Apr 06, 2020 11:29:10 AM org.apache.beam.runners.dataflow.DataflowPipelineJob logTerminalState INFO: Job 2020-04-06_04_24_53-4107355629350440775 finished with status DONE. gsutil cat gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/count* | grep Montague: Montague: 47 Verified Montague: 47 gsutil rm gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/count* Removing gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/counts-00000-of-00003... / [1 objects] Removing gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/counts-00001-of-00003... / [2 objects] Removing gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/counts-00002-of-00003... / [3 objects] Operation completed over 3 objects. [SUCCESS] FAILURE: Build failed with an exception. * What went wrong: Execution failed for task ':runners:google-cloud-dataflow-java:runMobileGamingJavaDataflow'. > Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with > non-zero exit value 1 * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. * Get more help at https://help.gradle.org Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings BUILD FAILED in 28m 55s 8 actionable tasks: 7 executed, 1 from cache Publishing build scan... https://gradle.com/s/5ct3rhafro32g Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
