See <https://builds.apache.org/job/beam_PostRelease_NightlySnapshot/673/display/redirect?page=changes>
Changes: [ttanay100] unskip ReifyTest.test_window [ttanay100] [BEAM-7437] Add streaming flag to BQ streaming inserts IT test [ttanay100] Change default timeout to 5 mins [kamil.wasilewski] [BEAM-7535] Created Jenkins job for BQ performance tests [kamil.wasilewski] [BEAM-7535] Delete existing data if the table already exists [iemejia] [BEAM-6740] Add PTransformTranslator for Combine.Globally [iemejia] [BEAM-6740] Add extractAcummulatorCoder for Combine.Globally and fix [iemejia] [BEAM-7640] Change tests to use PayloadTranslator instead of unused [iemejia] [BEAM-6740] Refactor to remove duplicated code in CombineTranslation [hannahjiang] BEAM-3645 add thread lock [cademarkegard] [BEAM-7690] Port WordCountTest off DoFnTester [kcweaver] [BEAM-7708] don't expect SQL shell bundled dependencies to be shadowed [github] [BEAM-7709] Re-use node for explicit flattens [boyuanz] Reformat CamelCase function naming style to underscore style for [boyuanz] fix lint [33895511+aromanenko-dev] [BEAM-6480] Adds AvroIO sink for generic records. (#9005) [github] [SQL][Doc] fix broken gradle command. [lcwik] Added new example on how to create a custom unbounded streaming source [chambers] Update Python Dataflow runner to patch side input coders on the unified [iemejia] [BEAM-7653] Add PTransformTranslator for Combine.GroupedValues [pachristopher] Update pyarrow version requirement in setup.py [github] Update code comments to improve readability in docs (#9024) ------------------------------------------ [...truncated 3.44 MB...] Jul 10, 2019 11:28:21 AM org.apache.beam.runners.dataflow.DataflowRunner fromOptions INFO: PipelineOptions.filesToStage was not specified. Defaulting to files from the classpath: will stage 131 files. Enable logging at DEBUG level to see which files will be staged. Jul 10, 2019 11:28:22 AM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Executing pipeline on the Dataflow Service, which will have billing implications related to Google Compute Engine usage and other Google Cloud Services. Jul 10, 2019 11:28:22 AM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements INFO: Uploading 131 files from PipelineOptions.filesToStage to staging location to prepare for execution. Jul 10, 2019 11:28:23 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-5977659997651563883-tmpdir/word-count-beam/target/classes to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/classes-txEKQu1BW0SpKgSRU5SfwQ.jar Jul 10, 2019 11:28:23 AM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements INFO: Staging files complete: 130 files cached, 1 files newly uploaded in 1 seconds Jul 10, 2019 11:28:24 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding ReadLines/Read as step s1 Jul 10, 2019 11:28:24 AM org.apache.beam.sdk.io.FileBasedSource getEstimatedSizeBytes INFO: Filepattern gs://apache-beam-samples/shakespeare/* matched 44 files with total size 5443510 Jul 10, 2019 11:28:24 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WordCount.CountWords/ParDo(ExtractWords) as step s2 Jul 10, 2019 11:28:24 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WordCount.CountWords/Count.PerElement/Init/Map as step s3 Jul 10, 2019 11:28:24 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey as step s4 Jul 10, 2019 11:28:24 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues as step s5 Jul 10, 2019 11:28:24 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding MapElements/Map as step s6 Jul 10, 2019 11:28:24 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/RewindowIntoGlobal/Window.Assign as step s7 Jul 10, 2019 11:28:24 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles as step s8 Jul 10, 2019 11:28:24 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten as step s9 Jul 10, 2019 11:28:24 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnwritten as step s10 Jul 10, 2019 11:28:24 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/DropShardNum as step s11 Jul 10, 2019 11:28:24 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/Flatten.PCollections as step s12 Jul 10, 2019 11:28:24 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow) as step s13 Jul 10, 2019 11:28:24 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/GatherTempFileResults/View.AsList/CreateDataflowView as step s14 Jul 10, 2019 11:28:24 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Create.Values/Read(CreateSource) as step s15 Jul 10, 2019 11:28:24 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Reify.ReifyView/ParDo(Anonymous) as step s16 Jul 10, 2019 11:28:24 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Values/Values/Map as step s17 Jul 10, 2019 11:28:24 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/FinalizeTempFileBundles/Finalize as step s18 Jul 10, 2019 11:28:24 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair with random key as step s19 Jul 10, 2019 11:28:24 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign as step s20 Jul 10, 2019 11:28:24 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey as step s21 Jul 10, 2019 11:28:24 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable as step s22 Jul 10, 2019 11:28:24 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Values/Values/Map as step s23 Jul 10, 2019 11:28:24 AM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Staging pipeline description to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/ Jul 10, 2019 11:28:24 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <64601 bytes, hash Ttxsk7kfPoHmeqENS3mZ5w> to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/pipeline-Ttxsk7kfPoHmeqENS3mZ5w.pb Dataflow SDK version: 2.15.0-SNAPSHOT Jul 10, 2019 11:28:26 AM org.apache.beam.runners.dataflow.DataflowRunner run INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-07-10_04_28_25-5738337480999237818?project=apache-beam-testing Submitted job: 2019-07-10_04_28_25-5738337480999237818 Jul 10, 2019 11:28:26 AM org.apache.beam.runners.dataflow.DataflowRunner run INFO: To cancel the job using the 'gcloud' tool, run: > gcloud dataflow jobs --project=apache-beam-testing cancel > --region=us-central1 2019-07-10_04_28_25-5738337480999237818 Jul 10, 2019 11:28:26 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:28:25.691Z: Autoscaling is enabled for job 2019-07-10_04_28_25-5738337480999237818. The number of workers will be between 1 and 1000. Jul 10, 2019 11:28:26 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:28:25.741Z: Autoscaling was automatically enabled for job 2019-07-10_04_28_25-5738337480999237818. Jul 10, 2019 11:28:28 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:28:28.451Z: Checking permissions granted to controller Service Account. Jul 10, 2019 11:28:33 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:28:32.049Z: Worker configuration: n1-standard-1 in us-central1-a. Jul 10, 2019 11:28:33 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:28:32.539Z: Expanding CoGroupByKey operations into optimizable parts. Jul 10, 2019 11:28:33 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:28:32.669Z: Expanding GroupByKey operations into optimizable parts. Jul 10, 2019 11:28:33 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:28:32.707Z: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns Jul 10, 2019 11:28:33 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:28:32.937Z: Fusing adjacent ParDo, Read, Write, and Flatten operations Jul 10, 2019 11:28:33 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:28:32.985Z: Fusing consumer WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Reify.ReifyView/ParDo(Anonymous) into WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Create.Values/Read(CreateSource) Jul 10, 2019 11:28:33 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:28:33.025Z: Fusing consumer WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign into WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair with random key Jul 10, 2019 11:28:34 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:28:33.071Z: Fusing consumer WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable into WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow Jul 10, 2019 11:28:34 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:28:33.121Z: Fusing consumer WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Values/Values/Map into WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable Jul 10, 2019 11:28:34 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:28:33.158Z: Fusing consumer WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify into WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign Jul 10, 2019 11:28:34 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:28:33.200Z: Fusing consumer WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write into WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify Jul 10, 2019 11:28:34 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:28:33.248Z: Fusing consumer WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow into WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read Jul 10, 2019 11:28:34 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:28:33.293Z: Fusing consumer WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Values/Values/Map into WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Reify.ReifyView/ParDo(Anonymous) Jul 10, 2019 11:28:34 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:28:33.331Z: Fusing consumer WriteCounts/WriteFiles/FinalizeTempFileBundles/Finalize into WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Values/Values/Map Jul 10, 2019 11:28:34 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:28:33.378Z: Fusing consumer WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair with random key into WriteCounts/WriteFiles/FinalizeTempFileBundles/Finalize Jul 10, 2019 11:28:34 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:28:33.424Z: Unzipping flatten s12 for input s11.org.apache.beam.sdk.values.PCollection.<init>:402#20ff67585e33a8f6 Jul 10, 2019 11:28:34 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:28:33.472Z: Fusing unzipped copy of WriteCounts/WriteFiles/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow), through flatten WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/Flatten.PCollections, into producer WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/DropShardNum Jul 10, 2019 11:28:34 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:28:33.516Z: Fusing consumer WriteCounts/WriteFiles/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow) into WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles Jul 10, 2019 11:28:34 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:28:33.556Z: Fusing consumer WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/DropShardNum into WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnwritten Jul 10, 2019 11:28:34 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:28:33.603Z: Fusing consumer WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues into WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Read Jul 10, 2019 11:28:34 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:28:33.649Z: Fusing consumer WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Write into WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Reify Jul 10, 2019 11:28:34 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:28:33.680Z: Fusing consumer WriteCounts/WriteFiles/RewindowIntoGlobal/Window.Assign into MapElements/Map Jul 10, 2019 11:28:34 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:28:33.722Z: Fusing consumer WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Extract into WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues Jul 10, 2019 11:28:34 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:28:33.758Z: Fusing consumer WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnwritten into WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/GroupByWindow Jul 10, 2019 11:28:34 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:28:33.795Z: Fusing consumer WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Partial into WordCount.CountWords/Count.PerElement/Init/Map Jul 10, 2019 11:28:34 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:28:33.830Z: Fusing consumer WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Write into WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Reify Jul 10, 2019 11:28:34 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:28:33.878Z: Fusing consumer WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles into WriteCounts/WriteFiles/RewindowIntoGlobal/Window.Assign Jul 10, 2019 11:28:34 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:28:33.925Z: Fusing consumer WordCount.CountWords/Count.PerElement/Init/Map into WordCount.CountWords/ParDo(ExtractWords) Jul 10, 2019 11:28:34 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:28:33.972Z: Fusing consumer WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/GroupByWindow into WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Read Jul 10, 2019 11:28:34 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:28:34.020Z: Fusing consumer MapElements/Map into WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Extract Jul 10, 2019 11:28:34 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:28:34.070Z: Fusing consumer WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Reify into WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Partial Jul 10, 2019 11:28:34 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:28:34.118Z: Fusing consumer WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Reify into WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles Jul 10, 2019 11:28:34 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:28:34.166Z: Fusing consumer WordCount.CountWords/ParDo(ExtractWords) into ReadLines/Read Jul 10, 2019 11:28:34 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:28:34.616Z: Executing operation WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Create Jul 10, 2019 11:28:34 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:28:34.660Z: Executing operation WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Create Jul 10, 2019 11:28:34 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:28:34.694Z: Executing operation WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Create Jul 10, 2019 11:28:34 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:28:34.718Z: Starting 1 workers in us-central1-a... Jul 10, 2019 11:28:34 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:28:34.772Z: Finished operation WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Create Jul 10, 2019 11:28:34 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:28:34.790Z: Finished operation WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Create Jul 10, 2019 11:28:34 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:28:34.790Z: Finished operation WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Create Jul 10, 2019 11:28:37 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:28:35.049Z: Executing operation ReadLines/Read+WordCount.CountWords/ParDo(ExtractWords)+WordCount.CountWords/Count.PerElement/Init/Map+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Partial+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Reify+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Write Jul 10, 2019 11:28:39 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:28:37.746Z: Worker configuration: n1-standard-1 in us-central1-a. Jul 10, 2019 11:29:26 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:29:26.185Z: Workers have started successfully. Jul 10, 2019 11:29:34 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:29:33.657Z: Autoscaling: Raised the number of workers to 1 based on the rate of progress in the currently running step(s). Jul 10, 2019 11:29:58 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:29:57.485Z: Workers have started successfully. Jul 10, 2019 11:30:57 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:30:56.007Z: Finished operation ReadLines/Read+WordCount.CountWords/ParDo(ExtractWords)+WordCount.CountWords/Count.PerElement/Init/Map+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Partial+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Reify+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Write Jul 10, 2019 11:30:57 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:30:56.088Z: Executing operation WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Close Jul 10, 2019 11:30:57 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:30:56.149Z: Finished operation WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Close Jul 10, 2019 11:30:57 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:30:56.230Z: Executing operation WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Read+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Extract+MapElements/Map+WriteCounts/WriteFiles/RewindowIntoGlobal/Window.Assign+WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles+WriteCounts/WriteFiles/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow)+WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Reify+WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Write Jul 10, 2019 11:31:06 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:31:05.039Z: Finished operation WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Read+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Extract+MapElements/Map+WriteCounts/WriteFiles/RewindowIntoGlobal/Window.Assign+WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles+WriteCounts/WriteFiles/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow)+WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Reify+WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Write Jul 10, 2019 11:31:06 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:31:05.168Z: Executing operation WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Close Jul 10, 2019 11:31:06 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:31:05.236Z: Finished operation WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Close Jul 10, 2019 11:31:06 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:31:05.328Z: Executing operation WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Read+WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/GroupByWindow+WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnwritten+WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/DropShardNum+WriteCounts/WriteFiles/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow) Jul 10, 2019 11:31:08 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:31:07.493Z: Finished operation WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Read+WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/GroupByWindow+WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnwritten+WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/DropShardNum+WriteCounts/WriteFiles/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow) Jul 10, 2019 11:31:08 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:31:07.688Z: Executing operation WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/Flatten.PCollections/Unzipped-1 Jul 10, 2019 11:31:08 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:31:07.776Z: Finished operation WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/Flatten.PCollections/Unzipped-1 Jul 10, 2019 11:31:08 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:31:07.975Z: Executing operation WriteCounts/WriteFiles/GatherTempFileResults/View.AsList/CreateDataflowView Jul 10, 2019 11:31:08 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:31:08.017Z: Finished operation WriteCounts/WriteFiles/GatherTempFileResults/View.AsList/CreateDataflowView Jul 10, 2019 11:31:08 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:31:08.196Z: Executing operation WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Create.Values/Read(CreateSource)+WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Reify.ReifyView/ParDo(Anonymous)+WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Values/Values/Map+WriteCounts/WriteFiles/FinalizeTempFileBundles/Finalize+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair with random key+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write Jul 10, 2019 11:31:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:31:11.709Z: Finished operation WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Create.Values/Read(CreateSource)+WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Reify.ReifyView/ParDo(Anonymous)+WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Values/Values/Map+WriteCounts/WriteFiles/FinalizeTempFileBundles/Finalize+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair with random key+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write Jul 10, 2019 11:31:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:31:11.807Z: Executing operation WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Close Jul 10, 2019 11:31:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:31:11.870Z: Finished operation WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Close Jul 10, 2019 11:31:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:31:11.960Z: Executing operation WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Values/Values/Map Jul 10, 2019 11:31:16 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:31:14.927Z: Finished operation WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Values/Values/Map Jul 10, 2019 11:31:16 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:31:15.158Z: Cleaning up. Jul 10, 2019 11:31:16 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:31:15.263Z: Stopping worker pool... Jul 10, 2019 11:33:57 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:33:55.802Z: Autoscaling: Resized worker pool from 1 to 0. Jul 10, 2019 11:33:57 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2019-07-10T11:33:55.857Z: Worker pool stopped. Jul 10, 2019 11:34:03 AM org.apache.beam.runners.dataflow.DataflowPipelineJob logTerminalState INFO: Job 2019-07-10_04_28_25-5738337480999237818 finished with status DONE. gsutil cat gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/count* | grep Montague: Montague: 47 Verified Montague: 47 gsutil rm gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/count* Removing gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/counts-00000-of-00003... / [1 objects] Removing gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/counts-00001-of-00003... / [2 objects] Removing gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/counts-00002-of-00003... / [3 objects] Operation completed over 3 objects. [SUCCESS] FAILURE: Build failed with an exception. * What went wrong: Execution failed for task ':runners:google-cloud-dataflow-java:runMobileGamingJavaDataflow'. > Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with > non-zero exit value 1 * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. * Get more help at https://help.gradle.org Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings BUILD FAILED in 33m 53s 8 actionable tasks: 7 executed, 1 from cache Publishing build scan... https://gradle.com/s/7biv7txas5ugs Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
