See <https://builds.apache.org/job/beam_PostRelease_NightlySnapshot/261/display/redirect?page=changes>
Changes: [ringwalt] Improve the TaggedOutput error when a tuple is passed as a tag. [github] print() is a function in Python 3 [echauchot] [BEAM-4283] Export nexmark query execution times to BigQuery [echauchot] [BEAM-4283] Make BigQueryIO Services classes public and experimental and [echauchot] [BEAM-4283] Fix PerfsToBigQueryTest and serialization issue in [kedin] [SQL] Make field names case-sensitive [robertwb] Suppress warning. [robertwb] Implement BeamFileSystemArtifactSource [robertwb] Add job submission capabilities to Flink runner. [robertwb] Update to new JobInvoker/Invocation interface. [robertwb] Remove references to Flink distributed file cache. [robertwb] A bit more documentation. [robertwb] Address comments on Flink code. [kirpichov] Removed some dead code, fixed some warnings [kirpichov] Adds tracing of watermarks in direct runner [kirpichov] Fixes typo in class name [kirpichov] Simplifies GBK expansion code a bit [kirpichov] Fixes bug: Processes values arriving for a GBK key sequentially [kirpichov] Introduces BundleProgressHandler. [swegner] Fix ErrorProne violations in direct runner and enable failOnWarnings [robertwb] Remove trailing whitespace. [kirpichov] [BEAM-4473,BEAM-4281] Makes ReferenceRunner use a Direct stream observer [ccy] [BEAM-4549] Revert #5623 [kirpichov] [BEAM-4291] Implements distributed artifact retrieval [kirpichov] Deletes ArtifactSource [thw] Fix Flink runner Maven build. [ccy] [BEAM-4549] Fix side inputs for streaming pipelines on Dataflow [Pablo] Improving state sampler Cython annotations. [Pablo] Adding msecs tracking to side inputs iterator [Pablo] Adding the sampling thread to side input microbench. [Pablo] Adding flag to mask change [Pablo] Fixing all lint raising-format-tuple warnings. ------------------------------------------ [...truncated 1.44 MB...] INFO: No stagingLocation provided, falling back to gcpTempLocation Jun 15, 2018 6:18:14 PM org.apache.beam.runners.dataflow.DataflowRunner fromOptions INFO: PipelineOptions.filesToStage was not specified. Defaulting to files from the classpath: will stage 110 files. Enable logging at DEBUG level to see which files will be staged. Jun 15, 2018 6:18:14 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Executing pipeline on the Dataflow Service, which will have billing implications related to Google Compute Engine usage and other Google Cloud Services. Jun 15, 2018 6:18:14 PM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements INFO: Uploading 110 files from PipelineOptions.filesToStage to staging location to prepare for execution. Jun 15, 2018 6:18:15 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/groovy-generated-3186513494882779622-tmpdir/word-count-beam/target/classes to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/classes-ZRe-PHY2KPW-kZWnovc0kw.jar Jun 15, 2018 6:18:15 PM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements INFO: Staging files complete: 109 files cached, 1 files newly uploaded Jun 15, 2018 6:18:16 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding ReadLines/Read as step s1 Jun 15, 2018 6:18:16 PM org.apache.beam.sdk.io.FileBasedSource getEstimatedSizeBytes INFO: Filepattern gs://apache-beam-samples/shakespeare/* matched 43 files with total size 5284696 Jun 15, 2018 6:18:16 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WordCount.CountWords/ParDo(ExtractWords) as step s2 Jun 15, 2018 6:18:16 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WordCount.CountWords/Count.PerElement/Init/Map as step s3 Jun 15, 2018 6:18:16 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey as step s4 Jun 15, 2018 6:18:16 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues as step s5 Jun 15, 2018 6:18:16 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding MapElements/Map as step s6 Jun 15, 2018 6:18:16 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/RewindowIntoGlobal/Window.Assign as step s7 Jun 15, 2018 6:18:16 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles as step s8 Jun 15, 2018 6:18:16 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten as step s9 Jun 15, 2018 6:18:16 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnwritten as step s10 Jun 15, 2018 6:18:16 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/DropShardNum as step s11 Jun 15, 2018 6:18:16 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/Flatten.PCollections as step s12 Jun 15, 2018 6:18:16 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow) as step s13 Jun 15, 2018 6:18:16 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/GatherTempFileResults/View.AsList/CreateDataflowView as step s14 Jun 15, 2018 6:18:16 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Create.Values/Read(CreateSource) as step s15 Jun 15, 2018 6:18:16 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Reify.ReifyView/ParDo(Anonymous) as step s16 Jun 15, 2018 6:18:16 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Values/Values/Map as step s17 Jun 15, 2018 6:18:16 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/FinalizeTempFileBundles/Finalize as step s18 Jun 15, 2018 6:18:16 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair with random key as step s19 Jun 15, 2018 6:18:16 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign as step s20 Jun 15, 2018 6:18:16 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey as step s21 Jun 15, 2018 6:18:16 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable as step s22 Jun 15, 2018 6:18:16 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Values/Values/Map as step s23 Jun 15, 2018 6:18:16 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Staging pipeline description to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/ Jun 15, 2018 6:18:16 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <63498 bytes, hash TNemLiZxIc4izQKc_Ow3rA> to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/pipeline-TNemLiZxIc4izQKc_Ow3rA.pb Dataflow SDK version: 2.6.0-SNAPSHOT Jun 15, 2018 6:18:17 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-06-15_11_18_16-1681204567593550754?project=apache-beam-testing Submitted job: 2018-06-15_11_18_16-1681204567593550754 Jun 15, 2018 6:18:17 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: To cancel the job using the 'gcloud' tool, run: > gcloud dataflow jobs --project=apache-beam-testing cancel > --region=us-central1 2018-06-15_11_18_16-1681204567593550754 Jun 15, 2018 6:18:29 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-15T18:18:16.796Z: Autoscaling is enabled for job 2018-06-15_11_18_16-1681204567593550754. The number of workers will be between 1 and 1000. Jun 15, 2018 6:18:29 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-15T18:18:16.818Z: Autoscaling was automatically enabled for job 2018-06-15_11_18_16-1681204567593550754. Jun 15, 2018 6:18:29 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-15T18:18:19.934Z: Checking required Cloud APIs are enabled. Jun 15, 2018 6:18:29 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-15T18:18:20.104Z: Checking permissions granted to controller Service Account. Jun 15, 2018 6:18:29 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-15T18:18:24.838Z: Worker configuration: n1-standard-1 in us-central1-f. Jun 15, 2018 6:18:29 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-15T18:18:25.198Z: Expanding CoGroupByKey operations into optimizable parts. Jun 15, 2018 6:18:29 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-15T18:18:25.323Z: Expanding GroupByKey operations into optimizable parts. Jun 15, 2018 6:18:29 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-15T18:18:25.366Z: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns Jun 15, 2018 6:18:29 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-15T18:18:25.576Z: Fusing adjacent ParDo, Read, Write, and Flatten operations Jun 15, 2018 6:18:29 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-15T18:18:25.657Z: Fusing consumer WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Reify.ReifyView/ParDo(Anonymous) into WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Create.Values/Read(CreateSource) Jun 15, 2018 6:18:29 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-15T18:18:25.691Z: Fusing consumer WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign into WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair with random key Jun 15, 2018 6:18:29 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-15T18:18:25.729Z: Fusing consumer WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable into WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow Jun 15, 2018 6:18:29 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-15T18:18:25.768Z: Fusing consumer WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Values/Values/Map into WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable Jun 15, 2018 6:18:29 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-15T18:18:25.794Z: Fusing consumer WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify into WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign Jun 15, 2018 6:18:29 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-15T18:18:25.837Z: Fusing consumer WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write into WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify Jun 15, 2018 6:18:29 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-15T18:18:25.864Z: Fusing consumer WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow into WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read Jun 15, 2018 6:18:29 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-15T18:18:25.890Z: Fusing consumer WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Values/Values/Map into WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Reify.ReifyView/ParDo(Anonymous) Jun 15, 2018 6:18:29 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-15T18:18:25.906Z: Fusing consumer WriteCounts/WriteFiles/FinalizeTempFileBundles/Finalize into WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Values/Values/Map Jun 15, 2018 6:18:29 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-15T18:18:25.932Z: Fusing consumer WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair with random key into WriteCounts/WriteFiles/FinalizeTempFileBundles/Finalize Jun 15, 2018 6:18:29 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-15T18:18:25.958Z: Unzipping flatten s12 for input s11.org.apache.beam.sdk.values.PCollection.<init>:329#20ff67585e33a8f6 Jun 15, 2018 6:18:29 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-15T18:18:25.987Z: Fusing unzipped copy of WriteCounts/WriteFiles/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow), through flatten WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/Flatten.PCollections, into producer WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/DropShardNum Jun 15, 2018 6:18:29 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-15T18:18:26.054Z: Fusing consumer WriteCounts/WriteFiles/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow) into WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles Jun 15, 2018 6:18:29 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-15T18:18:26.095Z: Fusing consumer WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/DropShardNum into WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnwritten Jun 15, 2018 6:18:29 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-15T18:18:26.119Z: Fusing consumer WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues into WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Read Jun 15, 2018 6:18:29 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-15T18:18:26.131Z: Fusing consumer WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Write into WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Reify Jun 15, 2018 6:18:29 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-15T18:18:26.178Z: Fusing consumer WriteCounts/WriteFiles/RewindowIntoGlobal/Window.Assign into MapElements/Map Jun 15, 2018 6:18:29 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-15T18:18:26.211Z: Fusing consumer WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Extract into WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues Jun 15, 2018 6:18:29 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-15T18:18:26.236Z: Fusing consumer WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnwritten into WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/GroupByWindow Jun 15, 2018 6:18:29 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-15T18:18:26.284Z: Fusing consumer WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Partial into WordCount.CountWords/Count.PerElement/Init/Map Jun 15, 2018 6:18:29 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-15T18:18:26.316Z: Fusing consumer WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Write into WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Reify Jun 15, 2018 6:18:29 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-15T18:18:26.343Z: Fusing consumer WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles into WriteCounts/WriteFiles/RewindowIntoGlobal/Window.Assign Jun 15, 2018 6:18:29 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-15T18:18:26.389Z: Fusing consumer WordCount.CountWords/Count.PerElement/Init/Map into WordCount.CountWords/ParDo(ExtractWords) Jun 15, 2018 6:18:29 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-15T18:18:26.417Z: Fusing consumer WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/GroupByWindow into WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Read Jun 15, 2018 6:18:29 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-15T18:18:26.440Z: Fusing consumer MapElements/Map into WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Extract Jun 15, 2018 6:18:29 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-15T18:18:26.467Z: Fusing consumer WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Reify into WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Partial Jun 15, 2018 6:18:29 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-15T18:18:26.483Z: Fusing consumer WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Reify into WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles Jun 15, 2018 6:18:29 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-15T18:18:26.515Z: Fusing consumer WordCount.CountWords/ParDo(ExtractWords) into ReadLines/Read Jun 15, 2018 6:18:29 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-15T18:18:26.857Z: Executing operation WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Create Jun 15, 2018 6:18:29 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-15T18:18:26.914Z: Executing operation WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Create Jun 15, 2018 6:18:29 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-15T18:18:26.949Z: Executing operation WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Create Jun 15, 2018 6:18:29 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-15T18:18:26.963Z: Starting 1 workers in us-central1-f... Jun 15, 2018 6:18:29 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-15T18:18:27.189Z: Executing operation ReadLines/Read+WordCount.CountWords/ParDo(ExtractWords)+WordCount.CountWords/Count.PerElement/Init/Map+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Partial+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Reify+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Write Jun 15, 2018 6:18:34 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-15T18:18:33.909Z: Autoscaling: Raised the number of workers to 0 based on the rate of progress in the currently running step(s). Jun 15, 2018 6:19:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-15T18:19:56.611Z: Autoscaling: Raised the number of workers to 1 based on the rate of progress in the currently running step(s). Jun 15, 2018 6:19:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-15T18:19:56.643Z: Autoscaling: Would further reduce the number of workers but reached the minimum number allowed for the job. Jun 15, 2018 6:20:22 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-15T18:20:22.614Z: Workers have started successfully. Jun 15, 2018 6:21:43 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-15T18:21:42.437Z: Executing operation WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Close Jun 15, 2018 6:21:43 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-15T18:21:42.515Z: Executing operation WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Read+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Extract+MapElements/Map+WriteCounts/WriteFiles/RewindowIntoGlobal/Window.Assign+WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles+WriteCounts/WriteFiles/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow)+WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Reify+WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Write Jun 15, 2018 6:21:55 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-15T18:21:52.794Z: Executing operation WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Close Jun 15, 2018 6:21:55 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-15T18:21:52.865Z: Executing operation WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Read+WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/GroupByWindow+WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnwritten+WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/DropShardNum+WriteCounts/WriteFiles/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow) Jun 15, 2018 6:21:57 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-15T18:21:55.778Z: Executing operation s12-u40 Jun 15, 2018 6:21:57 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-15T18:21:55.932Z: Executing operation WriteCounts/WriteFiles/GatherTempFileResults/View.AsList/CreateDataflowView Jun 15, 2018 6:21:57 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-15T18:21:56.082Z: Executing operation WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Create.Values/Read(CreateSource)+WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Reify.ReifyView/ParDo(Anonymous)+WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Values/Values/Map+WriteCounts/WriteFiles/FinalizeTempFileBundles/Finalize+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair with random key+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write Jun 15, 2018 6:22:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-15T18:21:59.079Z: Executing operation WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Close Jun 15, 2018 6:22:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-15T18:21:59.132Z: Executing operation WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Values/Values/Map Jun 15, 2018 6:22:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-15T18:22:00.430Z: Cleaning up. Jun 15, 2018 6:22:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-15T18:22:00.504Z: Stopping worker pool... Jun 15, 2018 6:23:39 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-15T18:23:38.035Z: Autoscaling: Resized worker pool from 1 to 0. Jun 15, 2018 6:23:39 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-15T18:23:38.066Z: Autoscaling: Would further reduce the number of workers but reached the minimum number allowed for the job. Jun 15, 2018 6:23:39 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-15T18:23:38.107Z: Worker pool stopped. Jun 15, 2018 6:23:47 PM org.apache.beam.runners.dataflow.DataflowPipelineJob waitUntilFinish INFO: Job 2018-06-15_11_18_16-1681204567593550754 finished with status DONE. gsutil cat gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/count* | grep Montague: Montague: 47 Verified Montague: 47 gsutil rm gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/count* Removing gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/counts-00000-of-00003... / [1 objects] Removing gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/counts-00001-of-00003... / [2 objects] Removing gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/counts-00002-of-00003... / [3 objects] Operation completed over 3 objects. [SUCCESS] :beam-runners-google-cloud-dataflow-java:runQuickstartJavaDataflow (Thread[Daemon worker,5,main]) completed. Took 6 mins 47.667 secs. FAILURE: Build completed with 2 failures. 1: Task failed with an exception. ----------- * What went wrong: Execution failed for task ':beam-runners-direct-java:runMobileGamingJavaDirect'. > Process 'command '/usr/local/asfpackages/java/jdk1.8.0_152/bin/java'' > finished with non-zero exit value 1 * Try: Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights. ============================================================================== 2: Task failed with an exception. ----------- * What went wrong: Execution failed for task ':beam-runners-google-cloud-dataflow-java:runMobileGamingJavaDataflow'. > Process 'command '/usr/local/asfpackages/java/jdk1.8.0_152/bin/java'' > finished with non-zero exit value 1 * Try: Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights. ============================================================================== * Get more help at https://help.gradle.org BUILD FAILED in 11m 49s 8 actionable tasks: 7 executed, 1 from cache Publishing build scan... https://gradle.com/s/ypnhwwqbg36s6 Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected]
