See
<https://builds.apache.org/job/beam_PostRelease_NightlySnapshot/549/display/redirect?page=changes>
Changes:
[coheigea] Use assertEquals(x, y) instead of assertTrue(x.equals(y))
[lukasz.gajowy] [BEAM-6677] Provide necessary Docker and Flink Dataproc init
actions
[lukasz.gajowy] [BEAM-6677] Create script that will pull SDK Harness images to
Dataproc
[lukasz.gajowy] [BEAM-6677] Add script to easily setup Flink Dataproc cluster
[lukasz.gajowy] [BEAM-6677] Adding apache license to file.
[lukasz.gajowy] [BEAM-6677] Fix documentation
[lukasz.gajowy] [BEAM-6677] Use env variables instead of numbered args. Pass
list of
[lukasz.gajowy] [BEAM-5985] Set default seed in SyntheticOptions
[lukasz.gajowy] [BEAM-5985] Change Big Query Dataset name according to job
running
[lukasz.gajowy] [BEAM-5985] Add load tests for Dataflow runner according to
proposal doc
[heejong] [BEAM-6666] subprocess.Popen hangs after use of gRPC channel
[pabloem] Running Dataflow examples on windmill service
[github] Update ReshuffleOverrideFactory to keep Reshuffle's
[25622840+adude3141] [BEAM-6726] explicitly specify signing key
[lukasz.gajowy] [BEAM-5985] Refactor: move smoke load test suite to separate
file
[lukasz.gajowy] [BEAM-5985] Add cron job for load tests
[lukasz.gajowy] [BEAM-5985] Extract common code
[michal.walenia] [BEAM-6703] Added a Jenkins job to test a Direct runner
compiled with
[chamikara] [BEAM-6748] Account for synchronization interval when estimating
amount
[github] Update Python 3 support roadmap.
[mxm] [BEAM-6754] Add option to use subprocesses instead of threads in
------------------------------------------
[...truncated 3.97 MB...]
Submitted job: 2019-03-12_04_17_22-15656180242737806888
Mar 12, 2019 11:17:23 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel
> --region=us-central1 2019-03-12_04_17_22-15656180242737806888
Mar 12, 2019 11:17:37 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-03-12T11:17:22.389Z: Autoscaling is enabled for job
2019-03-12_04_17_22-15656180242737806888. The number of workers will be between
1 and 1000.
Mar 12, 2019 11:17:37 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-03-12T11:17:22.460Z: Autoscaling was automatically enabled for job
2019-03-12_04_17_22-15656180242737806888.
Mar 12, 2019 11:17:37 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-03-12T11:17:25.146Z: Checking permissions granted to controller
Service Account.
Mar 12, 2019 11:17:37 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-03-12T11:17:29.941Z: Worker configuration: n1-standard-1 in
us-central1-a.
Mar 12, 2019 11:17:37 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-03-12T11:17:30.411Z: Expanding CoGroupByKey operations into
optimizable parts.
Mar 12, 2019 11:17:37 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-03-12T11:17:30.549Z: Expanding GroupByKey operations into
optimizable parts.
Mar 12, 2019 11:17:37 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-03-12T11:17:30.589Z: Lifting ValueCombiningMappingFns into
MergeBucketsMappingFns
Mar 12, 2019 11:17:37 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-03-12T11:17:30.801Z: Fusing adjacent ParDo, Read, Write, and Flatten
operations
Mar 12, 2019 11:17:37 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-03-12T11:17:30.834Z: Fusing consumer
WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Reify.ReifyView/ParDo(Anonymous)
into
WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Create.Values/Read(CreateSource)
Mar 12, 2019 11:17:37 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-03-12T11:17:30.871Z: Fusing consumer
WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign
into
WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair with
random key
Mar 12, 2019 11:17:37 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-03-12T11:17:30.920Z: Fusing consumer
WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable
into
WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow
Mar 12, 2019 11:17:37 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-03-12T11:17:30.957Z: Fusing consumer
WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Values/Values/Map
into
WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable
Mar 12, 2019 11:17:37 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-03-12T11:17:30.992Z: Fusing consumer
WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify
into
WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign
Mar 12, 2019 11:17:37 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-03-12T11:17:31.036Z: Fusing consumer
WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write
into
WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify
Mar 12, 2019 11:17:37 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-03-12T11:17:31.083Z: Fusing consumer
WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow
into
WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read
Mar 12, 2019 11:17:37 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-03-12T11:17:31.122Z: Fusing consumer
WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Values/Values/Map
into
WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Reify.ReifyView/ParDo(Anonymous)
Mar 12, 2019 11:17:37 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-03-12T11:17:31.163Z: Fusing consumer
WriteCounts/WriteFiles/FinalizeTempFileBundles/Finalize into
WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Values/Values/Map
Mar 12, 2019 11:17:37 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-03-12T11:17:31.206Z: Fusing consumer
WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair with
random key into WriteCounts/WriteFiles/FinalizeTempFileBundles/Finalize
Mar 12, 2019 11:17:37 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-03-12T11:17:31.243Z: Unzipping flatten s12 for input
s11.org.apache.beam.sdk.values.PCollection.<init>:402#20ff67585e33a8f6
Mar 12, 2019 11:17:37 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-03-12T11:17:31.289Z: Fusing unzipped copy of
WriteCounts/WriteFiles/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow),
through flatten
WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/Flatten.PCollections,
into producer
WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/DropShardNum
Mar 12, 2019 11:17:37 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-03-12T11:17:31.335Z: Fusing consumer
WriteCounts/WriteFiles/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow)
into
WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles
Mar 12, 2019 11:17:37 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-03-12T11:17:31.381Z: Fusing consumer
WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/DropShardNum into
WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnwritten
Mar 12, 2019 11:17:37 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-03-12T11:17:31.427Z: Fusing consumer
WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues
into
WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Read
Mar 12, 2019 11:17:37 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-03-12T11:17:31.465Z: Fusing consumer
WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Write
into
WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Reify
Mar 12, 2019 11:17:37 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-03-12T11:17:31.513Z: Fusing consumer
WriteCounts/WriteFiles/RewindowIntoGlobal/Window.Assign into MapElements/Map
Mar 12, 2019 11:17:37 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-03-12T11:17:31.551Z: Fusing consumer
WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Extract
into
WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues
Mar 12, 2019 11:17:37 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-03-12T11:17:31.598Z: Fusing consumer
WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnwritten into
WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/GroupByWindow
Mar 12, 2019 11:17:37 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-03-12T11:17:31.640Z: Fusing consumer
WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Partial
into WordCount.CountWords/Count.PerElement/Init/Map
Mar 12, 2019 11:17:37 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-03-12T11:17:31.666Z: Fusing consumer
WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Write
into
WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Reify
Mar 12, 2019 11:17:37 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-03-12T11:17:31.758Z: Fusing consumer
WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles
into WriteCounts/WriteFiles/RewindowIntoGlobal/Window.Assign
Mar 12, 2019 11:17:37 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-03-12T11:17:31.792Z: Fusing consumer
WordCount.CountWords/Count.PerElement/Init/Map into
WordCount.CountWords/ParDo(ExtractWords)
Mar 12, 2019 11:17:37 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-03-12T11:17:31.830Z: Fusing consumer
WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/GroupByWindow
into
WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Read
Mar 12, 2019 11:17:37 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-03-12T11:17:31.873Z: Fusing consumer MapElements/Map into
WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Extract
Mar 12, 2019 11:17:37 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-03-12T11:17:31.920Z: Fusing consumer
WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Reify
into
WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Partial
Mar 12, 2019 11:17:37 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-03-12T11:17:31.965Z: Fusing consumer
WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Reify
into
WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles
Mar 12, 2019 11:17:37 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-03-12T11:17:32.014Z: Fusing consumer
WordCount.CountWords/ParDo(ExtractWords) into ReadLines/Read
Mar 12, 2019 11:17:37 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-03-12T11:17:32.506Z: Executing operation
WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Create
Mar 12, 2019 11:17:37 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-03-12T11:17:32.553Z: Executing operation
WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Create
Mar 12, 2019 11:17:37 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-03-12T11:17:32.591Z: Executing operation
WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Create
Mar 12, 2019 11:17:37 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-03-12T11:17:32.601Z: Starting 1 workers in us-central1-a...
Mar 12, 2019 11:17:37 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-03-12T11:17:32.862Z: Executing operation
ReadLines/Read+WordCount.CountWords/ParDo(ExtractWords)+WordCount.CountWords/Count.PerElement/Init/Map+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Partial+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Reify+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Write
Mar 12, 2019 11:17:43 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-03-12T11:17:42.640Z: Autoscaling: Raised the number of workers to 0
based on the rate of progress in the currently running step(s).
Mar 12, 2019 11:18:25 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-03-12T11:18:23.093Z: Workers have started successfully.
Mar 12, 2019 11:18:25 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-03-12T11:18:24.906Z: Autoscaling: Raised the number of workers to 1
based on the rate of progress in the currently running step(s).
Mar 12, 2019 11:18:25 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-03-12T11:18:24.954Z: Autoscaling: Would further reduce the number of
workers but reached the minimum number allowed for the job.
Mar 12, 2019 11:18:38 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
SEVERE: 2019-03-12T11:18:36.362Z: java.lang.NoSuchMethodError:
org.apache.beam.sdk.transforms.reflect.DoFnSignature$ProcessElementMethod.getRowParameter()Lorg/apache/beam/sdk/transforms/reflect/DoFnSignature$Parameter$RowParameter;
at
org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.<init>(SimpleDoFnRunner.java:175)
at
org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.DoFnRunners.simpleRunner(DoFnRunners.java:63)
at
org.apache.beam.runners.dataflow.worker.SimpleDoFnRunnerFactory.createRunner(SimpleDoFnRunnerFactory.java:52)
at
org.apache.beam.runners.dataflow.worker.SimpleParDoFn.reallyStartBundle(SimpleParDoFn.java:288)
at
org.apache.beam.runners.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:312)
at
org.apache.beam.runners.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:44)
at
org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:49)
at
org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:201)
at
org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:159)
at
org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:76)
at
org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.executeWork(BatchDataflowWorker.java:409)
at
org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:378)
at
org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:306)
at
org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:135)
at
org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:115)
at
org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:102)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Mar 12, 2019 11:18:44 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
SEVERE: 2019-03-12T11:18:43.323Z: java.lang.NoSuchMethodError:
org.apache.beam.sdk.transforms.reflect.DoFnSignature$ProcessElementMethod.getRowParameter()Lorg/apache/beam/sdk/transforms/reflect/DoFnSignature$Parameter$RowParameter;
at
org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.<init>(SimpleDoFnRunner.java:175)
at
org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.DoFnRunners.simpleRunner(DoFnRunners.java:63)
at
org.apache.beam.runners.dataflow.worker.SimpleDoFnRunnerFactory.createRunner(SimpleDoFnRunnerFactory.java:52)
at
org.apache.beam.runners.dataflow.worker.SimpleParDoFn.reallyStartBundle(SimpleParDoFn.java:288)
at
org.apache.beam.runners.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:312)
at
org.apache.beam.runners.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:44)
at
org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:49)
at
org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:201)
at
org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:159)
at
org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:76)
at
org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.executeWork(BatchDataflowWorker.java:409)
at
org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:378)
at
org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:306)
at
org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:135)
at
org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:115)
at
org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:102)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Mar 12, 2019 11:18:55 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
SEVERE: 2019-03-12T11:18:53.223Z: java.lang.NoSuchMethodError:
org.apache.beam.sdk.transforms.reflect.DoFnSignature$ProcessElementMethod.getRowParameter()Lorg/apache/beam/sdk/transforms/reflect/DoFnSignature$Parameter$RowParameter;
at
org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.<init>(SimpleDoFnRunner.java:175)
at
org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.DoFnRunners.simpleRunner(DoFnRunners.java:63)
at
org.apache.beam.runners.dataflow.worker.SimpleDoFnRunnerFactory.createRunner(SimpleDoFnRunnerFactory.java:52)
at
org.apache.beam.runners.dataflow.worker.SimpleParDoFn.reallyStartBundle(SimpleParDoFn.java:288)
at
org.apache.beam.runners.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:312)
at
org.apache.beam.runners.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:44)
at
org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:49)
at
org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:201)
at
org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:159)
at
org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:76)
at
org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.executeWork(BatchDataflowWorker.java:409)
at
org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:378)
at
org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:306)
at
org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:135)
at
org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:115)
at
org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:102)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Mar 12, 2019 11:18:55 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-03-12T11:18:54.253Z: Workers have started successfully.
Mar 12, 2019 11:19:03 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
SEVERE: 2019-03-12T11:19:02.016Z: java.lang.NoSuchMethodError:
org.apache.beam.sdk.transforms.reflect.DoFnSignature$ProcessElementMethod.getRowParameter()Lorg/apache/beam/sdk/transforms/reflect/DoFnSignature$Parameter$RowParameter;
at
org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.<init>(SimpleDoFnRunner.java:175)
at
org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.DoFnRunners.simpleRunner(DoFnRunners.java:63)
at
org.apache.beam.runners.dataflow.worker.SimpleDoFnRunnerFactory.createRunner(SimpleDoFnRunnerFactory.java:52)
at
org.apache.beam.runners.dataflow.worker.SimpleParDoFn.reallyStartBundle(SimpleParDoFn.java:288)
at
org.apache.beam.runners.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:312)
at
org.apache.beam.runners.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:44)
at
org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:49)
at
org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:201)
at
org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:159)
at
org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:76)
at
org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.executeWork(BatchDataflowWorker.java:409)
at
org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:378)
at
org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:306)
at
org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:135)
at
org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:115)
at
org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:102)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Mar 12, 2019 11:19:03 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
SEVERE: 2019-03-12T11:19:02.197Z: Workflow failed. Causes:
S02:ReadLines/Read+WordCount.CountWords/ParDo(ExtractWords)+WordCount.CountWords/Count.PerElement/Init/Map+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Partial+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Reify+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Write
failed., A work item was attempted 4 times without success. Each time the
worker eventually lost contact with the service. The work item was attempted
on:
wordcount-jenkins-0312111-03120417-4o14-harness-t12j,
wordcount-jenkins-0312111-03120417-4o14-harness-t12j,
wordcount-jenkins-0312111-03120417-4o14-harness-t12j,
wordcount-jenkins-0312111-03120417-4o14-harness-t12j
Mar 12, 2019 11:19:03 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-03-12T11:19:02.367Z: Cleaning up.
Mar 12, 2019 11:19:03 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-03-12T11:19:02.497Z: Stopping worker pool...
Mar 12, 2019 11:21:40 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-03-12T11:21:39.725Z: Autoscaling: Reduced the number of workers to 0
based on the rate of progress in the currently running step(s).
Mar 12, 2019 11:21:40 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-03-12T11:21:39.764Z: Autoscaling: Would further reduce the number of
workers but reached the minimum number allowed for the job.
Mar 12, 2019 11:21:40 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2019-03-12T11:21:39.826Z: Worker pool stopped.
Mar 12, 2019 11:21:46 AM org.apache.beam.runners.dataflow.DataflowPipelineJob
waitUntilFinish
INFO: Job 2019-03-12_04_17_22-15656180242737806888 failed with status FAILED.
gsutil cat
gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/count*
| grep Montague:
CommandException: No URLs matched:
gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/count*
CommandException: No URLs matched:
gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/count*
[ERROR] Failed command
> Task :beam-runners-google-cloud-dataflow-java:runQuickstartJavaDataflow FAILED
FAILURE: Build completed with 2 failures.
1: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task
':beam-runners-google-cloud-dataflow-java:runMobileGamingJavaDataflow'.
> Process 'command '/usr/local/asfpackages/java/jdk1.8.0_191/bin/java''
> finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug
option to get more log output. Run with --scan to get full insights.
==============================================================================
2: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task
':beam-runners-google-cloud-dataflow-java:runQuickstartJavaDataflow'.
> Process 'command '/usr/local/asfpackages/java/jdk1.8.0_191/bin/java''
> finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug
option to get more log output. Run with --scan to get full insights.
==============================================================================
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with
Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See
https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 21m 40s
8 actionable tasks: 7 executed, 1 from cache
Publishing build scan...
https://gradle.com/s/sfawp2itry3k2
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
