See <https://ci-beam.apache.org/job/beam_PostRelease_NightlySnapshot/1347/display/redirect?page=changes>
Changes: [Elliotte Rusty Harold] use toMinutes [Elliotte Rusty Harold] update test [samuelw] [BEAM-12118] Fix racy precondition exception in QueueingBeamFnDataClient [noreply] Change kafka table provider properties structure. (#14507) [noreply] [BEAM-11839] Improve DeferredFrameTest._run_test (#14552) [noreply] [BEAM-12191] Fix a bug that upload_graph doesn't reduce template file [noreply] [BEAM-10029] Spanner IO read and write performance tests (#13857) ------------------------------------------ [...truncated 4.63 MB...] INFO: To cancel the job using the 'gcloud' tool, run: > gcloud dataflow jobs --project=apache-beam-testing cancel > --region=us-central1 2021-04-20_04_28_07-11868786282907612053 Apr 20, 2021 11:28:13 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-04-20T11:28:10.763Z: Autoscaling is enabled for job 2021-04-20_04_28_07-11868786282907612053. The number of workers will be between 1 and 1000. Apr 20, 2021 11:28:13 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-04-20T11:28:11.365Z: Autoscaling was automatically enabled for job 2021-04-20_04_28_07-11868786282907612053. Apr 20, 2021 11:28:21 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-04-20T11:28:19.672Z: Worker configuration: n1-standard-1 in us-central1-c. Apr 20, 2021 11:28:21 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-04-20T11:28:20.603Z: Expanding CoGroupByKey operations into optimizable parts. Apr 20, 2021 11:28:21 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-04-20T11:28:21.093Z: Expanding GroupByKey operations into optimizable parts. Apr 20, 2021 11:28:21 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-04-20T11:28:21.209Z: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns Apr 20, 2021 11:28:23 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-04-20T11:28:21.783Z: Fusing adjacent ParDo, Read, Write, and Flatten operations Apr 20, 2021 11:28:23 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-04-20T11:28:21.895Z: Fusing consumer WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Reify.ReifyView/ParDo(Anonymous) into WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Create.Values/Read(CreateSource) Apr 20, 2021 11:28:23 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-04-20T11:28:22.009Z: Fusing consumer WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Values/Values/Map into WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Reify.ReifyView/ParDo(Anonymous) Apr 20, 2021 11:28:23 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-04-20T11:28:22.139Z: Fusing consumer WriteCounts/WriteFiles/FinalizeTempFileBundles/Finalize into WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Values/Values/Map Apr 20, 2021 11:28:23 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-04-20T11:28:22.245Z: Fusing consumer WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair with random key into WriteCounts/WriteFiles/FinalizeTempFileBundles/Finalize Apr 20, 2021 11:28:23 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-04-20T11:28:22.382Z: Fusing consumer WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign into WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair with random key Apr 20, 2021 11:28:23 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-04-20T11:28:22.509Z: Fusing consumer WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify into WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign Apr 20, 2021 11:28:23 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-04-20T11:28:22.638Z: Fusing consumer WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write into WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify Apr 20, 2021 11:28:23 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-04-20T11:28:22.747Z: Fusing consumer WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow into WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read Apr 20, 2021 11:28:23 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-04-20T11:28:22.846Z: Fusing consumer WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable into WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow Apr 20, 2021 11:28:23 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-04-20T11:28:22.955Z: Fusing consumer WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Values/Values/Map into WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable Apr 20, 2021 11:28:23 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-04-20T11:28:23.065Z: Unzipping flatten s12 for input s11.org.apache.beam.sdk.values.PCollection.<init>:402#5f2ef1f005ae0b4 Apr 20, 2021 11:28:24 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-04-20T11:28:23.196Z: Fusing unzipped copy of WriteCounts/WriteFiles/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow), through flatten WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/Flatten.PCollections, into producer WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/DropShardNum Apr 20, 2021 11:28:24 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-04-20T11:28:23.306Z: Fusing consumer WriteCounts/WriteFiles/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow) into WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles Apr 20, 2021 11:28:24 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-04-20T11:28:23.413Z: Fusing consumer WordCount.CountWords/ParDo(ExtractWords) into ReadLines/Read Apr 20, 2021 11:28:24 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-04-20T11:28:23.534Z: Fusing consumer WordCount.CountWords/Count.PerElement/Init/Map into WordCount.CountWords/ParDo(ExtractWords) Apr 20, 2021 11:28:24 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-04-20T11:28:23.640Z: Fusing consumer WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Partial into WordCount.CountWords/Count.PerElement/Init/Map Apr 20, 2021 11:28:24 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-04-20T11:28:23.755Z: Fusing consumer WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Reify into WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Partial Apr 20, 2021 11:28:24 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-04-20T11:28:23.891Z: Fusing consumer WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Write into WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Reify Apr 20, 2021 11:28:24 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-04-20T11:28:24.005Z: Fusing consumer WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues into WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Read Apr 20, 2021 11:28:24 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-04-20T11:28:24.122Z: Fusing consumer WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Extract into WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues Apr 20, 2021 11:28:24 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-04-20T11:28:24.232Z: Fusing consumer MapElements/Map into WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Extract Apr 20, 2021 11:28:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-04-20T11:28:24.347Z: Fusing consumer WriteCounts/WriteFiles/RewindowIntoGlobal/Window.Assign into MapElements/Map Apr 20, 2021 11:28:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-04-20T11:28:24.476Z: Fusing consumer WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles into WriteCounts/WriteFiles/RewindowIntoGlobal/Window.Assign Apr 20, 2021 11:28:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-04-20T11:28:24.589Z: Fusing consumer WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Reify into WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles Apr 20, 2021 11:28:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-04-20T11:28:24.697Z: Fusing consumer WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Write into WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Reify Apr 20, 2021 11:28:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-04-20T11:28:24.801Z: Fusing consumer WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/GroupByWindow into WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Read Apr 20, 2021 11:28:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-04-20T11:28:24.902Z: Fusing consumer WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnwritten into WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/GroupByWindow Apr 20, 2021 11:28:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-04-20T11:28:25.029Z: Fusing consumer WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/DropShardNum into WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnwritten Apr 20, 2021 11:28:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-04-20T11:28:26.260Z: Executing operation WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Create Apr 20, 2021 11:28:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-04-20T11:28:26.542Z: Starting 1 workers in us-central1-c... Apr 20, 2021 11:28:29 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-04-20T11:28:26.962Z: Finished operation WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Create Apr 20, 2021 11:28:29 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-04-20T11:28:27.610Z: Executing operation ReadLines/Read+WordCount.CountWords/ParDo(ExtractWords)+WordCount.CountWords/Count.PerElement/Init/Map+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Partial+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Reify+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Write Apr 20, 2021 11:28:40 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-04-20T11:28:39.213Z: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete Apr 20, 2021 11:29:19 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-04-20T11:29:18.544Z: Autoscaling: Raised the number of workers to 1 based on the rate of progress in the currently running stage(s). Apr 20, 2021 11:29:55 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-04-20T11:29:54.175Z: Workers have started successfully. Apr 20, 2021 11:29:55 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-04-20T11:29:54.286Z: Workers have started successfully. Apr 20, 2021 11:30:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process SEVERE: 2021-04-20T11:30:02.564Z: java.lang.NoClassDefFoundError: org/apache/beam/sdk/metrics/MetricsLogger at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:763) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) at java.net.URLClassLoader.access$100(URLClassLoader.java:73) at java.net.URLClassLoader$1.run(URLClassLoader.java:368) at java.net.URLClassLoader$1.run(URLClassLoader.java:362) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:361) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at org.apache.beam.runners.dataflow.worker.BatchModeExecutionContext$1.createContainer(BatchModeExecutionContext.java:106) at org.apache.beam.runners.dataflow.worker.BatchModeExecutionContext$1.createContainer(BatchModeExecutionContext.java:103) at org.apache.beam.runners.dataflow.worker.MetricsContainerRegistry.lambda$getContainer$0(MetricsContainerRegistry.java:39) at java.util.concurrent.ConcurrentSkipListMap.computeIfAbsent(ConcurrentSkipListMap.java:1682) at org.apache.beam.runners.dataflow.worker.MetricsContainerRegistry.getContainer(MetricsContainerRegistry.java:39) at org.apache.beam.runners.dataflow.worker.DataflowExecutionContext.createOperationContext(DataflowExecutionContext.java:212) at org.apache.beam.runners.dataflow.worker.SourceOperationExecutorFactory.create(SourceOperationExecutorFactory.java:49) at org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:376) at org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:314) at org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:140) at org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:120) at org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:107) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.ClassNotFoundException: org.apache.beam.sdk.metrics.MetricsLogger at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 28 more Apr 20, 2021 11:30:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process SEVERE: 2021-04-20T11:30:07.324Z: java.lang.NoClassDefFoundError: org/apache/beam/runners/dataflow/worker/repackaged/org/apache/beam/runners/core/metrics/MetricsContainerImpl at org.apache.beam.runners.dataflow.worker.BatchModeExecutionContext$1.createContainer(BatchModeExecutionContext.java:106) at org.apache.beam.runners.dataflow.worker.BatchModeExecutionContext$1.createContainer(BatchModeExecutionContext.java:103) at org.apache.beam.runners.dataflow.worker.MetricsContainerRegistry.lambda$getContainer$0(MetricsContainerRegistry.java:39) at java.util.concurrent.ConcurrentSkipListMap.computeIfAbsent(ConcurrentSkipListMap.java:1682) at org.apache.beam.runners.dataflow.worker.MetricsContainerRegistry.getContainer(MetricsContainerRegistry.java:39) at org.apache.beam.runners.dataflow.worker.DataflowExecutionContext.createOperationContext(DataflowExecutionContext.java:212) at org.apache.beam.runners.dataflow.worker.SourceOperationExecutorFactory.create(SourceOperationExecutorFactory.java:49) at org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:376) at org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:314) at org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:140) at org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:120) at org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:107) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Apr 20, 2021 11:30:22 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process SEVERE: 2021-04-20T11:30:20.527Z: java.lang.NoClassDefFoundError: org/apache/beam/runners/dataflow/worker/repackaged/org/apache/beam/runners/core/metrics/MetricsContainerImpl at org.apache.beam.runners.dataflow.worker.BatchModeExecutionContext$1.createContainer(BatchModeExecutionContext.java:106) at org.apache.beam.runners.dataflow.worker.BatchModeExecutionContext$1.createContainer(BatchModeExecutionContext.java:103) at org.apache.beam.runners.dataflow.worker.MetricsContainerRegistry.lambda$getContainer$0(MetricsContainerRegistry.java:39) at java.util.concurrent.ConcurrentSkipListMap.computeIfAbsent(ConcurrentSkipListMap.java:1682) at org.apache.beam.runners.dataflow.worker.MetricsContainerRegistry.getContainer(MetricsContainerRegistry.java:39) at org.apache.beam.runners.dataflow.worker.DataflowExecutionContext.createOperationContext(DataflowExecutionContext.java:212) at org.apache.beam.runners.dataflow.worker.SourceOperationExecutorFactory.create(SourceOperationExecutorFactory.java:49) at org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:376) at org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:314) at org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:140) at org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:120) at org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:107) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Apr 20, 2021 11:30:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process SEVERE: 2021-04-20T11:30:30.058Z: java.lang.NoClassDefFoundError: org/apache/beam/runners/dataflow/worker/repackaged/org/apache/beam/runners/core/metrics/MetricsContainerImpl at org.apache.beam.runners.dataflow.worker.BatchModeExecutionContext$1.createContainer(BatchModeExecutionContext.java:106) at org.apache.beam.runners.dataflow.worker.BatchModeExecutionContext$1.createContainer(BatchModeExecutionContext.java:103) at org.apache.beam.runners.dataflow.worker.MetricsContainerRegistry.lambda$getContainer$0(MetricsContainerRegistry.java:39) at java.util.concurrent.ConcurrentSkipListMap.computeIfAbsent(ConcurrentSkipListMap.java:1682) at org.apache.beam.runners.dataflow.worker.MetricsContainerRegistry.getContainer(MetricsContainerRegistry.java:39) at org.apache.beam.runners.dataflow.worker.DataflowExecutionContext.createOperationContext(DataflowExecutionContext.java:212) at org.apache.beam.runners.dataflow.worker.SourceOperationExecutorFactory.create(SourceOperationExecutorFactory.java:49) at org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:376) at org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:314) at org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:140) at org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:120) at org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:107) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Apr 20, 2021 11:30:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-04-20T11:30:30.082Z: Finished operation ReadLines/Read+WordCount.CountWords/ParDo(ExtractWords)+WordCount.CountWords/Count.PerElement/Init/Map+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Partial+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Reify+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Write Apr 20, 2021 11:30:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process SEVERE: 2021-04-20T11:30:30.397Z: Workflow failed. Causes: S02:ReadLines/Read+WordCount.CountWords/ParDo(ExtractWords)+WordCount.CountWords/Count.PerElement/Init/Map+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Partial+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Reify+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Write failed., Internal Issue (7213968c576c929c): 63963027:24514 Apr 20, 2021 11:30:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-04-20T11:30:30.656Z: Cleaning up. Apr 20, 2021 11:30:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-04-20T11:30:30.911Z: Stopping worker pool... Apr 20, 2021 11:31:21 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-04-20T11:31:20.760Z: Autoscaling: Resized worker pool from 1 to 0. Apr 20, 2021 11:31:21 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-04-20T11:31:20.895Z: Worker pool stopped. Apr 20, 2021 11:31:31 AM org.apache.beam.runners.dataflow.DataflowPipelineJob logTerminalState INFO: Job 2021-04-20_04_28_07-11868786282907612053 failed with status FAILED. gsutil cat gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/count* | grep Montague: CommandException: No URLs matched: gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/count* CommandException: No URLs matched: gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/count* [ERROR] Failed command > Task :runners:google-cloud-dataflow-java:runQuickstartJavaDataflow FAILED FAILURE: Build completed with 3 failures. 1: Task failed with an exception. ----------- * What went wrong: Execution failed for task ':runners:google-cloud-dataflow-java:runMobileGamingJavaDataflow'. > Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with > non-zero exit value 1 * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. ============================================================================== 2: Task failed with an exception. ----------- * What went wrong: Execution failed for task ':runners:google-cloud-dataflow-java:runMobileGamingJavaDataflowBom'. > Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with > non-zero exit value 1 * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. ============================================================================== 3: Task failed with an exception. ----------- * What went wrong: Execution failed for task ':runners:google-cloud-dataflow-java:runQuickstartJavaDataflow'. > Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with > non-zero exit value 1 * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. ============================================================================== * Get more help at https://help.gradle.org Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/6.8.3/userguide/command_line_interface.html#sec:command_line_warnings BUILD FAILED in 31m 12s 9 actionable tasks: 8 executed, 1 from cache Publishing build scan... https://gradle.com/s/wd33r6tymwjcw Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
