See
<https://ci-beam.apache.org/job/beam_BiqQueryIO_Streaming_Performance_Test_Java/1190/display/redirect?page=changes>
Changes:
[Robert Bradshaw] Extend elementwise to generic proxy fn.
[Robert Bradshaw] Add a more methods to dataframes.
[Robert Bradshaw] Add support for dataframe scalar values.
[Robert Bradshaw] More dataframes methods and test filters.
[simonepri] Support creation of empty PCollection in beam CreateList
[simonepri] Update docstrings
[simonepri] Refactor CreateList
[simonepri] Refactor CreateList
[Robert Bradshaw] Reviewer comments.
[noreply] [BEAM-7390] Add latest code snippets (#12394)
[Robert Bradshaw] Skip dict-order-sensitive tests on 3.5.
[noreply] Add a blogpost for Apache Beam 2.23.0. (#12213)
[noreply] [BEAM-8543] Dataflow streaming timers are not strictly time ordered
when
[noreply] [BEAM-10595] Remove rules that not work well with Java UDF (#12400)
[noreply] Prepare a website update for 2.23.0 (#12212)
------------------------------------------
[...truncated 256.83 KB...]
Jul 30, 2020 12:34:12 AM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read from source as step s1
Jul 30, 2020 12:34:12 AM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Gather time as step s2
Jul 30, 2020 12:34:12 AM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Map records as step s3
Jul 30, 2020 12:34:12 AM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Write to BQ/PrepareWrite/ParDo(Anonymous) as step s4
Jul 30, 2020 12:34:12 AM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Write to BQ/StreamingInserts/CreateTables/ParDo(CreateTables)
as step s5
Jul 30, 2020 12:34:12 AM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Write to
BQ/StreamingInserts/StreamingWriteTables/ShardTableWrites as step s6
Jul 30, 2020 12:34:12 AM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Write to
BQ/StreamingInserts/StreamingWriteTables/TagWithUniqueIds as step s7
Jul 30, 2020 12:34:12 AM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Write to
BQ/StreamingInserts/StreamingWriteTables/Reshuffle/Window.Into()/Window.Assign
as step s8
Jul 30, 2020 12:34:12 AM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Write to
BQ/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey as step s9
Jul 30, 2020 12:34:12 AM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Write to
BQ/StreamingInserts/StreamingWriteTables/Reshuffle/ExpandIterable as step s10
Jul 30, 2020 12:34:12 AM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Write to
BQ/StreamingInserts/StreamingWriteTables/GlobalWindow/Window.Assign as step s11
Jul 30, 2020 12:34:12 AM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Write to
BQ/StreamingInserts/StreamingWriteTables/StreamingWrite as step s12
Jul 30, 2020 12:34:12 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to
gs://temp-storage-for-perf-tests/loadtests/staging/
Jul 30, 2020 12:34:12 AM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage
INFO: Uploading <104153 bytes, hash
afe88e07938b7a9e61c77f690429387d6d760bee76551aec7b98482682a6b2b3> to
gs://temp-storage-for-perf-tests/loadtests/staging/pipeline-r-iOB5OLep5hx39pBCk4fW12C-52VRrse5hIJoKmsrM.pb
Jul 30, 2020 12:34:13 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Dataflow SDK version: 2.24.0-SNAPSHOT
Jul 30, 2020 12:34:14 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to
https://console.cloud.google.com/dataflow/jobs/us-central1/2020-07-29_17_34_13-4484747166171184165?project=apache-beam-testing
Jul 30, 2020 12:34:14 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Submitted job: 2020-07-29_17_34_13-4484747166171184165
Jul 30, 2020 12:34:14 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel
--region=us-central1 2020-07-29_17_34_13-4484747166171184165
Jul 30, 2020 12:34:14 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
WARNING: 2020-07-30T00:34:13.104Z: The requested max number of ****s (5) is
ignored as autoscaling is explicitly disabled (autoscalingAlgorithm=NONE).
Jul 30, 2020 12:34:23 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-30T00:34:21.675Z: Worker configuration: n1-standard-1 in
us-central1-a.
Jul 30, 2020 12:34:23 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-30T00:34:22.478Z: Expanding CoGroupByKey operations into
optimizable parts.
Jul 30, 2020 12:34:23 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-30T00:34:22.556Z: Expanding GroupByKey operations into
optimizable parts.
Jul 30, 2020 12:34:23 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-30T00:34:22.575Z: Lifting ValueCombiningMappingFns into
MergeBucketsMappingFns
Jul 30, 2020 12:34:23 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-30T00:34:22.656Z: Fusing adjacent ParDo, Read, Write, and
Flatten operations
Jul 30, 2020 12:34:23 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-30T00:34:22.678Z: Fusing consumer Gather time into Read from
source
Jul 30, 2020 12:34:23 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-30T00:34:22.701Z: Fusing consumer Map records into Gather time
Jul 30, 2020 12:34:23 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-30T00:34:22.727Z: Fusing consumer Write to
BQ/PrepareWrite/ParDo(Anonymous) into Map records
Jul 30, 2020 12:34:23 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-30T00:34:22.750Z: Fusing consumer Write to
BQ/StreamingInserts/CreateTables/ParDo(CreateTables) into Write to
BQ/PrepareWrite/ParDo(Anonymous)
Jul 30, 2020 12:34:23 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-30T00:34:22.773Z: Fusing consumer Write to
BQ/StreamingInserts/StreamingWriteTables/ShardTableWrites into Write to
BQ/StreamingInserts/CreateTables/ParDo(CreateTables)
Jul 30, 2020 12:34:23 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-30T00:34:22.802Z: Fusing consumer Write to
BQ/StreamingInserts/StreamingWriteTables/TagWithUniqueIds into Write to
BQ/StreamingInserts/StreamingWriteTables/ShardTableWrites
Jul 30, 2020 12:34:23 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-30T00:34:22.822Z: Fusing consumer Write to
BQ/StreamingInserts/StreamingWriteTables/Reshuffle/Window.Into()/Window.Assign
into Write to BQ/StreamingInserts/StreamingWriteTables/TagWithUniqueIds
Jul 30, 2020 12:34:23 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-30T00:34:22.851Z: Fusing consumer Write to
BQ/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey/Reify into Write
to
BQ/StreamingInserts/StreamingWriteTables/Reshuffle/Window.Into()/Window.Assign
Jul 30, 2020 12:34:23 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-30T00:34:22.870Z: Fusing consumer Write to
BQ/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey/Write into Write
to BQ/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey/Reify
Jul 30, 2020 12:34:23 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-30T00:34:22.898Z: Fusing consumer Write to
BQ/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey/GroupByWindow
into Write to BQ/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey/Read
Jul 30, 2020 12:34:23 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-30T00:34:22.921Z: Fusing consumer Write to
BQ/StreamingInserts/StreamingWriteTables/Reshuffle/ExpandIterable into Write to
BQ/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey/GroupByWindow
Jul 30, 2020 12:34:23 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-30T00:34:22.945Z: Fusing consumer Write to
BQ/StreamingInserts/StreamingWriteTables/GlobalWindow/Window.Assign into Write
to BQ/StreamingInserts/StreamingWriteTables/Reshuffle/ExpandIterable
Jul 30, 2020 12:34:23 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-30T00:34:22.982Z: Fusing consumer Write to
BQ/StreamingInserts/StreamingWriteTables/StreamingWrite into Write to
BQ/StreamingInserts/StreamingWriteTables/GlobalWindow/Window.Assign
Jul 30, 2020 12:34:23 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-30T00:34:23.277Z: Executing operation Write to
BQ/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey/Create
Jul 30, 2020 12:34:23 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-30T00:34:23.341Z: Starting 5 ****s in us-central1-a...
Jul 30, 2020 12:34:23 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-30T00:34:23.380Z: Finished operation Write to
BQ/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey/Create
Jul 30, 2020 12:34:23 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-30T00:34:23.482Z: Executing operation Read from source+Gather
time+Map records+Write to BQ/PrepareWrite/ParDo(Anonymous)+Write to
BQ/StreamingInserts/CreateTables/ParDo(CreateTables)+Write to
BQ/StreamingInserts/StreamingWriteTables/ShardTableWrites+Write to
BQ/StreamingInserts/StreamingWriteTables/TagWithUniqueIds+Write to
BQ/StreamingInserts/StreamingWriteTables/Reshuffle/Window.Into()/Window.Assign+Write
to BQ/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey/Reify+Write
to BQ/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey/Write
Jul 30, 2020 12:34:51 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-30T00:34:51.579Z: Autoscaling: Raised the number of ****s to
4 based on the rate of progress in the currently running stage(s).
Jul 30, 2020 12:34:51 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-30T00:34:51.603Z: Resized **** pool to 4, though goal was 5.
This could be a quota issue.
Jul 30, 2020 12:34:58 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-30T00:34:56.988Z: Autoscaling: Raised the number of ****s to
5 based on the rate of progress in the currently running stage(s).
Jul 30, 2020 12:34:58 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
WARNING: 2020-07-30T00:34:57.381Z: Your project already contains 100
Dataflow-created metric descriptors and Stackdriver will not create new
Dataflow custom metrics for this job. Each unique user-defined metric name
(independent of the DoFn in which it is defined) produces a new metric
descriptor. To delete old / unused metric descriptors see
https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list
and
https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
Jul 30, 2020 12:35:12 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-30T00:35:12.425Z: Workers have started successfully.
Jul 30, 2020 12:35:12 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-30T00:35:12.462Z: Workers have started successfully.
Jul 30, 2020 12:37:48 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-30T00:37:48.608Z: Finished operation Read from source+Gather
time+Map records+Write to BQ/PrepareWrite/ParDo(Anonymous)+Write to
BQ/StreamingInserts/CreateTables/ParDo(CreateTables)+Write to
BQ/StreamingInserts/StreamingWriteTables/ShardTableWrites+Write to
BQ/StreamingInserts/StreamingWriteTables/TagWithUniqueIds+Write to
BQ/StreamingInserts/StreamingWriteTables/Reshuffle/Window.Into()/Window.Assign+Write
to BQ/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey/Reify+Write
to BQ/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey/Write
Jul 30, 2020 12:37:48 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-30T00:37:48.679Z: Executing operation Write to
BQ/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey/Close
Jul 30, 2020 12:37:48 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-30T00:37:48.735Z: Finished operation Write to
BQ/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey/Close
Jul 30, 2020 12:37:48 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2020-07-30T00:37:48.821Z: Executing operation Write to
BQ/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey/Read+Write to
BQ/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey/GroupByWindow+Write
to BQ/StreamingInserts/StreamingWriteTables/Reshuffle/ExpandIterable+Write to
BQ/StreamingInserts/StreamingWriteTables/GlobalWindow/Window.Assign+Write to
BQ/StreamingInserts/StreamingWriteTables/StreamingWrite
Jul 30, 2020 12:46:32 AM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
SEVERE: 2020-07-30T00:46:32.364Z: An OutOfMemoryException occurred.
Consider specifying higher memory instances in PipelineOptions.
java.lang.RuntimeException:
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.ExecutionError:
java.lang.OutOfMemoryError: Java heap space
at
org.apache.beam.runners.dataflow.****.util.common.****.BatchingShuffleEntryReader$ShuffleReadIterator.fillEntries(BatchingShuffleEntryReader.java:132)
at
org.apache.beam.runners.dataflow.****.util.common.****.BatchingShuffleEntryReader$ShuffleReadIterator.fillEntriesIfNeeded(BatchingShuffleEntryReader.java:119)
at
org.apache.beam.runners.dataflow.****.util.common.****.BatchingShuffleEntryReader$ShuffleReadIterator.hasNext(BatchingShuffleEntryReader.java:84)
at
org.apache.beam.runners.dataflow.****.util.common.ForwardingReiterator.hasNext(ForwardingReiterator.java:63)
at
org.apache.beam.runners.dataflow.****.util.common.****.GroupingShuffleEntryIterator$ValuesIterator.advance(GroupingShuffleEntryIterator.java:271)
at
org.apache.beam.runners.dataflow.****.util.common.****.GroupingShuffleEntryIterator$ValuesIterator.hasNext(GroupingShuffleEntryIterator.java:263)
at
org.apache.beam.runners.dataflow.****.GroupingShuffleReader$GroupingShuffleReaderIterator$ValuesIterator.hasNext(GroupingShuffleReader.java:397)
at
org.apache.beam.runners.dataflow.****.util.BatchGroupAlsoByWindowReshuffleFn.processElement(BatchGroupAlsoByWindowReshuffleFn.java:55)
at
org.apache.beam.runners.dataflow.****.util.BatchGroupAlsoByWindowReshuffleFn.processElement(BatchGroupAlsoByWindowReshuffleFn.java:39)
at
org.apache.beam.runners.dataflow.****.GroupAlsoByWindowFnRunner.invokeProcessElement(GroupAlsoByWindowFnRunner.java:121)
at
org.apache.beam.runners.dataflow.****.GroupAlsoByWindowFnRunner.processElement(GroupAlsoByWindowFnRunner.java:73)
at
org.apache.beam.runners.dataflow.****.GroupAlsoByWindowsParDoFn.processElement(GroupAlsoByWindowsParDoFn.java:114)
at
org.apache.beam.runners.dataflow.****.util.common.****.ParDoOperation.process(ParDoOperation.java:44)
at
org.apache.beam.runners.dataflow.****.util.common.****.OutputReceiver.process(OutputReceiver.java:49)
at
org.apache.beam.runners.dataflow.****.util.common.****.ReadOperation.runReadLoop(ReadOperation.java:201)
at
org.apache.beam.runners.dataflow.****.util.common.****.ReadOperation.start(ReadOperation.java:159)
at
org.apache.beam.runners.dataflow.****.util.common.****.MapTaskExecutor.execute(MapTaskExecutor.java:77)
at
org.apache.beam.runners.dataflow.****.BatchDataflowWorker.executeWork(BatchDataflowWorker.java:417)
at
org.apache.beam.runners.dataflow.****.BatchDataflowWorker.doWork(BatchDataflowWorker.java:386)
at
org.apache.beam.runners.dataflow.****.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:311)
at
org.apache.beam.runners.dataflow.****.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:140)
at
org.apache.beam.runners.dataflow.****.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:120)
at
org.apache.beam.runners.dataflow.****.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:107)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by:
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.ExecutionError:
java.lang.OutOfMemoryError: Java heap space
at
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2048)
at
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache.get(LocalCache.java:3952)
at
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974)
at
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4958)
at
org.apache.beam.runners.dataflow.****.util.common.****.CachingShuffleBatchReader.read(CachingShuffleBatchReader.java:74)
at
org.apache.beam.runners.dataflow.****.util.common.****.BatchingShuffleEntryReader$ShuffleReadIterator.fillEntries(BatchingShuffleEntryReader.java:125)
... 26 more
Caused by: java.lang.OutOfMemoryError: Java heap space
at
org.apache.beam.runners.dataflow.****.ApplianceShuffleReader.readIncludingPosition(Native
Method)
at
org.apache.beam.runners.dataflow.****.ChunkingShuffleBatchReader.read(ChunkingShuffleBatchReader.java:58)
at
org.apache.beam.runners.dataflow.****.util.common.****.CachingShuffleBatchReader$1.load(CachingShuffleBatchReader.java:55)
at
org.apache.beam.runners.dataflow.****.util.common.****.CachingShuffleBatchReader$1.load(CachingShuffleBatchReader.java:51)
at
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3528)
at
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2277)
at
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2154)
at
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2044)
at
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache.get(LocalCache.java:3952)
at
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974)
at
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4958)
at
org.apache.beam.runners.dataflow.****.util.common.****.CachingShuffleBatchReader.read(CachingShuffleBatchReader.java:74)
at
org.apache.beam.runners.dataflow.****.util.common.****.BatchingShuffleEntryReader$ShuffleReadIterator.fillEntries(BatchingShuffleEntryReader.java:125)
at
org.apache.beam.runners.dataflow.****.util.common.****.BatchingShuffleEntryReader$ShuffleReadIterator.fillEntriesIfNeeded(BatchingShuffleEntryReader.java:119)
at
org.apache.beam.runners.dataflow.****.util.common.****.BatchingShuffleEntryReader$ShuffleReadIterator.hasNext(BatchingShuffleEntryReader.java:84)
at
org.apache.beam.runners.dataflow.****.util.common.ForwardingReiterator.hasNext(ForwardingReiterator.java:63)
at
org.apache.beam.runners.dataflow.****.util.common.****.GroupingShuffleEntryIterator$ValuesIterator.advance(GroupingShuffleEntryIterator.java:271)
at
org.apache.beam.runners.dataflow.****.util.common.****.GroupingShuffleEntryIterator$ValuesIterator.hasNext(GroupingShuffleEntryIterator.java:263)
at
org.apache.beam.runners.dataflow.****.GroupingShuffleReader$GroupingShuffleReaderIterator$ValuesIterator.hasNext(GroupingShuffleReader.java:397)
at
org.apache.beam.runners.dataflow.****.util.BatchGroupAlsoByWindowReshuffleFn.processElement(BatchGroupAlsoByWindowReshuffleFn.java:55)
at
org.apache.beam.runners.dataflow.****.util.BatchGroupAlsoByWindowReshuffleFn.processElement(BatchGroupAlsoByWindowReshuffleFn.java:39)
at
org.apache.beam.runners.dataflow.****.GroupAlsoByWindowFnRunner.invokeProcessElement(GroupAlsoByWindowFnRunner.java:121)
at
org.apache.beam.runners.dataflow.****.GroupAlsoByWindowFnRunner.processElement(GroupAlsoByWindowFnRunner.java:73)
at
org.apache.beam.runners.dataflow.****.GroupAlsoByWindowsParDoFn.processElement(GroupAlsoByWindowsParDoFn.java:114)
at
org.apache.beam.runners.dataflow.****.util.common.****.ParDoOperation.process(ParDoOperation.java:44)
at
org.apache.beam.runners.dataflow.****.util.common.****.OutputReceiver.process(OutputReceiver.java:49)
at
org.apache.beam.runners.dataflow.****.util.common.****.ReadOperation.runReadLoop(ReadOperation.java:201)
at
org.apache.beam.runners.dataflow.****.util.common.****.ReadOperation.start(ReadOperation.java:159)
at
org.apache.beam.runners.dataflow.****.util.common.****.MapTaskExecutor.execute(MapTaskExecutor.java:77)
at
org.apache.beam.runners.dataflow.****.BatchDataflowWorker.executeWork(BatchDataflowWorker.java:417)
at
org.apache.beam.runners.dataflow.****.BatchDataflowWorker.doWork(BatchDataflowWorker.java:386)
at
org.apache.beam.runners.dataflow.****.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:311)
Jul 30, 2020 12:46:38 AM
org.apache.beam.runners.dataflow.DataflowPipelineJob lambda$waitUntilFinish$0
WARNING: Job is already running in Google Cloud Platform, Ctrl-C will not
cancel it.
To cancel the job in the cloud, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel
--region=us-central1 2020-07-29_17_34_13-4484747166171184165
org.apache.beam.sdk.bigqueryioperftests.BigQueryIOIT > testWriteThenRead SKIPPED
> Task :sdks:java:io:bigquery-io-perf-tests:integrationTest FAILED
:sdks:java:io:bigquery-io-perf-tests:integrationTest (Thread[Execution **** for
':' Thread 3,5,main]) completed. Took 12 mins 34.718 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task
':sdks:java:io:bigquery-io-perf-tests:integrationTest'.
> Process 'Gradle Test Executor 8' finished with non-zero exit value 143
This problem might be caused by incorrect test process configuration.
Please refer to the test execution section in the User Manual at
https://docs.gradle.org/5.2.1/userguide/java_testing.html#sec:test_execution
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to
get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with
Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See
https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 13m 22s
84 actionable tasks: 57 executed, 27 from cache
Publishing build scan...
https://gradle.com/s/euiand5333bys
Unexpected exception thrown.
org.gradle.internal.remote.internal.MessageIOException: Could not write
'/0:0:0:0:0:0:0:1:56534'.
at
org.gradle.internal.remote.internal.inet.SocketConnection.flush(SocketConnection.java:135)
at
org.gradle.internal.remote.internal.hub.MessageHub$ConnectionDispatch.run(MessageHub.java:325)
at
org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:63)
at
org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:46)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at
org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:55)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
at sun.nio.ch.IOUtil.write(IOUtil.java:51)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:468)
at
org.gradle.internal.remote.internal.inet.SocketConnection$SocketOutputStream.writeWithNonBlockingRetry(SocketConnection.java:273)
at
org.gradle.internal.remote.internal.inet.SocketConnection$SocketOutputStream.writeBufferToChannel(SocketConnection.java:261)
at
org.gradle.internal.remote.internal.inet.SocketConnection$SocketOutputStream.flush(SocketConnection.java:255)
at
org.gradle.internal.remote.internal.inet.SocketConnection.flush(SocketConnection.java:133)
... 7 more
FAILURE: Build failed with an exception.
* What went wrong:
Not all **** daemon(s) could be stopped.
> Process 'Gradle Worker Daemon 6' finished with non-zero exit value 143
> Process 'Gradle Worker Daemon 7' finished with non-zero exit value 143
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to
get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with
Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See
https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 13m 23s
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
