See
<https://ci-beam.apache.org/job/beam_BiqQueryIO_Batch_Performance_Test_Java_Avro/2795/display/redirect?page=changes>
Changes:
[aydar.zaynutdinov] [BEAM-13178][Playground]
[noreply] [BEAM-3293] Finish E2E implementation of MultiMap side inputs, add
[noreply] Merge pull request #15740 from [BEAM-12936][Playground] Code editor -
[noreply] Merge pull request #15976 from [Playground][BEAM-12941][Bugfix] Fix
[noreply] Merge pull request #15991 [BEAM-13251][Playground] [Bugfix] Lint Fails
------------------------------------------
[...truncated 371.41 KB...]
Nov 16, 2021 6:33:43 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-16T18:33:40.778Z: Finished operation Write to
BQ/BatchLoads/ReifyResults/Create.Values/Read(CreateSource)+Write to
BQ/BatchLoads/ReifyResults/ParDo(Anonymous)+Write to
BQ/BatchLoads/WritePartitionUntriggered+Write to
BQ/BatchLoads/MultiPartitionsReshuffle/Window.Into()/Window.Assign+Write to
BQ/BatchLoads/SinglePartitionsReshuffle/Window.Into()/Window.Assign+Write to
BQ/BatchLoads/MultiPartitionsReshuffle/GroupByKey/Reify+Write to
BQ/BatchLoads/MultiPartitionsReshuffle/GroupByKey/Write+Write to
BQ/BatchLoads/SinglePartitionsReshuffle/GroupByKey/Reify+Write to
BQ/BatchLoads/SinglePartitionsReshuffle/GroupByKey/Write
Nov 16, 2021 6:33:43 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-16T18:33:40.887Z: Executing operation Write to
BQ/BatchLoads/SinglePartitionsReshuffle/GroupByKey/Close
Nov 16, 2021 6:33:43 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-16T18:33:40.920Z: Executing operation Write to
BQ/BatchLoads/MultiPartitionsReshuffle/GroupByKey/Close
Nov 16, 2021 6:33:43 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-16T18:33:40.987Z: Finished operation Write to
BQ/BatchLoads/MultiPartitionsReshuffle/GroupByKey/Close
Nov 16, 2021 6:33:43 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-16T18:33:41.053Z: Executing operation Write to
BQ/BatchLoads/MultiPartitionsWriteTables/GroupByKey/Create
Nov 16, 2021 6:33:43 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-16T18:33:41.161Z: Finished operation Write to
BQ/BatchLoads/MultiPartitionsWriteTables/GroupByKey/Create
Nov 16, 2021 6:33:43 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-16T18:33:41.315Z: Executing operation Write to
BQ/BatchLoads/MultiPartitionsReshuffle/GroupByKey/Read+Write to
BQ/BatchLoads/MultiPartitionsReshuffle/GroupByKey/GroupByWindow+Write to
BQ/BatchLoads/MultiPartitionsReshuffle/ExpandIterable+Write to
BQ/BatchLoads/MultiPartitionsWriteTables/ParMultiDo(WriteTables)+Write to
BQ/BatchLoads/MultiPartitionsWriteTables/WithKeys/AddKeys/Map+Write to
BQ/BatchLoads/ReifyRenameInput/View.AsIterable/ParDo(ToIsmRecordForGlobalWindow)+Write
to BQ/BatchLoads/MultiPartitionsWriteTables/Window.Into()/Window.Assign+Write
to BQ/BatchLoads/MultiPartitionsWriteTables/GroupByKey/Reify+Write to
BQ/BatchLoads/MultiPartitionsWriteTables/GroupByKey/Write
Nov 16, 2021 6:33:43 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-16T18:33:41.725Z: Finished operation Write to
BQ/BatchLoads/SinglePartitionsReshuffle/GroupByKey/Close
Nov 16, 2021 6:33:43 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-16T18:33:41.807Z: Executing operation Write to
BQ/BatchLoads/SinglePartitionWriteTables/GroupByKey/Create
Nov 16, 2021 6:33:43 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-16T18:33:41.934Z: Finished operation Write to
BQ/BatchLoads/SinglePartitionWriteTables/GroupByKey/Create
Nov 16, 2021 6:33:43 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-16T18:33:42.092Z: Executing operation Write to
BQ/BatchLoads/SinglePartitionsReshuffle/GroupByKey/Read+Write to
BQ/BatchLoads/SinglePartitionsReshuffle/GroupByKey/GroupByWindow+Write to
BQ/BatchLoads/SinglePartitionsReshuffle/ExpandIterable+Write to
BQ/BatchLoads/SinglePartitionWriteTables/ParMultiDo(WriteTables)+Write to
BQ/BatchLoads/SinglePartitionWriteTables/WithKeys/AddKeys/Map+Write to
BQ/BatchLoads/SinglePartitionWriteTables/Window.Into()/Window.Assign+Write to
BQ/BatchLoads/SinglePartitionWriteTables/GroupByKey/Reify+Write to
BQ/BatchLoads/SinglePartitionWriteTables/GroupByKey/Write
Nov 16, 2021 6:33:43 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-16T18:33:42.417Z: Finished operation Write to
BQ/BatchLoads/MultiPartitionsReshuffle/GroupByKey/Read+Write to
BQ/BatchLoads/MultiPartitionsReshuffle/GroupByKey/GroupByWindow+Write to
BQ/BatchLoads/MultiPartitionsReshuffle/ExpandIterable+Write to
BQ/BatchLoads/MultiPartitionsWriteTables/ParMultiDo(WriteTables)+Write to
BQ/BatchLoads/MultiPartitionsWriteTables/WithKeys/AddKeys/Map+Write to
BQ/BatchLoads/ReifyRenameInput/View.AsIterable/ParDo(ToIsmRecordForGlobalWindow)+Write
to BQ/BatchLoads/MultiPartitionsWriteTables/Window.Into()/Window.Assign+Write
to BQ/BatchLoads/MultiPartitionsWriteTables/GroupByKey/Reify+Write to
BQ/BatchLoads/MultiPartitionsWriteTables/GroupByKey/Write
Nov 16, 2021 6:33:43 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-16T18:33:42.533Z: Executing operation Write to
BQ/BatchLoads/MultiPartitionsWriteTables/GroupByKey/Close
Nov 16, 2021 6:33:43 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-16T18:33:42.572Z: Executing operation Write to
BQ/BatchLoads/ReifyRenameInput/View.AsIterable/CreateDataflowView
Nov 16, 2021 6:33:43 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-16T18:33:42.597Z: Finished operation Write to
BQ/BatchLoads/MultiPartitionsWriteTables/GroupByKey/Close
Nov 16, 2021 6:33:43 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-16T18:33:42.611Z: Finished operation Write to
BQ/BatchLoads/ReifyRenameInput/View.AsIterable/CreateDataflowView
Nov 16, 2021 6:33:43 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-16T18:33:42.666Z: Executing operation Write to
BQ/BatchLoads/MultiPartitionsWriteTables/GroupByKey/Read+Write to
BQ/BatchLoads/MultiPartitionsWriteTables/GroupByKey/GroupByWindow+Write to
BQ/BatchLoads/MultiPartitionsWriteTables/Values/Values/Map+Write to
BQ/BatchLoads/MultiPartitionsWriteTables/ParDo(GarbageCollectTemporaryFiles)
Nov 16, 2021 6:33:43 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-16T18:33:42.807Z: Executing operation Write to
BQ/BatchLoads/ReifyRenameInput/Create.Values/Read(CreateSource)+Write to
BQ/BatchLoads/ReifyRenameInput/ParDo(Anonymous)+Write to
BQ/BatchLoads/WriteRenameUntriggered
Nov 16, 2021 6:33:43 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-16T18:33:42.983Z: Finished operation Write to
BQ/BatchLoads/MultiPartitionsWriteTables/GroupByKey/Read+Write to
BQ/BatchLoads/MultiPartitionsWriteTables/GroupByKey/GroupByWindow+Write to
BQ/BatchLoads/MultiPartitionsWriteTables/Values/Values/Map+Write to
BQ/BatchLoads/MultiPartitionsWriteTables/ParDo(GarbageCollectTemporaryFiles)
Nov 16, 2021 6:33:45 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-16T18:33:43.923Z: Finished operation Write to
BQ/BatchLoads/ReifyRenameInput/Create.Values/Read(CreateSource)+Write to
BQ/BatchLoads/ReifyRenameInput/ParDo(Anonymous)+Write to
BQ/BatchLoads/WriteRenameUntriggered
Nov 16, 2021 6:34:04 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-16T18:34:03.106Z: Finished operation Write to
BQ/BatchLoads/SinglePartitionsReshuffle/GroupByKey/Read+Write to
BQ/BatchLoads/SinglePartitionsReshuffle/GroupByKey/GroupByWindow+Write to
BQ/BatchLoads/SinglePartitionsReshuffle/ExpandIterable+Write to
BQ/BatchLoads/SinglePartitionWriteTables/ParMultiDo(WriteTables)+Write to
BQ/BatchLoads/SinglePartitionWriteTables/WithKeys/AddKeys/Map+Write to
BQ/BatchLoads/SinglePartitionWriteTables/Window.Into()/Window.Assign+Write to
BQ/BatchLoads/SinglePartitionWriteTables/GroupByKey/Reify+Write to
BQ/BatchLoads/SinglePartitionWriteTables/GroupByKey/Write
Nov 16, 2021 6:34:04 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-16T18:34:03.171Z: Executing operation Write to
BQ/BatchLoads/SinglePartitionWriteTables/GroupByKey/Close
Nov 16, 2021 6:34:04 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-16T18:34:03.280Z: Finished operation Write to
BQ/BatchLoads/SinglePartitionWriteTables/GroupByKey/Close
Nov 16, 2021 6:34:04 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-16T18:34:03.341Z: Executing operation Write to
BQ/BatchLoads/SinglePartitionWriteTables/GroupByKey/Read+Write to
BQ/BatchLoads/SinglePartitionWriteTables/GroupByKey/GroupByWindow+Write to
BQ/BatchLoads/SinglePartitionWriteTables/Values/Values/Map+Write to
BQ/BatchLoads/SinglePartitionWriteTables/ParDo(GarbageCollectTemporaryFiles)
Nov 16, 2021 6:34:04 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-16T18:34:03.774Z: Finished operation Write to
BQ/BatchLoads/SinglePartitionWriteTables/GroupByKey/Read+Write to
BQ/BatchLoads/SinglePartitionWriteTables/GroupByKey/GroupByWindow+Write to
BQ/BatchLoads/SinglePartitionWriteTables/Values/Values/Map+Write to
BQ/BatchLoads/SinglePartitionWriteTables/ParDo(GarbageCollectTemporaryFiles)
Nov 16, 2021 6:34:04 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-16T18:34:03.967Z: Cleaning up.
Nov 16, 2021 6:34:04 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-16T18:34:04.056Z: Stopping **** pool...
Nov 16, 2021 6:36:29 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-16T18:36:29.109Z: Autoscaling: Resized **** pool from 5 to 0.
Nov 16, 2021 6:36:29 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-16T18:36:29.148Z: Worker pool stopped.
Nov 16, 2021 6:37:07 PM
org.apache.beam.runners.dataflow.DataflowPipelineJob logTerminalState
INFO: Job 2021-11-16_10_30_58-14716132195125721815 finished with status
DONE.
org.apache.beam.sdk.bigqueryioperftests.BigQueryIOIT > testWriteThenRead
STANDARD_OUT
Load test results for test (ID): 570fb3ea-1c8f-4ec0-88e4-ef9c6b7d7e72 and
timestamp: 2021-11-16T18:30:46.989000000Z:
Metric: Value:
avro_write_time 37.163
org.apache.beam.sdk.bigqueryioperftests.BigQueryIOIT > testWriteThenRead
STANDARD_ERROR
Nov 16, 2021 6:37:08 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Executing pipeline on the Dataflow Service, which will have billing
implications related to Google Compute Engine usage and other Google Cloud
Services.
Nov 16, 2021 6:37:09 PM org.apache.beam.runners.dataflow.util.PackageUtil
stageClasspathElements
INFO: Uploading 225 files from PipelineOptions.filesToStage to staging
location to prepare for execution.
Nov 16, 2021 6:37:09 PM
org.apache.beam.runners.dataflow.util.PackageUtil$PackageAttributes
forFileToStage
INFO: Staging custom dataflow-****.jar as
beam-runners-google-cloud-dataflow-java-legacy-****-2.35.0-SNAPSHOT-CxDB2MFbq9vTw9dZ8Zb10JhAyMvZWoQKUaKm85AO368.jar
Nov 16, 2021 6:37:10 PM org.apache.beam.runners.dataflow.util.PackageUtil
stageClasspathElements
INFO: Staging files complete: 225 files cached, 0 files newly uploaded in 0
seconds
Nov 16, 2021 6:37:10 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging portable pipeline proto to
gs://temp-storage-for-perf-tests/loadtests/staging/
Nov 16, 2021 6:37:10 PM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage
INFO: Uploading <145707 bytes, hash
e620af6000660a99fb09c9ee95997061f1de37bb5a4792f3eb54f7b0adbb8aa1> to
gs://temp-storage-for-perf-tests/loadtests/staging/pipeline-5iCvYABmCpn7CcnulZlwYfHeN7taR5Lz61T3sK27iqE.pb
Nov 16, 2021 6:37:11 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read from BQ/TriggerIdCreation/Read(CreateSource) as step s1
Nov 16, 2021 6:37:11 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read from
BQ/ViewId/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map
as step s2
Nov 16, 2021 6:37:11 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read from
BQ/ViewId/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey
as step s3
Nov 16, 2021 6:37:11 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read from
BQ/ViewId/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues
as step s4
Nov 16, 2021 6:37:11 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read from
BQ/ViewId/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map
as step s5
Nov 16, 2021 6:37:11 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read from
BQ/ViewId/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
as step s6
Nov 16, 2021 6:37:11 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read from
BQ/ViewId/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly
as step s7
Nov 16, 2021 6:37:11 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read from
BQ/ViewId/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow)
as step s8
Nov 16, 2021 6:37:11 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read from
BQ/ViewId/Combine.GloballyAsSingletonView/CreateDataflowView as step s9
Nov 16, 2021 6:37:11 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read from BQ/Read(BigQueryTableSource) as step s10
Nov 16, 2021 6:37:11 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read from BQ/PassThroughThenCleanup/ParMultiDo(Identity) as
step s11
Nov 16, 2021 6:37:11 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read from
BQ/PassThroughThenCleanup/View.AsIterable/ParDo(ToIsmRecordForGlobalWindow) as
step s12
Nov 16, 2021 6:37:11 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read from
BQ/PassThroughThenCleanup/View.AsIterable/CreateDataflowView as step s13
Nov 16, 2021 6:37:11 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read from
BQ/PassThroughThenCleanup/Create(CleanupOperation)/Read(CreateSource) as step
s14
Nov 16, 2021 6:37:11 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read from BQ/PassThroughThenCleanup/Cleanup as step s15
Nov 16, 2021 6:37:11 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Gather time as step s16
Nov 16, 2021 6:37:11 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Dataflow SDK version: 2.35.0-SNAPSHOT
Nov 16, 2021 6:37:12 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-11-16_10_37_11-958810606441382474?project=apache-beam-testing
Nov 16, 2021 6:37:12 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Submitted job: 2021-11-16_10_37_11-958810606441382474
Nov 16, 2021 6:37:12 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel
--region=us-central1 2021-11-16_10_37_11-958810606441382474
Nov 16, 2021 6:37:15 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
WARNING: 2021-11-16T18:37:14.938Z: The requested max number of ****s (5) is
ignored as autoscaling is explicitly disabled (autoscalingAlgorithm=NONE).
Nov 16, 2021 6:37:23 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-16T18:37:21.318Z: Worker configuration: e2-standard-2 in
us-central1-a.
Nov 16, 2021 6:37:23 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-16T18:37:22.023Z: Expanding CoGroupByKey operations into
optimizable parts.
Nov 16, 2021 6:37:23 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-16T18:37:22.120Z: Expanding GroupByKey operations into
optimizable parts.
Nov 16, 2021 6:37:23 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-16T18:37:22.150Z: Lifting ValueCombiningMappingFns into
MergeBucketsMappingFns
Nov 16, 2021 6:37:23 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-16T18:37:22.241Z: Fusing adjacent ParDo, Read, Write, and
Flatten operations
Nov 16, 2021 6:37:23 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-16T18:37:22.270Z: Fusing consumer Read from
BQ/PassThroughThenCleanup/ParMultiDo(Identity) into Read from
BQ/Read(BigQueryTableSource)
Nov 16, 2021 6:37:23 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-16T18:37:22.307Z: Fusing consumer Gather time into Read from
BQ/PassThroughThenCleanup/ParMultiDo(Identity)
Nov 16, 2021 6:37:23 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-16T18:37:22.340Z: Fusing consumer Read from
BQ/PassThroughThenCleanup/View.AsIterable/ParDo(ToIsmRecordForGlobalWindow)
into Read from BQ/PassThroughThenCleanup/ParMultiDo(Identity)
Nov 16, 2021 6:37:23 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-16T18:37:22.379Z: Fusing consumer Read from
BQ/PassThroughThenCleanup/Cleanup into Read from
BQ/PassThroughThenCleanup/Create(CleanupOperation)/Read(CreateSource)
Nov 16, 2021 6:37:23 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-16T18:37:22.411Z: Fusing consumer Read from
BQ/ViewId/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map
into Read from BQ/TriggerIdCreation/Read(CreateSource)
Nov 16, 2021 6:37:23 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-16T18:37:22.442Z: Fusing consumer Read from
BQ/ViewId/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey+Read
from
BQ/ViewId/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Partial
into Read from
BQ/ViewId/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map
Nov 16, 2021 6:37:23 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-16T18:37:22.467Z: Fusing consumer Read from
BQ/ViewId/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Reify
into Read from
BQ/ViewId/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey+Read
from
BQ/ViewId/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Partial
Nov 16, 2021 6:37:23 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-16T18:37:22.500Z: Fusing consumer Read from
BQ/ViewId/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Write
into Read from
BQ/ViewId/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Reify
Nov 16, 2021 6:37:23 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-16T18:37:22.528Z: Fusing consumer Read from
BQ/ViewId/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues
into Read from
BQ/ViewId/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Read
Nov 16, 2021 6:37:23 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-16T18:37:22.557Z: Fusing consumer Read from
BQ/ViewId/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Extract
into Read from
BQ/ViewId/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues
Nov 16, 2021 6:37:23 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-16T18:37:22.582Z: Fusing consumer Read from
BQ/ViewId/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map
into Read from
BQ/ViewId/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Extract
Nov 16, 2021 6:37:23 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-16T18:37:22.616Z: Fusing consumer Read from
BQ/ViewId/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
into Read from
BQ/ViewId/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map
Nov 16, 2021 6:37:23 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-16T18:37:22.636Z: Fusing consumer Read from
BQ/ViewId/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
into Read from
BQ/ViewId/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
Nov 16, 2021 6:37:23 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-16T18:37:22.676Z: Fusing consumer Read from
BQ/ViewId/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow)
into Read from
BQ/ViewId/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
Nov 16, 2021 6:37:23 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-16T18:37:23.019Z: Executing operation Read from
BQ/Read(BigQueryTableSource)+Read from
BQ/PassThroughThenCleanup/ParMultiDo(Identity)+Gather time+Read from
BQ/PassThroughThenCleanup/View.AsIterable/ParDo(ToIsmRecordForGlobalWindow)
Nov 16, 2021 6:37:23 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-16T18:37:23.048Z: Executing operation Read from
BQ/ViewId/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Create
Nov 16, 2021 6:37:23 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-16T18:37:23.083Z: Starting 5 ****s in us-central1-a...
Nov 16, 2021 6:37:24 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-16T18:37:23.504Z: Finished operation Read from
BQ/ViewId/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Create
Nov 16, 2021 6:37:24 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-16T18:37:23.620Z: Executing operation Read from
BQ/TriggerIdCreation/Read(CreateSource)+Read from
BQ/ViewId/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map+Read
from
BQ/ViewId/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey+Read
from
BQ/ViewId/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Partial+Read
from
BQ/ViewId/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Reify+Read
from
BQ/ViewId/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Write
Nov 16, 2021 6:37:47 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-16T18:37:46.340Z: Your project already contains 100
Dataflow-created metric descriptors, so new user metrics of the form
custom.googleapis.com/* will not be created. However, all user metrics are also
available in the metric dataflow.googleapis.com/job/user_counter. If you rely
on the custom metrics, you can delete old / unused metric descriptors. See
https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list
and
https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
Nov 16, 2021 6:37:59 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-16T18:37:58.932Z: Autoscaling: Raised the number of ****s to
1 based on the rate of progress in the currently running stage(s).
Nov 16, 2021 6:37:59 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-16T18:37:58.965Z: Resized **** pool to 1, though goal was 5.
This could be a quota issue.
Nov 16, 2021 6:38:10 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-16T18:38:09.260Z: Autoscaling: Raised the number of ****s to
5 based on the rate of progress in the currently running stage(s).
Nov 16, 2021 6:38:13 PM
org.apache.beam.runners.dataflow.DataflowPipelineJob lambda$waitUntilFinish$0
WARNING: Job is already running in Google Cloud Platform, Ctrl-C will not
cancel it.
To cancel the job in the cloud, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel
--region=us-central1 2021-11-16_10_37_11-958810606441382474
org.apache.beam.sdk.bigqueryioperftests.BigQueryIOIT > testWriteThenRead SKIPPED
> Task :sdks:java:io:bigquery-io-perf-tests:integrationTest FAILED
:sdks:java:io:bigquery-io-perf-tests:integrationTest (Thread[Execution **** for
':' Thread 7,5,main]) completed. Took 7 mins 28.963 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task
':sdks:java:io:bigquery-io-perf-tests:integrationTest'.
> Process 'Gradle Test Executor 1' finished with non-zero exit value 143
This problem might be caused by incorrect test process configuration.
Please refer to the test execution section in the User Manual at
https://docs.gradle.org/6.9.1/userguide/java_testing.html#sec:test_execution
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to
get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with
Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See
https://docs.gradle.org/6.9.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 7m 53s
118 actionable tasks: 72 executed, 46 from cache
Publishing build scan...
The message received from the daemon indicates that the daemon has disappeared.
Build request sent: Build{id=363cd7ae-17cf-49d5-bc3c-f0d61af708bb,
currentDir=<https://ci-beam.apache.org/job/beam_BiqQueryIO_Batch_Performance_Test_Java_Avro/ws/src}>
Attempting to read last messages from the daemon log...
Daemon pid: 24728
log file: /home/jenkins/.gradle/daemon/6.9.1/daemon-24728.out.log
----- Last 20 lines from daemon log file - daemon-24728.out.log -----
* What went wrong:
Execution failed for task
':sdks:java:io:bigquery-io-perf-tests:integrationTest'.
> Process 'Gradle Test Executor 1' finished with non-zero exit value 143
This problem might be caused by incorrect test process configuration.
Please refer to the test execution section in the User Manual at
https://docs.gradle.org/6.9.1/userguide/java_testing.html#sec:test_execution
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to
get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with
Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See
https://docs.gradle.org/6.9.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 7m 53s
118 actionable tasks: 72 executed, 46 from cache
Publishing build scan...
Daemon vm is shutting down... The daemon has exited normally or was terminated
in response to a user interrupt.
----- End of the daemon log -----
FAILURE: Build failed with an exception.
* What went wrong:
Gradle build daemon disappeared unexpectedly (it may have been killed or may
have crashed)
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to
get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
