See
<https://ci-beam.apache.org/job/beam_BiqQueryIO_Batch_Performance_Test_Java_Json/2789/display/redirect?page=changes>
Changes:
[noreply] [BEAM-13226] Batch metrics parsing errors (#15972)
[noreply] [BEAM-13239] pass through io.EOF when decoding row (#15971)
[noreply] [BEAM-5378] Register types & funcs for examples. (#15970)
------------------------------------------
[...truncated 366.69 KB...]
Nov 15, 2021 6:51:42 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-15T18:51:41.244Z: Finished operation Write to
BQ/BatchLoads/MultiPartitionsReshuffle/GroupByKey/Create
Nov 15, 2021 6:51:42 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-15T18:51:41.409Z: Executing operation Write to
BQ/BatchLoads/ReifyResults/Create.Values/Read(CreateSource)+Write to
BQ/BatchLoads/ReifyResults/ParDo(Anonymous)+Write to
BQ/BatchLoads/WritePartitionUntriggered+Write to
BQ/BatchLoads/MultiPartitionsReshuffle/Window.Into()/Window.Assign+Write to
BQ/BatchLoads/SinglePartitionsReshuffle/Window.Into()/Window.Assign+Write to
BQ/BatchLoads/MultiPartitionsReshuffle/GroupByKey/Reify+Write to
BQ/BatchLoads/MultiPartitionsReshuffle/GroupByKey/Write+Write to
BQ/BatchLoads/SinglePartitionsReshuffle/GroupByKey/Reify+Write to
BQ/BatchLoads/SinglePartitionsReshuffle/GroupByKey/Write
Nov 15, 2021 6:51:43 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-15T18:51:42.129Z: Finished operation Write to
BQ/BatchLoads/ReifyResults/Create.Values/Read(CreateSource)+Write to
BQ/BatchLoads/ReifyResults/ParDo(Anonymous)+Write to
BQ/BatchLoads/WritePartitionUntriggered+Write to
BQ/BatchLoads/MultiPartitionsReshuffle/Window.Into()/Window.Assign+Write to
BQ/BatchLoads/SinglePartitionsReshuffle/Window.Into()/Window.Assign+Write to
BQ/BatchLoads/MultiPartitionsReshuffle/GroupByKey/Reify+Write to
BQ/BatchLoads/MultiPartitionsReshuffle/GroupByKey/Write+Write to
BQ/BatchLoads/SinglePartitionsReshuffle/GroupByKey/Reify+Write to
BQ/BatchLoads/SinglePartitionsReshuffle/GroupByKey/Write
Nov 15, 2021 6:51:43 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-15T18:51:42.187Z: Executing operation Write to
BQ/BatchLoads/SinglePartitionsReshuffle/GroupByKey/Close
Nov 15, 2021 6:51:43 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-15T18:51:42.216Z: Executing operation Write to
BQ/BatchLoads/MultiPartitionsReshuffle/GroupByKey/Close
Nov 15, 2021 6:51:43 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-15T18:51:42.273Z: Finished operation Write to
BQ/BatchLoads/MultiPartitionsReshuffle/GroupByKey/Close
Nov 15, 2021 6:51:43 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-15T18:51:42.358Z: Executing operation Write to
BQ/BatchLoads/MultiPartitionsWriteTables/GroupByKey/Create
Nov 15, 2021 6:51:43 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-15T18:51:42.576Z: Finished operation Write to
BQ/BatchLoads/MultiPartitionsWriteTables/GroupByKey/Create
Nov 15, 2021 6:51:43 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-15T18:51:42.718Z: Executing operation Write to
BQ/BatchLoads/MultiPartitionsReshuffle/GroupByKey/Read+Write to
BQ/BatchLoads/MultiPartitionsReshuffle/GroupByKey/GroupByWindow+Write to
BQ/BatchLoads/MultiPartitionsReshuffle/ExpandIterable+Write to
BQ/BatchLoads/MultiPartitionsWriteTables/ParMultiDo(WriteTables)+Write to
BQ/BatchLoads/MultiPartitionsWriteTables/WithKeys/AddKeys/Map+Write to
BQ/BatchLoads/ReifyRenameInput/View.AsIterable/ParDo(ToIsmRecordForGlobalWindow)+Write
to BQ/BatchLoads/MultiPartitionsWriteTables/Window.Into()/Window.Assign+Write
to BQ/BatchLoads/MultiPartitionsWriteTables/GroupByKey/Reify+Write to
BQ/BatchLoads/MultiPartitionsWriteTables/GroupByKey/Write
Nov 15, 2021 6:51:43 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-15T18:51:43.488Z: Finished operation Write to
BQ/BatchLoads/SinglePartitionsReshuffle/GroupByKey/Close
Nov 15, 2021 6:51:43 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-15T18:51:43.559Z: Executing operation Write to
BQ/BatchLoads/SinglePartitionWriteTables/GroupByKey/Create
Nov 15, 2021 6:51:46 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-15T18:51:43.817Z: Finished operation Write to
BQ/BatchLoads/SinglePartitionWriteTables/GroupByKey/Create
Nov 15, 2021 6:51:46 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-15T18:51:43.962Z: Executing operation Write to
BQ/BatchLoads/SinglePartitionsReshuffle/GroupByKey/Read+Write to
BQ/BatchLoads/SinglePartitionsReshuffle/GroupByKey/GroupByWindow+Write to
BQ/BatchLoads/SinglePartitionsReshuffle/ExpandIterable+Write to
BQ/BatchLoads/SinglePartitionWriteTables/ParMultiDo(WriteTables)+Write to
BQ/BatchLoads/SinglePartitionWriteTables/WithKeys/AddKeys/Map+Write to
BQ/BatchLoads/SinglePartitionWriteTables/Window.Into()/Window.Assign+Write to
BQ/BatchLoads/SinglePartitionWriteTables/GroupByKey/Reify+Write to
BQ/BatchLoads/SinglePartitionWriteTables/GroupByKey/Write
Nov 15, 2021 6:51:46 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-15T18:51:44.005Z: Finished operation Write to
BQ/BatchLoads/MultiPartitionsReshuffle/GroupByKey/Read+Write to
BQ/BatchLoads/MultiPartitionsReshuffle/GroupByKey/GroupByWindow+Write to
BQ/BatchLoads/MultiPartitionsReshuffle/ExpandIterable+Write to
BQ/BatchLoads/MultiPartitionsWriteTables/ParMultiDo(WriteTables)+Write to
BQ/BatchLoads/MultiPartitionsWriteTables/WithKeys/AddKeys/Map+Write to
BQ/BatchLoads/ReifyRenameInput/View.AsIterable/ParDo(ToIsmRecordForGlobalWindow)+Write
to BQ/BatchLoads/MultiPartitionsWriteTables/Window.Into()/Window.Assign+Write
to BQ/BatchLoads/MultiPartitionsWriteTables/GroupByKey/Reify+Write to
BQ/BatchLoads/MultiPartitionsWriteTables/GroupByKey/Write
Nov 15, 2021 6:51:46 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-15T18:51:44.131Z: Executing operation Write to
BQ/BatchLoads/MultiPartitionsWriteTables/GroupByKey/Close
Nov 15, 2021 6:51:46 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-15T18:51:44.168Z: Executing operation Write to
BQ/BatchLoads/ReifyRenameInput/View.AsIterable/CreateDataflowView
Nov 15, 2021 6:51:46 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-15T18:51:44.193Z: Finished operation Write to
BQ/BatchLoads/MultiPartitionsWriteTables/GroupByKey/Close
Nov 15, 2021 6:51:46 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-15T18:51:44.242Z: Finished operation Write to
BQ/BatchLoads/ReifyRenameInput/View.AsIterable/CreateDataflowView
Nov 15, 2021 6:51:46 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-15T18:51:44.273Z: Executing operation Write to
BQ/BatchLoads/MultiPartitionsWriteTables/GroupByKey/Read+Write to
BQ/BatchLoads/MultiPartitionsWriteTables/GroupByKey/GroupByWindow+Write to
BQ/BatchLoads/MultiPartitionsWriteTables/Values/Values/Map+Write to
BQ/BatchLoads/MultiPartitionsWriteTables/ParDo(GarbageCollectTemporaryFiles)
Nov 15, 2021 6:51:46 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-15T18:51:44.413Z: Executing operation Write to
BQ/BatchLoads/ReifyRenameInput/Create.Values/Read(CreateSource)+Write to
BQ/BatchLoads/ReifyRenameInput/ParDo(Anonymous)+Write to
BQ/BatchLoads/WriteRenameUntriggered
Nov 15, 2021 6:51:46 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-15T18:51:44.583Z: Finished operation Write to
BQ/BatchLoads/MultiPartitionsWriteTables/GroupByKey/Read+Write to
BQ/BatchLoads/MultiPartitionsWriteTables/GroupByKey/GroupByWindow+Write to
BQ/BatchLoads/MultiPartitionsWriteTables/Values/Values/Map+Write to
BQ/BatchLoads/MultiPartitionsWriteTables/ParDo(GarbageCollectTemporaryFiles)
Nov 15, 2021 6:51:46 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-15T18:51:45.481Z: Finished operation Write to
BQ/BatchLoads/ReifyRenameInput/Create.Values/Read(CreateSource)+Write to
BQ/BatchLoads/ReifyRenameInput/ParDo(Anonymous)+Write to
BQ/BatchLoads/WriteRenameUntriggered
Nov 15, 2021 6:52:32 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-15T18:52:30.180Z: Finished operation Write to
BQ/BatchLoads/SinglePartitionsReshuffle/GroupByKey/Read+Write to
BQ/BatchLoads/SinglePartitionsReshuffle/GroupByKey/GroupByWindow+Write to
BQ/BatchLoads/SinglePartitionsReshuffle/ExpandIterable+Write to
BQ/BatchLoads/SinglePartitionWriteTables/ParMultiDo(WriteTables)+Write to
BQ/BatchLoads/SinglePartitionWriteTables/WithKeys/AddKeys/Map+Write to
BQ/BatchLoads/SinglePartitionWriteTables/Window.Into()/Window.Assign+Write to
BQ/BatchLoads/SinglePartitionWriteTables/GroupByKey/Reify+Write to
BQ/BatchLoads/SinglePartitionWriteTables/GroupByKey/Write
Nov 15, 2021 6:52:32 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-15T18:52:30.272Z: Executing operation Write to
BQ/BatchLoads/SinglePartitionWriteTables/GroupByKey/Close
Nov 15, 2021 6:52:32 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-15T18:52:30.691Z: Finished operation Write to
BQ/BatchLoads/SinglePartitionWriteTables/GroupByKey/Close
Nov 15, 2021 6:52:32 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-15T18:52:30.761Z: Executing operation Write to
BQ/BatchLoads/SinglePartitionWriteTables/GroupByKey/Read+Write to
BQ/BatchLoads/SinglePartitionWriteTables/GroupByKey/GroupByWindow+Write to
BQ/BatchLoads/SinglePartitionWriteTables/Values/Values/Map+Write to
BQ/BatchLoads/SinglePartitionWriteTables/ParDo(GarbageCollectTemporaryFiles)
Nov 15, 2021 6:52:32 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-15T18:52:31.204Z: Finished operation Write to
BQ/BatchLoads/SinglePartitionWriteTables/GroupByKey/Read+Write to
BQ/BatchLoads/SinglePartitionWriteTables/GroupByKey/GroupByWindow+Write to
BQ/BatchLoads/SinglePartitionWriteTables/Values/Values/Map+Write to
BQ/BatchLoads/SinglePartitionWriteTables/ParDo(GarbageCollectTemporaryFiles)
Nov 15, 2021 6:52:32 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-15T18:52:31.437Z: Cleaning up.
Nov 15, 2021 6:52:32 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-15T18:52:31.527Z: Stopping **** pool...
Nov 15, 2021 6:54:57 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-15T18:54:56.830Z: Autoscaling: Resized **** pool from 5 to 0.
Nov 15, 2021 6:54:57 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-15T18:54:56.878Z: Worker pool stopped.
Nov 15, 2021 6:55:42 PM
org.apache.beam.runners.dataflow.DataflowPipelineJob logTerminalState
INFO: Job 2021-11-15_10_48_47-12954882519030310376 finished with status
DONE.
org.apache.beam.sdk.bigqueryioperftests.BigQueryIOIT > testWriteThenRead
STANDARD_OUT
Load test results for test (ID): 3621c513-68af-4dc1-8b39-3d5394fd6fe4 and
timestamp: 2021-11-15T18:48:36.338000000Z:
Metric: Value:
write_time 55.528
org.apache.beam.sdk.bigqueryioperftests.BigQueryIOIT > testWriteThenRead
STANDARD_ERROR
Nov 15, 2021 6:55:43 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Executing pipeline on the Dataflow Service, which will have billing
implications related to Google Compute Engine usage and other Google Cloud
Services.
Nov 15, 2021 6:55:44 PM org.apache.beam.runners.dataflow.util.PackageUtil
stageClasspathElements
INFO: Uploading 225 files from PipelineOptions.filesToStage to staging
location to prepare for execution.
Nov 15, 2021 6:55:44 PM
org.apache.beam.runners.dataflow.util.PackageUtil$PackageAttributes
forFileToStage
INFO: Staging custom dataflow-****.jar as
beam-runners-google-cloud-dataflow-java-legacy-****-2.35.0-SNAPSHOT-CxDB2MFbq9vTw9dZ8Zb10JhAyMvZWoQKUaKm85AO368.jar
Nov 15, 2021 6:55:45 PM org.apache.beam.runners.dataflow.util.PackageUtil
stageClasspathElements
INFO: Staging files complete: 225 files cached, 0 files newly uploaded in 0
seconds
Nov 15, 2021 6:55:45 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging portable pipeline proto to
gs://temp-storage-for-perf-tests/loadtests/staging/
Nov 15, 2021 6:55:45 PM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage
INFO: Uploading <145704 bytes, hash
7b6eb45c20d889bcc9e454d2671dbb716a231ab2826ec3bfd4b3fa968513d355> to
gs://temp-storage-for-perf-tests/loadtests/staging/pipeline-e260XCDYibzJ5FTSZx27cWojGrKCbsO_1LP6loUT01U.pb
Nov 15, 2021 6:55:46 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read from BQ/TriggerIdCreation/Read(CreateSource) as step s1
Nov 15, 2021 6:55:46 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read from
BQ/ViewId/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map
as step s2
Nov 15, 2021 6:55:46 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read from
BQ/ViewId/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey
as step s3
Nov 15, 2021 6:55:46 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read from
BQ/ViewId/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues
as step s4
Nov 15, 2021 6:55:46 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read from
BQ/ViewId/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map
as step s5
Nov 15, 2021 6:55:46 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read from
BQ/ViewId/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
as step s6
Nov 15, 2021 6:55:46 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read from
BQ/ViewId/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly
as step s7
Nov 15, 2021 6:55:46 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read from
BQ/ViewId/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow)
as step s8
Nov 15, 2021 6:55:46 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read from
BQ/ViewId/Combine.GloballyAsSingletonView/CreateDataflowView as step s9
Nov 15, 2021 6:55:46 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read from BQ/Read(BigQueryTableSource) as step s10
Nov 15, 2021 6:55:46 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read from BQ/PassThroughThenCleanup/ParMultiDo(Identity) as
step s11
Nov 15, 2021 6:55:46 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read from
BQ/PassThroughThenCleanup/View.AsIterable/ParDo(ToIsmRecordForGlobalWindow) as
step s12
Nov 15, 2021 6:55:46 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read from
BQ/PassThroughThenCleanup/View.AsIterable/CreateDataflowView as step s13
Nov 15, 2021 6:55:46 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read from
BQ/PassThroughThenCleanup/Create(CleanupOperation)/Read(CreateSource) as step
s14
Nov 15, 2021 6:55:46 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read from BQ/PassThroughThenCleanup/Cleanup as step s15
Nov 15, 2021 6:55:46 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Gather time as step s16
Nov 15, 2021 6:55:46 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Dataflow SDK version: 2.35.0-SNAPSHOT
Nov 15, 2021 6:55:47 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-11-15_10_55_46-13825191762259675449?project=apache-beam-testing
Nov 15, 2021 6:55:47 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Submitted job: 2021-11-15_10_55_46-13825191762259675449
Nov 15, 2021 6:55:47 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel
--region=us-central1 2021-11-15_10_55_46-13825191762259675449
Nov 15, 2021 6:55:50 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
WARNING: 2021-11-15T18:55:50.107Z: The requested max number of ****s (5) is
ignored as autoscaling is explicitly disabled (autoscalingAlgorithm=NONE).
Nov 15, 2021 6:55:58 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-15T18:55:56.335Z: Worker configuration: e2-standard-2 in
us-central1-a.
Nov 15, 2021 6:55:58 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-15T18:55:56.839Z: Expanding CoGroupByKey operations into
optimizable parts.
Nov 15, 2021 6:55:58 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-15T18:55:56.917Z: Expanding GroupByKey operations into
optimizable parts.
Nov 15, 2021 6:55:58 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-15T18:55:56.946Z: Lifting ValueCombiningMappingFns into
MergeBucketsMappingFns
Nov 15, 2021 6:55:58 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-15T18:55:57.043Z: Fusing adjacent ParDo, Read, Write, and
Flatten operations
Nov 15, 2021 6:55:58 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-15T18:55:57.066Z: Fusing consumer Read from
BQ/PassThroughThenCleanup/ParMultiDo(Identity) into Read from
BQ/Read(BigQueryTableSource)
Nov 15, 2021 6:55:58 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-15T18:55:57.097Z: Fusing consumer Gather time into Read from
BQ/PassThroughThenCleanup/ParMultiDo(Identity)
Nov 15, 2021 6:55:58 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-15T18:55:57.139Z: Fusing consumer Read from
BQ/PassThroughThenCleanup/View.AsIterable/ParDo(ToIsmRecordForGlobalWindow)
into Read from BQ/PassThroughThenCleanup/ParMultiDo(Identity)
Nov 15, 2021 6:55:58 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-15T18:55:57.166Z: Fusing consumer Read from
BQ/PassThroughThenCleanup/Cleanup into Read from
BQ/PassThroughThenCleanup/Create(CleanupOperation)/Read(CreateSource)
Nov 15, 2021 6:55:58 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-15T18:55:57.193Z: Fusing consumer Read from
BQ/ViewId/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map
into Read from BQ/TriggerIdCreation/Read(CreateSource)
Nov 15, 2021 6:55:58 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-15T18:55:57.219Z: Fusing consumer Read from
BQ/ViewId/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey+Read
from
BQ/ViewId/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Partial
into Read from
BQ/ViewId/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map
Nov 15, 2021 6:55:58 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-15T18:55:57.251Z: Fusing consumer Read from
BQ/ViewId/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Reify
into Read from
BQ/ViewId/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey+Read
from
BQ/ViewId/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Partial
Nov 15, 2021 6:55:58 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-15T18:55:57.284Z: Fusing consumer Read from
BQ/ViewId/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Write
into Read from
BQ/ViewId/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Reify
Nov 15, 2021 6:55:58 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-15T18:55:57.321Z: Fusing consumer Read from
BQ/ViewId/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues
into Read from
BQ/ViewId/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Read
Nov 15, 2021 6:55:58 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-15T18:55:57.349Z: Fusing consumer Read from
BQ/ViewId/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Extract
into Read from
BQ/ViewId/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues
Nov 15, 2021 6:55:58 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-15T18:55:57.391Z: Fusing consumer Read from
BQ/ViewId/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map
into Read from
BQ/ViewId/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Extract
Nov 15, 2021 6:55:58 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-15T18:55:57.469Z: Fusing consumer Read from
BQ/ViewId/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
into Read from
BQ/ViewId/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map
Nov 15, 2021 6:55:58 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-15T18:55:57.516Z: Fusing consumer Read from
BQ/ViewId/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
into Read from
BQ/ViewId/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
Nov 15, 2021 6:55:58 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-15T18:55:57.553Z: Fusing consumer Read from
BQ/ViewId/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow)
into Read from
BQ/ViewId/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
Nov 15, 2021 6:55:58 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-15T18:55:57.860Z: Executing operation Read from
BQ/Read(BigQueryTableSource)+Read from
BQ/PassThroughThenCleanup/ParMultiDo(Identity)+Gather time+Read from
BQ/PassThroughThenCleanup/View.AsIterable/ParDo(ToIsmRecordForGlobalWindow)
Nov 15, 2021 6:55:58 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-15T18:55:57.893Z: Executing operation Read from
BQ/ViewId/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Create
Nov 15, 2021 6:55:58 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-15T18:55:57.940Z: Starting 5 ****s in us-central1-a...
Nov 15, 2021 6:55:58 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-15T18:55:58.254Z: Finished operation Read from
BQ/ViewId/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Create
Nov 15, 2021 6:55:58 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-15T18:55:58.382Z: Executing operation Read from
BQ/TriggerIdCreation/Read(CreateSource)+Read from
BQ/ViewId/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map+Read
from
BQ/ViewId/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey+Read
from
BQ/ViewId/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Partial+Read
from
BQ/ViewId/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Reify+Read
from
BQ/ViewId/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Write
Nov 15, 2021 6:56:09 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-15T18:56:07.481Z: Your project already contains 100
Dataflow-created metric descriptors, so new user metrics of the form
custom.googleapis.com/* will not be created. However, all user metrics are also
available in the metric dataflow.googleapis.com/job/user_counter. If you rely
on the custom metrics, you can delete old / unused metric descriptors. See
https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list
and
https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
Nov 15, 2021 6:56:44 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-15T18:56:44.376Z: Autoscaling: Raised the number of ****s to
5 based on the rate of progress in the currently running stage(s).
Nov 15, 2021 6:57:09 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-15T18:57:09.697Z: Workers have started successfully.
Nov 15, 2021 6:57:09 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-11-15T18:57:09.731Z: Workers have started successfully.
org.apache.beam.sdk.bigqueryioperftests.BigQueryIOIT > testWriteThenRead SKIPPED
> Task :sdks:java:io:bigquery-io-perf-tests:integrationTest FAILED
:sdks:java:io:bigquery-io-perf-tests:integrationTest (Thread[Execution **** for
':' Thread 2,5,main]) completed. Took 8 mins 43.096 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task
':sdks:java:io:bigquery-io-perf-tests:integrationTest'.
> Process 'Gradle Test Executor 4' finished with non-zero exit value 143
This problem might be caused by incorrect test process configuration.
Please refer to the test execution section in the User Manual at
https://docs.gradle.org/6.9.1/userguide/java_testing.html#sec:test_execution
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to
get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with
Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See
https://docs.gradle.org/6.9.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 8m 58s
118 actionable tasks: 72 executed, 46 from cache
Publishing build scan...
The message received from the daemon indicates that the daemon has disappeared.
Build request sent: Build{id=d4412539-7243-467d-8da3-58d638bb5bb8,
currentDir=<https://ci-beam.apache.org/job/beam_BiqQueryIO_Batch_Performance_Test_Java_Json/ws/src}>
Attempting to read last messages from the daemon log...
Daemon pid: 17691
log file: /home/jenkins/.gradle/daemon/6.9.1/daemon-17691.out.log
----- Last 20 lines from daemon log file - daemon-17691.out.log -----
* What went wrong:
Execution failed for task
':sdks:java:io:bigquery-io-perf-tests:integrationTest'.
> Process 'Gradle Test Executor 4' finished with non-zero exit value 143
This problem might be caused by incorrect test process configuration.
Please refer to the test execution section in the User Manual at
https://docs.gradle.org/6.9.1/userguide/java_testing.html#sec:test_execution
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to
get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with
Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See
https://docs.gradle.org/6.9.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 8m 58s
118 actionable tasks: 72 executed, 46 from cache
Publishing build scan...
Daemon vm is shutting down... The daemon has exited normally or was terminated
in response to a user interrupt.
----- End of the daemon log -----
FAILURE: Build failed with an exception.
* What went wrong:
Gradle build daemon disappeared unexpectedly (it may have been killed or may
have crashed)
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to
get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
