See
<https://ci-beam.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_V2_Streaming_Java11/416/display/redirect?page=changes>
Changes:
[alexey.inkin] Fix retaining unsaved pipeline options (#22075)
[108862444+oborysevych] removed VladMatyunin from beam collaborators
[anandinguva98] Add stdlib distutils while building the wheels
[noreply] Fix for #22631 KafkaIO considers readCommitted() as it would commit
back
[noreply] [CdapIO] Add CdapIO dashboard in Grafana (#22641)
[noreply] Add information on how to take/close issues in the contribution guide.
[noreply] Skip
[noreply] Persist ghprbPullId parameter in seed job (#22579)
[noreply] Adhoc: Fix logging in Spark runner to avoid unnecessary creation of
------------------------------------------
[...truncated 226.06 KB...]
826c3ddbb29c: Preparing
b626401ef603: Preparing
9b55156abf26: Preparing
293d5db30c9f: Preparing
03127cdb479b: Preparing
9c742cd6c7a5: Preparing
8a00657a0dfd: Waiting
fe232d1b5382: Waiting
3b71caba8f20: Waiting
b06a77c2a5cd: Waiting
826c3ddbb29c: Waiting
0714080dab75: Waiting
b626401ef603: Waiting
03127cdb479b: Waiting
7b7f3078e1db: Waiting
54a83e661a5e: Waiting
a205551df86b: Waiting
9c742cd6c7a5: Waiting
293d5db30c9f: Waiting
9b55156abf26: Waiting
463de2fbdc02: Waiting
9e94d85e7e23: Waiting
829a76c2d262: Pushed
357fb3026812: Pushed
04010c7fe13d: Pushed
87c39a7eda51: Pushed
a54ea2c5cb82: Pushed
3b71caba8f20: Pushed
fe232d1b5382: Pushed
b06a77c2a5cd: Pushed
8a00657a0dfd: Pushed
a205551df86b: Pushed
9e94d85e7e23: Pushed
0714080dab75: Pushed
7b7f3078e1db: Layer already exists
826c3ddbb29c: Layer already exists
b626401ef603: Layer already exists
9b55156abf26: Layer already exists
293d5db30c9f: Layer already exists
03127cdb479b: Layer already exists
9c742cd6c7a5: Layer already exists
54a83e661a5e: Pushed
463de2fbdc02: Pushed
72ffadca8067: Pushed
20220810132750: digest:
sha256:ad4c3d10a05a6b0b155315f13ddfeeacdf9ac70aad2afd04c10a30e3d3735d9b size:
4935
> Task :sdks:java:testing:load-tests:run
Aug 10, 2022 1:28:24 PM
org.apache.beam.runners.dataflow.options.DataflowPipelineOptions$StagingLocationFactory
create
INFO: No stagingLocation provided, falling back to gcpTempLocation
Aug 10, 2022 1:28:24 PM org.apache.beam.runners.dataflow.DataflowRunner
fromOptions
INFO: PipelineOptions.filesToStage was not specified. Defaulting to files from
the classpath: will stage 222 files. Enable logging at DEBUG level to see which
files will be staged.
Aug 10, 2022 1:28:25 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Executing pipeline on the Dataflow Service, which will have billing
implications related to Google Compute Engine usage and other Google Cloud
Services.
Aug 10, 2022 1:28:27 PM org.apache.beam.runners.dataflow.util.PackageUtil
stageClasspathElements
INFO: Uploading 222 files from PipelineOptions.filesToStage to staging location
to prepare for execution.
Aug 10, 2022 1:28:28 PM org.apache.beam.runners.dataflow.util.PackageUtil
stageClasspathElements
INFO: Staging files complete: 222 files cached, 0 files newly uploaded in 0
seconds
Aug 10, 2022 1:28:28 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging portable pipeline proto to
gs://temp-storage-for-perf-tests/loadtests/staging/
Aug 10, 2022 1:28:28 PM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage
INFO: Uploading <112157 bytes, hash
e0fc74157f1e3880cc34433779b16b5c4066e19bbd622c37bebe861ee5540b44> to
gs://temp-storage-for-perf-tests/loadtests/staging/pipeline-4Px0FX8eOIDMNEM3ebFrXEBm4Zu9Yiw3vr6GHuVUC0Q.pb
Aug 10, 2022 1:28:30 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read input/DataflowRunner.StreamingUnboundedRead.ReadWithIds as
step s1
Aug 10, 2022 1:28:30 PM
org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource split
INFO: Split into 64 bundles: [SyntheticUnboundedSource{startOffset=0,
endOffset=78125}, SyntheticUnboundedSource{startOffset=78125,
endOffset=156250}, SyntheticUnboundedSource{startOffset=156250,
endOffset=234375}, SyntheticUnboundedSource{startOffset=234375,
endOffset=312500}, SyntheticUnboundedSource{startOffset=312500,
endOffset=390625}, SyntheticUnboundedSource{startOffset=390625,
endOffset=468750}, SyntheticUnboundedSource{startOffset=468750,
endOffset=546875}, SyntheticUnboundedSource{startOffset=546875,
endOffset=625000}, SyntheticUnboundedSource{startOffset=625000,
endOffset=703125}, SyntheticUnboundedSource{startOffset=703125,
endOffset=781250}, SyntheticUnboundedSource{startOffset=781250,
endOffset=859375}, SyntheticUnboundedSource{startOffset=859375,
endOffset=937500}, SyntheticUnboundedSource{startOffset=937500,
endOffset=1015625}, SyntheticUnboundedSource{startOffset=1015625,
endOffset=1093750}, SyntheticUnboundedSource{startOffset=1093750,
endOffset=1171875}, SyntheticUnboundedSource{startOffset=1171875,
endOffset=1250000}, SyntheticUnboundedSource{startOffset=1250000,
endOffset=1328125}, SyntheticUnboundedSource{startOffset=1328125,
endOffset=1406250}, SyntheticUnboundedSource{startOffset=1406250,
endOffset=1484375}, SyntheticUnboundedSource{startOffset=1484375,
endOffset=1562500}, SyntheticUnboundedSource{startOffset=1562500,
endOffset=1640625}, SyntheticUnboundedSource{startOffset=1640625,
endOffset=1718750}, SyntheticUnboundedSource{startOffset=1718750,
endOffset=1796875}, SyntheticUnboundedSource{startOffset=1796875,
endOffset=1875000}, SyntheticUnboundedSource{startOffset=1875000,
endOffset=1953125}, SyntheticUnboundedSource{startOffset=1953125,
endOffset=2031250}, SyntheticUnboundedSource{startOffset=2031250,
endOffset=2109375}, SyntheticUnboundedSource{startOffset=2109375,
endOffset=2187500}, SyntheticUnboundedSource{startOffset=2187500,
endOffset=2265625}, SyntheticUnboundedSource{startOffset=2265625,
endOffset=2343750}, SyntheticUnboundedSource{startOffset=2343750,
endOffset=2421875}, SyntheticUnboundedSource{startOffset=2421875,
endOffset=2500000}, SyntheticUnboundedSource{startOffset=2500000,
endOffset=2578125}, SyntheticUnboundedSource{startOffset=2578125,
endOffset=2656250}, SyntheticUnboundedSource{startOffset=2656250,
endOffset=2734375}, SyntheticUnboundedSource{startOffset=2734375,
endOffset=2812500}, SyntheticUnboundedSource{startOffset=2812500,
endOffset=2890625}, SyntheticUnboundedSource{startOffset=2890625,
endOffset=2968750}, SyntheticUnboundedSource{startOffset=2968750,
endOffset=3046875}, SyntheticUnboundedSource{startOffset=3046875,
endOffset=3125000}, SyntheticUnboundedSource{startOffset=3125000,
endOffset=3203125}, SyntheticUnboundedSource{startOffset=3203125,
endOffset=3281250}, SyntheticUnboundedSource{startOffset=3281250,
endOffset=3359375}, SyntheticUnboundedSource{startOffset=3359375,
endOffset=3437500}, SyntheticUnboundedSource{startOffset=3437500,
endOffset=3515625}, SyntheticUnboundedSource{startOffset=3515625,
endOffset=3593750}, SyntheticUnboundedSource{startOffset=3593750,
endOffset=3671875}, SyntheticUnboundedSource{startOffset=3671875,
endOffset=3750000}, SyntheticUnboundedSource{startOffset=3750000,
endOffset=3828125}, SyntheticUnboundedSource{startOffset=3828125,
endOffset=3906250}, SyntheticUnboundedSource{startOffset=3906250,
endOffset=3984375}, SyntheticUnboundedSource{startOffset=3984375,
endOffset=4062500}, SyntheticUnboundedSource{startOffset=4062500,
endOffset=4140625}, SyntheticUnboundedSource{startOffset=4140625,
endOffset=4218750}, SyntheticUnboundedSource{startOffset=4218750,
endOffset=4296875}, SyntheticUnboundedSource{startOffset=4296875,
endOffset=4375000}, SyntheticUnboundedSource{startOffset=4375000,
endOffset=4453125}, SyntheticUnboundedSource{startOffset=4453125,
endOffset=4531250}, SyntheticUnboundedSource{startOffset=4531250,
endOffset=4609375}, SyntheticUnboundedSource{startOffset=4609375,
endOffset=4687500}, SyntheticUnboundedSource{startOffset=4687500,
endOffset=4765625}, SyntheticUnboundedSource{startOffset=4765625,
endOffset=4843750}, SyntheticUnboundedSource{startOffset=4843750,
endOffset=4921875}, SyntheticUnboundedSource{startOffset=4921875,
endOffset=5000000}]
Aug 10, 2022 1:28:30 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read input/StripIds as step s2
Aug 10, 2022 1:28:30 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Collect start time metrics as step s3
Aug 10, 2022 1:28:30 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Total bytes monitor as step s4
Aug 10, 2022 1:28:30 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Window.Into()/Window.Assign as step s5
Aug 10, 2022 1:28:30 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Group by key (0) as step s6
Aug 10, 2022 1:28:30 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Ungroup and reiterate (0) as step s7
Aug 10, 2022 1:28:30 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Collect end time metrics (0) as step s8
Aug 10, 2022 1:28:30 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Group by key (1) as step s9
Aug 10, 2022 1:28:30 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Ungroup and reiterate (1) as step s10
Aug 10, 2022 1:28:30 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Collect end time metrics (1) as step s11
Aug 10, 2022 1:28:30 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Group by key (2) as step s12
Aug 10, 2022 1:28:30 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Ungroup and reiterate (2) as step s13
Aug 10, 2022 1:28:30 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Collect end time metrics (2) as step s14
Aug 10, 2022 1:28:30 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Group by key (3) as step s15
Aug 10, 2022 1:28:30 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Ungroup and reiterate (3) as step s16
Aug 10, 2022 1:28:30 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Collect end time metrics (3) as step s17
Aug 10, 2022 1:28:30 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Dataflow SDK version: 2.42.0-SNAPSHOT
Aug 10, 2022 1:28:30 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to
https://console.cloud.google.com/dataflow/jobs/us-central1/2022-08-10_06_28_30-494794312093828965?project=apache-beam-testing
Aug 10, 2022 1:28:30 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Submitted job: 2022-08-10_06_28_30-494794312093828965
Aug 10, 2022 1:28:30 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel
> --region=us-central1 2022-08-10_06_28_30-494794312093828965
Aug 10, 2022 1:28:46 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
WARNING: 2022-08-10T13:28:40.893Z: The workflow name is not a valid Cloud
Label. Labels applied to Cloud resources (such as GCE Instances) for monitoring
will be labeled with this modified job name:
load0tests0java110dataflow0v20streaming0gbk04-jenkins-0810-ssed. For the best
monitoring experience, please name your job with a valid Cloud Label. For
details, see:
https://cloud.google.com/compute/docs/labeling-resources#restrictions
Aug 10, 2022 1:28:53 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-08-10T13:28:51.012Z: Worker configuration: e2-standard-2 in
us-central1-a.
Aug 10, 2022 1:28:55 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-08-10T13:28:54.050Z: Expanding SplittableParDo operations into
optimizable parts.
Aug 10, 2022 1:28:55 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-08-10T13:28:54.119Z: Expanding CollectionToSingleton operations into
optimizable parts.
Aug 10, 2022 1:28:55 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-08-10T13:28:54.184Z: Expanding CoGroupByKey operations into
optimizable parts.
Aug 10, 2022 1:28:55 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-08-10T13:28:54.358Z: Expanding SplittableProcessKeyed operations
into optimizable parts.
Aug 10, 2022 1:28:55 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-08-10T13:28:54.387Z: Expanding GroupByKey operations into streaming
Read/Write steps
Aug 10, 2022 1:28:55 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-08-10T13:28:54.511Z: Lifting ValueCombiningMappingFns into
MergeBucketsMappingFns
Aug 10, 2022 1:28:55 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-08-10T13:28:54.677Z: Fusing adjacent ParDo, Read, Write, and Flatten
operations
Aug 10, 2022 1:28:55 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-08-10T13:28:54.708Z: Fusing consumer Group by key (0)/WriteStream
into Window.Into()/Window.Assign
Aug 10, 2022 1:28:55 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-08-10T13:28:54.738Z: Fusing consumer Group by key (1)/WriteStream
into Window.Into()/Window.Assign
Aug 10, 2022 1:28:55 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-08-10T13:28:54.765Z: Fusing consumer Group by key (2)/WriteStream
into Window.Into()/Window.Assign
Aug 10, 2022 1:28:55 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-08-10T13:28:54.796Z: Fusing consumer Group by key (3)/WriteStream
into Window.Into()/Window.Assign
Aug 10, 2022 1:28:55 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-08-10T13:28:54.829Z: Fusing consumer Read
input/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource) into Read
input/Impulse
Aug 10, 2022 1:28:55 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-08-10T13:28:54.859Z: Fusing consumer
Read-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/PairWithRestriction
into Read input/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource)
Aug 10, 2022 1:28:55 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-08-10T13:28:54.883Z: Fusing consumer
Read-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/SplitWithSizing
into
Read-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/PairWithRestriction
Aug 10, 2022 1:28:55 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-08-10T13:28:54.906Z: Fusing consumer Read
input/ParDo(StripIds)/ParMultiDo(StripIds) into
Read-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/ProcessElementAndRestrictionWithSizing
Aug 10, 2022 1:28:55 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-08-10T13:28:54.929Z: Fusing consumer Collect start time
metrics/ParMultiDo(TimeMonitor) into Read
input/ParDo(StripIds)/ParMultiDo(StripIds)
Aug 10, 2022 1:28:55 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-08-10T13:28:54.956Z: Fusing consumer Total bytes
monitor/ParMultiDo(ByteMonitor) into Collect start time
metrics/ParMultiDo(TimeMonitor)
Aug 10, 2022 1:28:55 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-08-10T13:28:54.988Z: Fusing consumer Window.Into()/Window.Assign
into Total bytes monitor/ParMultiDo(ByteMonitor)
Aug 10, 2022 1:28:55 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-08-10T13:28:55.020Z: Fusing consumer Group by key (0)/MergeBuckets
into Group by key (0)/ReadStream
Aug 10, 2022 1:28:55 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-08-10T13:28:55.084Z: Fusing consumer Ungroup and reiterate
(0)/ParMultiDo(UngroupAndReiterate) into Group by key (0)/MergeBuckets
Aug 10, 2022 1:28:55 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-08-10T13:28:55.109Z: Fusing consumer Collect end time metrics
(0)/ParMultiDo(TimeMonitor) into Ungroup and reiterate
(0)/ParMultiDo(UngroupAndReiterate)
Aug 10, 2022 1:28:55 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-08-10T13:28:55.138Z: Fusing consumer Group by key (1)/MergeBuckets
into Group by key (1)/ReadStream
Aug 10, 2022 1:28:55 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-08-10T13:28:55.163Z: Fusing consumer Ungroup and reiterate
(1)/ParMultiDo(UngroupAndReiterate) into Group by key (1)/MergeBuckets
Aug 10, 2022 1:28:55 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-08-10T13:28:55.197Z: Fusing consumer Collect end time metrics
(1)/ParMultiDo(TimeMonitor) into Ungroup and reiterate
(1)/ParMultiDo(UngroupAndReiterate)
Aug 10, 2022 1:28:55 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-08-10T13:28:55.221Z: Fusing consumer Group by key (2)/MergeBuckets
into Group by key (2)/ReadStream
Aug 10, 2022 1:28:55 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-08-10T13:28:55.259Z: Fusing consumer Ungroup and reiterate
(2)/ParMultiDo(UngroupAndReiterate) into Group by key (2)/MergeBuckets
Aug 10, 2022 1:28:55 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-08-10T13:28:55.285Z: Fusing consumer Collect end time metrics
(2)/ParMultiDo(TimeMonitor) into Ungroup and reiterate
(2)/ParMultiDo(UngroupAndReiterate)
Aug 10, 2022 1:28:55 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-08-10T13:28:55.307Z: Fusing consumer Group by key (3)/MergeBuckets
into Group by key (3)/ReadStream
Aug 10, 2022 1:28:55 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-08-10T13:28:55.343Z: Fusing consumer Ungroup and reiterate
(3)/ParMultiDo(UngroupAndReiterate) into Group by key (3)/MergeBuckets
Aug 10, 2022 1:28:55 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-08-10T13:28:55.374Z: Fusing consumer Collect end time metrics
(3)/ParMultiDo(TimeMonitor) into Ungroup and reiterate
(3)/ParMultiDo(UngroupAndReiterate)
Aug 10, 2022 1:28:55 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-08-10T13:28:55.482Z: Running job using Streaming Engine
Aug 10, 2022 1:28:55 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-08-10T13:28:55.682Z: Starting 16 ****s in us-central1-a...
Aug 10, 2022 1:29:04 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-08-10T13:29:03.809Z: Your project already contains 100
Dataflow-created metric descriptors, so new user metrics of the form
custom.googleapis.com/* will not be created. However, all user metrics are also
available in the metric dataflow.googleapis.com/job/user_counter. If you rely
on the custom metrics, you can delete old / unused metric descriptors. See
https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list
and
https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
Aug 10, 2022 1:29:33 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-08-10T13:29:33.349Z: Autoscaling: Raised the number of ****s to 16
so that the pipeline can catch up with its backlog and keep up with its input
rate.
Aug 10, 2022 1:30:36 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-08-10T13:30:34.245Z: Workers have started successfully.
Aug 10, 2022 1:43:24 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-08-10T13:43:22.539Z: Cleaning up.
Aug 10, 2022 1:43:24 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-08-10T13:43:22.625Z: Stopping **** pool...
Aug 10, 2022 1:43:24 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-08-10T13:43:22.680Z: Stopping **** pool...
Aug 10, 2022 1:45:37 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-08-10T13:45:36.685Z: Autoscaling: Reduced the number of ****s to 0
based on low average **** CPU utilization, and the pipeline having sufficiently
low backlog and keeping up with input rate.
Aug 10, 2022 1:45:37 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-08-10T13:45:36.724Z: Worker pool stopped.
Aug 10, 2022 1:45:42 PM org.apache.beam.runners.dataflow.DataflowPipelineJob
logTerminalState
INFO: Job 2022-08-10_06_28_30-494794312093828965 finished with status DONE.
Load test results for test (ID): 59cc87b6-d3dc-416b-bc95-0bb7f29a07c7 and
timestamp: 2022-08-10T13:28:24.952000000Z:
Metric: Value:
dataflow_v2_java11_runtime_sec 650.883
dataflow_v2_java11_total_bytes_count 4.99998E8
> Task :runners:google-cloud-dataflow-java:cleanUpDockerJavaImages
Untagged: us.gcr.io/apache-beam-testing/java-postcommit-it/java:20220810132750
Untagged:
us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:ad4c3d10a05a6b0b155315f13ddfeeacdf9ac70aad2afd04c10a30e3d3735d9b
Tag: [us.gcr.io/apache-beam-testing/java-postcommit-it/java:20220810132750]
- referencing digest:
[us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:ad4c3d10a05a6b0b155315f13ddfeeacdf9ac70aad2afd04c10a30e3d3735d9b]
Deleted [[us.gcr.io/apache-beam-testing/java-postcommit-it/java:20220810132750]
(referencing
[us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:ad4c3d10a05a6b0b155315f13ddfeeacdf9ac70aad2afd04c10a30e3d3735d9b])].
Removing untagged image
us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:ad4c3d10a05a6b0b155315f13ddfeeacdf9ac70aad2afd04c10a30e3d3735d9b
Digests:
-
us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:ad4c3d10a05a6b0b155315f13ddfeeacdf9ac70aad2afd04c10a30e3d3735d9b
Deleted
[us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:ad4c3d10a05a6b0b155315f13ddfeeacdf9ac70aad2afd04c10a30e3d3735d9b].
Removing untagged image
us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:f779e77f9e9acbdf83ced5403e5b460095ad28df8497d80af219f40f72be610c
Digests:
-
us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:f779e77f9e9acbdf83ced5403e5b460095ad28df8497d80af219f40f72be610c
ERROR: (gcloud.container.images.delete) Not found: response:
{'docker-distribution-api-version': 'registry/2.0', 'content-type':
'application/json', 'date': 'Wed, 10 Aug 2022 13:45:52 GMT', 'server': 'Docker
Registry', 'cache-control': 'private', 'x-xss-protection': '0',
'x-frame-options': 'SAMEORIGIN', 'transfer-encoding': 'chunked', 'status':
'404', 'content-length': '168', '-content-encoding': 'gzip'}
Failed to compute blob liveness for manifest:
'sha256:f779e77f9e9acbdf83ced5403e5b460095ad28df8497d80af219f40f72be610c': None
> Task :runners:google-cloud-dataflow-java:cleanUpDockerJavaImages FAILED
FAILURE: Build failed with an exception.
* Where:
Build file
'<https://ci-beam.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_V2_Streaming_Java11/ws/src/runners/google-cloud-dataflow-java/build.gradle'>
line: 298
* What went wrong:
Execution failed for task
':runners:google-cloud-dataflow-java:cleanUpDockerJavaImages'.
> Process 'command './scripts/cleanup_untagged_gcr_images.sh'' finished with
> non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with
Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings
and determine if they come from your own scripts or plugins.
See
https://docs.gradle.org/7.4/userguide/command_line_interface.html#sec:command_line_warnings
Execution optimizations have been disabled for 1 invalid unit(s) of work during
this build to ensure correctness.
Please consult deprecation warnings for more details.
BUILD FAILED in 18m 3s
105 actionable tasks: 8 executed, 97 up-to-date
Publishing build scan...
https://gradle.com/s/m3lhwfitetrva
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
