See
<https://ci-beam.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_V2_Streaming_Java11/457/display/redirect?page=changes>
Changes:
[Pablo Estrada] Revert "Trying out property-based tests for Beam python coders
(#22233)"
[noreply] Bump google.golang.org/api from 0.95.0 to 0.96.0 in /sdks (#23246)
[noreply] [Go SDK] Add timer coder support (#23222)
[noreply] Fix wrong comment (#23272)
[noreply] [Playground] [Backend] Cache component for playground examples
(#22869)
[noreply] [BEAM-13416] Introduce Schema provider for AWS models and deprecate
low
[noreply] [BEAM-14378] [CdapIO] SparkReceiverIO Read via SDF (#17828)
------------------------------------------
[...truncated 292.05 KB...]
Sep 20, 2022 1:51:01 PM org.apache.beam.runners.dataflow.util.PackageUtil
stageClasspathElements
INFO: Uploading 227 files from PipelineOptions.filesToStage to staging location
to prepare for execution.
Sep 20, 2022 1:51:01 PM org.apache.beam.runners.dataflow.util.PackageUtil
stageClasspathElements
INFO: Staging files complete: 227 files cached, 0 files newly uploaded in 0
seconds
Sep 20, 2022 1:51:01 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging portable pipeline proto to
gs://temp-storage-for-perf-tests/loadtests/staging/
Sep 20, 2022 1:51:02 PM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage
INFO: Uploading <132211 bytes, hash
a6b6ae64186f57a0916917550b47337441f258e9887b0aabf146038b0d8e3f61> to
gs://temp-storage-for-perf-tests/loadtests/staging/pipeline-prauZBhvV6CRaRdVC0czdEHyWOmIewqr8UYDiw2OP2E.pb
Sep 20, 2022 1:51:03 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read input/DataflowRunner.StreamingUnboundedRead.ReadWithIds as
step s1
Sep 20, 2022 1:51:03 PM
org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource split
INFO: Split into 64 bundles: [SyntheticUnboundedSource{startOffset=0,
endOffset=39062}, SyntheticUnboundedSource{startOffset=39062, endOffset=78125},
SyntheticUnboundedSource{startOffset=78125, endOffset=117187},
SyntheticUnboundedSource{startOffset=117187, endOffset=156250},
SyntheticUnboundedSource{startOffset=156250, endOffset=195312},
SyntheticUnboundedSource{startOffset=195312, endOffset=234375},
SyntheticUnboundedSource{startOffset=234375, endOffset=273437},
SyntheticUnboundedSource{startOffset=273437, endOffset=312500},
SyntheticUnboundedSource{startOffset=312500, endOffset=351562},
SyntheticUnboundedSource{startOffset=351562, endOffset=390625},
SyntheticUnboundedSource{startOffset=390625, endOffset=429687},
SyntheticUnboundedSource{startOffset=429687, endOffset=468750},
SyntheticUnboundedSource{startOffset=468750, endOffset=507812},
SyntheticUnboundedSource{startOffset=507812, endOffset=546875},
SyntheticUnboundedSource{startOffset=546875, endOffset=585937},
SyntheticUnboundedSource{startOffset=585937, endOffset=625000},
SyntheticUnboundedSource{startOffset=625000, endOffset=664062},
SyntheticUnboundedSource{startOffset=664062, endOffset=703125},
SyntheticUnboundedSource{startOffset=703125, endOffset=742187},
SyntheticUnboundedSource{startOffset=742187, endOffset=781250},
SyntheticUnboundedSource{startOffset=781250, endOffset=820312},
SyntheticUnboundedSource{startOffset=820312, endOffset=859375},
SyntheticUnboundedSource{startOffset=859375, endOffset=898437},
SyntheticUnboundedSource{startOffset=898437, endOffset=937500},
SyntheticUnboundedSource{startOffset=937500, endOffset=976562},
SyntheticUnboundedSource{startOffset=976562, endOffset=1015625},
SyntheticUnboundedSource{startOffset=1015625, endOffset=1054687},
SyntheticUnboundedSource{startOffset=1054687, endOffset=1093750},
SyntheticUnboundedSource{startOffset=1093750, endOffset=1132812},
SyntheticUnboundedSource{startOffset=1132812, endOffset=1171875},
SyntheticUnboundedSource{startOffset=1171875, endOffset=1210937},
SyntheticUnboundedSource{startOffset=1210937, endOffset=1250000},
SyntheticUnboundedSource{startOffset=1250000, endOffset=1289062},
SyntheticUnboundedSource{startOffset=1289062, endOffset=1328125},
SyntheticUnboundedSource{startOffset=1328125, endOffset=1367187},
SyntheticUnboundedSource{startOffset=1367187, endOffset=1406250},
SyntheticUnboundedSource{startOffset=1406250, endOffset=1445312},
SyntheticUnboundedSource{startOffset=1445312, endOffset=1484375},
SyntheticUnboundedSource{startOffset=1484375, endOffset=1523437},
SyntheticUnboundedSource{startOffset=1523437, endOffset=1562500},
SyntheticUnboundedSource{startOffset=1562500, endOffset=1601562},
SyntheticUnboundedSource{startOffset=1601562, endOffset=1640625},
SyntheticUnboundedSource{startOffset=1640625, endOffset=1679687},
SyntheticUnboundedSource{startOffset=1679687, endOffset=1718750},
SyntheticUnboundedSource{startOffset=1718750, endOffset=1757812},
SyntheticUnboundedSource{startOffset=1757812, endOffset=1796875},
SyntheticUnboundedSource{startOffset=1796875, endOffset=1835937},
SyntheticUnboundedSource{startOffset=1835937, endOffset=1875000},
SyntheticUnboundedSource{startOffset=1875000, endOffset=1914062},
SyntheticUnboundedSource{startOffset=1914062, endOffset=1953125},
SyntheticUnboundedSource{startOffset=1953125, endOffset=1992187},
SyntheticUnboundedSource{startOffset=1992187, endOffset=2031250},
SyntheticUnboundedSource{startOffset=2031250, endOffset=2070312},
SyntheticUnboundedSource{startOffset=2070312, endOffset=2109375},
SyntheticUnboundedSource{startOffset=2109375, endOffset=2148437},
SyntheticUnboundedSource{startOffset=2148437, endOffset=2187500},
SyntheticUnboundedSource{startOffset=2187500, endOffset=2226562},
SyntheticUnboundedSource{startOffset=2226562, endOffset=2265625},
SyntheticUnboundedSource{startOffset=2265625, endOffset=2304687},
SyntheticUnboundedSource{startOffset=2304687, endOffset=2343750},
SyntheticUnboundedSource{startOffset=2343750, endOffset=2382812},
SyntheticUnboundedSource{startOffset=2382812, endOffset=2421875},
SyntheticUnboundedSource{startOffset=2421875, endOffset=2460937},
SyntheticUnboundedSource{startOffset=2460937, endOffset=2500000}]
Sep 20, 2022 1:51:03 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read input/StripIds as step s2
Sep 20, 2022 1:51:03 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Collect start time metrics as step s3
Sep 20, 2022 1:51:03 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Total bytes monitor as step s4
Sep 20, 2022 1:51:03 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Window.Into()/Window.Assign as step s5
Sep 20, 2022 1:51:03 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Group by key (0) as step s6
Sep 20, 2022 1:51:03 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Ungroup and reiterate (0) as step s7
Sep 20, 2022 1:51:03 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Collect end time metrics (0) as step s8
Sep 20, 2022 1:51:03 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Group by key (1) as step s9
Sep 20, 2022 1:51:03 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Ungroup and reiterate (1) as step s10
Sep 20, 2022 1:51:03 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Collect end time metrics (1) as step s11
Sep 20, 2022 1:51:03 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Group by key (2) as step s12
Sep 20, 2022 1:51:03 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Ungroup and reiterate (2) as step s13
Sep 20, 2022 1:51:03 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Collect end time metrics (2) as step s14
Sep 20, 2022 1:51:03 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Group by key (3) as step s15
Sep 20, 2022 1:51:03 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Ungroup and reiterate (3) as step s16
Sep 20, 2022 1:51:03 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Collect end time metrics (3) as step s17
Sep 20, 2022 1:51:03 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Group by key (4) as step s18
Sep 20, 2022 1:51:03 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Ungroup and reiterate (4) as step s19
Sep 20, 2022 1:51:03 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Collect end time metrics (4) as step s20
Sep 20, 2022 1:51:03 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Group by key (5) as step s21
Sep 20, 2022 1:51:03 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Ungroup and reiterate (5) as step s22
Sep 20, 2022 1:51:03 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Collect end time metrics (5) as step s23
Sep 20, 2022 1:51:03 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Group by key (6) as step s24
Sep 20, 2022 1:51:03 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Ungroup and reiterate (6) as step s25
Sep 20, 2022 1:51:04 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Collect end time metrics (6) as step s26
Sep 20, 2022 1:51:04 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Group by key (7) as step s27
Sep 20, 2022 1:51:04 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Ungroup and reiterate (7) as step s28
Sep 20, 2022 1:51:04 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Collect end time metrics (7) as step s29
Sep 20, 2022 1:51:04 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Dataflow SDK version: 2.43.0-SNAPSHOT
Sep 20, 2022 1:51:04 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to
https://console.cloud.google.com/dataflow/jobs/us-central1/2022-09-20_06_51_04-17662576294321140826?project=apache-beam-testing
Sep 20, 2022 1:51:04 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Submitted job: 2022-09-20_06_51_04-17662576294321140826
Sep 20, 2022 1:51:04 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel
> --region=us-central1 2022-09-20_06_51_04-17662576294321140826
Sep 20, 2022 1:51:09 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
WARNING: 2022-09-20T13:51:08.202Z: The workflow name is not a valid Cloud
Label. Labels applied to Cloud resources (such as GCE Instances) for monitoring
will be labeled with this modified job name:
load0tests0java110dataflow0v20streaming0gbk05-jenkins-0920-zjoa. For the best
monitoring experience, please name your job with a valid Cloud Label. For
details, see:
https://cloud.google.com/compute/docs/labeling-resources#restrictions
Sep 20, 2022 1:51:15 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-09-20T13:51:14.732Z: Worker configuration: e2-standard-2 in
us-central1-a.
Sep 20, 2022 1:51:16 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-09-20T13:51:15.878Z: Expanding SplittableParDo operations into
optimizable parts.
Sep 20, 2022 1:51:16 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-09-20T13:51:15.910Z: Expanding CollectionToSingleton operations into
optimizable parts.
Sep 20, 2022 1:51:16 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-09-20T13:51:15.987Z: Expanding CoGroupByKey operations into
optimizable parts.
Sep 20, 2022 1:51:16 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-09-20T13:51:16.284Z: Expanding SplittableProcessKeyed operations
into optimizable parts.
Sep 20, 2022 1:51:16 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-09-20T13:51:16.333Z: Expanding GroupByKey operations into streaming
Read/Write steps
Sep 20, 2022 1:51:20 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-09-20T13:51:16.661Z: Lifting ValueCombiningMappingFns into
MergeBucketsMappingFns
Sep 20, 2022 1:51:20 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-09-20T13:51:17.014Z: Fusing adjacent ParDo, Read, Write, and Flatten
operations
Sep 20, 2022 1:51:20 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-09-20T13:51:17.046Z: Fusing consumer Group by key (0)/WriteStream
into Window.Into()/Window.Assign
Sep 20, 2022 1:51:20 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-09-20T13:51:17.081Z: Fusing consumer Group by key (1)/WriteStream
into Window.Into()/Window.Assign
Sep 20, 2022 1:51:20 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-09-20T13:51:17.115Z: Fusing consumer Group by key (2)/WriteStream
into Window.Into()/Window.Assign
Sep 20, 2022 1:51:20 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-09-20T13:51:17.152Z: Fusing consumer Group by key (3)/WriteStream
into Window.Into()/Window.Assign
Sep 20, 2022 1:51:20 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-09-20T13:51:17.187Z: Fusing consumer Group by key (4)/WriteStream
into Window.Into()/Window.Assign
Sep 20, 2022 1:51:20 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-09-20T13:51:17.236Z: Fusing consumer Group by key (5)/WriteStream
into Window.Into()/Window.Assign
Sep 20, 2022 1:51:20 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-09-20T13:51:17.272Z: Fusing consumer Group by key (6)/WriteStream
into Window.Into()/Window.Assign
Sep 20, 2022 1:51:20 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-09-20T13:51:17.305Z: Fusing consumer Group by key (7)/WriteStream
into Window.Into()/Window.Assign
Sep 20, 2022 1:51:20 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-09-20T13:51:17.342Z: Fusing consumer Read
input/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource) into Read
input/Impulse
Sep 20, 2022 1:51:20 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-09-20T13:51:17.380Z: Fusing consumer
Read-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/PairWithRestriction
into Read input/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource)
Sep 20, 2022 1:51:20 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-09-20T13:51:17.416Z: Fusing consumer
Read-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/SplitWithSizing
into
Read-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/PairWithRestriction
Sep 20, 2022 1:51:20 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-09-20T13:51:17.450Z: Fusing consumer Read
input/ParDo(StripIds)/ParMultiDo(StripIds) into
Read-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/ProcessElementAndRestrictionWithSizing
Sep 20, 2022 1:51:20 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-09-20T13:51:17.485Z: Fusing consumer Collect start time
metrics/ParMultiDo(TimeMonitor) into Read
input/ParDo(StripIds)/ParMultiDo(StripIds)
Sep 20, 2022 1:51:20 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-09-20T13:51:17.541Z: Fusing consumer Total bytes
monitor/ParMultiDo(ByteMonitor) into Collect start time
metrics/ParMultiDo(TimeMonitor)
Sep 20, 2022 1:51:20 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-09-20T13:51:17.577Z: Fusing consumer Window.Into()/Window.Assign
into Total bytes monitor/ParMultiDo(ByteMonitor)
Sep 20, 2022 1:51:20 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-09-20T13:51:17.612Z: Fusing consumer Group by key (0)/MergeBuckets
into Group by key (0)/ReadStream
Sep 20, 2022 1:51:20 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-09-20T13:51:17.658Z: Fusing consumer Ungroup and reiterate
(0)/ParMultiDo(UngroupAndReiterate) into Group by key (0)/MergeBuckets
Sep 20, 2022 1:51:20 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-09-20T13:51:17.694Z: Fusing consumer Collect end time metrics
(0)/ParMultiDo(TimeMonitor) into Ungroup and reiterate
(0)/ParMultiDo(UngroupAndReiterate)
Sep 20, 2022 1:51:20 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-09-20T13:51:17.719Z: Fusing consumer Group by key (1)/MergeBuckets
into Group by key (1)/ReadStream
Sep 20, 2022 1:51:20 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-09-20T13:51:17.764Z: Fusing consumer Ungroup and reiterate
(1)/ParMultiDo(UngroupAndReiterate) into Group by key (1)/MergeBuckets
Sep 20, 2022 1:51:20 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-09-20T13:51:17.798Z: Fusing consumer Collect end time metrics
(1)/ParMultiDo(TimeMonitor) into Ungroup and reiterate
(1)/ParMultiDo(UngroupAndReiterate)
Sep 20, 2022 1:51:20 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-09-20T13:51:17.823Z: Fusing consumer Group by key (2)/MergeBuckets
into Group by key (2)/ReadStream
Sep 20, 2022 1:51:20 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-09-20T13:51:17.860Z: Fusing consumer Ungroup and reiterate
(2)/ParMultiDo(UngroupAndReiterate) into Group by key (2)/MergeBuckets
Sep 20, 2022 1:51:20 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-09-20T13:51:17.894Z: Fusing consumer Collect end time metrics
(2)/ParMultiDo(TimeMonitor) into Ungroup and reiterate
(2)/ParMultiDo(UngroupAndReiterate)
Sep 20, 2022 1:51:20 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-09-20T13:51:17.919Z: Fusing consumer Group by key (3)/MergeBuckets
into Group by key (3)/ReadStream
Sep 20, 2022 1:51:20 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-09-20T13:51:17.955Z: Fusing consumer Ungroup and reiterate
(3)/ParMultiDo(UngroupAndReiterate) into Group by key (3)/MergeBuckets
Sep 20, 2022 1:51:20 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-09-20T13:51:17.980Z: Fusing consumer Collect end time metrics
(3)/ParMultiDo(TimeMonitor) into Ungroup and reiterate
(3)/ParMultiDo(UngroupAndReiterate)
Sep 20, 2022 1:51:20 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-09-20T13:51:18.011Z: Fusing consumer Group by key (4)/MergeBuckets
into Group by key (4)/ReadStream
Sep 20, 2022 1:51:20 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-09-20T13:51:18.044Z: Fusing consumer Ungroup and reiterate
(4)/ParMultiDo(UngroupAndReiterate) into Group by key (4)/MergeBuckets
Sep 20, 2022 1:51:20 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-09-20T13:51:18.078Z: Fusing consumer Collect end time metrics
(4)/ParMultiDo(TimeMonitor) into Ungroup and reiterate
(4)/ParMultiDo(UngroupAndReiterate)
Sep 20, 2022 1:51:20 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-09-20T13:51:18.104Z: Fusing consumer Group by key (5)/MergeBuckets
into Group by key (5)/ReadStream
Sep 20, 2022 1:51:20 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-09-20T13:51:18.136Z: Fusing consumer Ungroup and reiterate
(5)/ParMultiDo(UngroupAndReiterate) into Group by key (5)/MergeBuckets
Sep 20, 2022 1:51:20 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-09-20T13:51:18.167Z: Fusing consumer Collect end time metrics
(5)/ParMultiDo(TimeMonitor) into Ungroup and reiterate
(5)/ParMultiDo(UngroupAndReiterate)
Sep 20, 2022 1:51:20 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-09-20T13:51:18.202Z: Fusing consumer Group by key (6)/MergeBuckets
into Group by key (6)/ReadStream
Sep 20, 2022 1:51:20 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-09-20T13:51:18.236Z: Fusing consumer Ungroup and reiterate
(6)/ParMultiDo(UngroupAndReiterate) into Group by key (6)/MergeBuckets
Sep 20, 2022 1:51:20 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-09-20T13:51:18.270Z: Fusing consumer Collect end time metrics
(6)/ParMultiDo(TimeMonitor) into Ungroup and reiterate
(6)/ParMultiDo(UngroupAndReiterate)
Sep 20, 2022 1:51:20 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-09-20T13:51:18.304Z: Fusing consumer Group by key (7)/MergeBuckets
into Group by key (7)/ReadStream
Sep 20, 2022 1:51:20 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-09-20T13:51:18.343Z: Fusing consumer Ungroup and reiterate
(7)/ParMultiDo(UngroupAndReiterate) into Group by key (7)/MergeBuckets
Sep 20, 2022 1:51:20 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-09-20T13:51:18.379Z: Fusing consumer Collect end time metrics
(7)/ParMultiDo(TimeMonitor) into Ungroup and reiterate
(7)/ParMultiDo(UngroupAndReiterate)
Sep 20, 2022 1:51:20 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-09-20T13:51:18.479Z: Running job using Streaming Engine
Sep 20, 2022 1:51:20 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-09-20T13:51:18.747Z: Starting 16 ****s in us-central1-a...
Sep 20, 2022 1:51:49 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-09-20T13:51:27.898Z: Your project already contains 100
Dataflow-created metric descriptors, so new user metrics of the form
custom.googleapis.com/* will not be created. However, all user metrics are also
available in the metric dataflow.googleapis.com/job/user_counter. If you rely
on the custom metrics, you can delete old / unused metric descriptors. See
https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list
and
https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
Sep 20, 2022 1:52:30 PM org.apache.beam.sdk.metrics.MetricsEnvironment
getCurrentContainer
WARNING: Reporting metrics are not supported in the current execution
environment.
Sep 20, 2022 1:52:30 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-09-20T13:52:00.055Z: Autoscaling: Raised the number of ****s to 16
so that the pipeline can catch up with its backlog and keep up with its input
rate.
Sep 20, 2022 1:53:09 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-09-20T13:53:07.813Z: Workers have started successfully.
Sep 20, 2022 2:08:38 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-09-20T14:08:37.737Z: Cleaning up.
Sep 20, 2022 2:08:38 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-09-20T14:08:37.882Z: Stopping **** pool...
Sep 20, 2022 2:08:38 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-09-20T14:08:37.940Z: Stopping **** pool...
Sep 20, 2022 2:10:51 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-09-20T14:10:49.183Z: Autoscaling: Reduced the number of ****s to 0
based on low average **** CPU utilization, and the pipeline having sufficiently
low backlog and keeping up with input rate.
Sep 20, 2022 2:10:51 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-09-20T14:10:49.242Z: Worker pool stopped.
Sep 20, 2022 2:10:55 PM org.apache.beam.runners.dataflow.DataflowPipelineJob
logTerminalState
INFO: Job 2022-09-20_06_51_04-17662576294321140826 finished with status DONE.
Load test results for test (ID): 3702c9e5-520b-40f2-8d73-f7c65cb1066b and
timestamp: 2022-09-20T13:50:58.818000000Z:
Metric: Value:
dataflow_v2_java11_runtime_sec 859.197
dataflow_v2_java11_total_bytes_count 2.49998E8
> Task :runners:google-cloud-dataflow-java:cleanUpDockerJavaImages
Untagged: us.gcr.io/apache-beam-testing/java-postcommit-it/java:20220920135024
Untagged:
us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:f93125c06b7cc914c9935569ca1d2582718ff326a98658399bc261f803b14a0f
Tag: [us.gcr.io/apache-beam-testing/java-postcommit-it/java:20220920135024]
- referencing digest:
[us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:f93125c06b7cc914c9935569ca1d2582718ff326a98658399bc261f803b14a0f]
Deleted [[us.gcr.io/apache-beam-testing/java-postcommit-it/java:20220920135024]
(referencing
[us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:f93125c06b7cc914c9935569ca1d2582718ff326a98658399bc261f803b14a0f])].
Removing untagged image
us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:1c3f49a575c56a62c51a1fb898e918c81316192a87b4f71117a5e902cb84b98c
Digests:
-
us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:1c3f49a575c56a62c51a1fb898e918c81316192a87b4f71117a5e902cb84b98c
ERROR: (gcloud.container.images.delete) Not found: response:
{'docker-distribution-api-version': 'registry/2.0', 'content-type':
'application/json', 'date': 'Tue, 20 Sep 2022 14:11:03 GMT', 'server': 'Docker
Registry', 'cache-control': 'private', 'x-xss-protection': '0',
'x-frame-options': 'SAMEORIGIN', 'transfer-encoding': 'chunked', 'status':
'404', 'content-length': '168', '-content-encoding': 'gzip'}
Failed to compute blob liveness for manifest:
'sha256:1c3f49a575c56a62c51a1fb898e918c81316192a87b4f71117a5e902cb84b98c': None
> Task :runners:google-cloud-dataflow-java:cleanUpDockerJavaImages FAILED
FAILURE: Build failed with an exception.
* Where:
Build file
'<https://ci-beam.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_V2_Streaming_Java11/ws/src/runners/google-cloud-dataflow-java/build.gradle'>
line: 302
* What went wrong:
Execution failed for task
':runners:google-cloud-dataflow-java:cleanUpDockerJavaImages'.
> Process 'command './scripts/cleanup_untagged_gcr_images.sh'' finished with
> non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with
Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings
and determine if they come from your own scripts or plugins.
See
https://docs.gradle.org/7.5.1/userguide/command_line_interface.html#sec:command_line_warnings
Execution optimizations have been disabled for 1 invalid unit(s) of work during
this build to ensure correctness.
Please consult deprecation warnings for more details.
BUILD FAILED in 20m 40s
105 actionable tasks: 8 executed, 97 up-to-date
Publishing build scan...
https://gradle.com/s/htaf4zewvxa5a
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
