See <https://ci-beam.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Streaming/466/display/redirect>
Changes: ------------------------------------------ [...truncated 24.51 KB...] > Task :sdks:java:extensions:google-cloud-platform-core:compileJava FROM-CACHE > Task :sdks:java:extensions:google-cloud-platform-core:classes UP-TO-DATE > Task :sdks:java:extensions:protobuf:compileJava FROM-CACHE > Task :sdks:java:extensions:protobuf:classes UP-TO-DATE > Task :sdks:java:io:synthetic:jar > Task :sdks:java:io:kinesis:jar > Task :sdks:java:fn-execution:compileJava FROM-CACHE > Task :sdks:java:fn-execution:classes UP-TO-DATE > Task :runners:core-construction-java:compileJava FROM-CACHE > Task :runners:core-construction-java:classes UP-TO-DATE > Task :sdks:java:fn-execution:jar > Task :vendor:sdks-java-extensions-protobuf:shadowJar > Task :sdks:java:extensions:google-cloud-platform-core:jar > Task :sdks:java:extensions:protobuf:jar > Task :runners:core-construction-java:jar > Task :sdks:java:testing:test-utils:compileJava FROM-CACHE > Task :sdks:java:testing:test-utils:classes UP-TO-DATE > Task :runners:core-java:compileJava FROM-CACHE > Task :runners:core-java:classes UP-TO-DATE > Task :sdks:java:testing:test-utils:jar > Task :runners:core-java:jar > Task :sdks:java:harness:compileJava FROM-CACHE > Task :sdks:java:harness:classes UP-TO-DATE > Task :sdks:java:harness:jar > Task :sdks:java:harness:shadowJar > Task :runners:java-fn-execution:compileJava FROM-CACHE > Task :runners:java-fn-execution:classes UP-TO-DATE > Task :runners:java-fn-execution:jar > Task :sdks:java:expansion-service:compileJava FROM-CACHE > Task :sdks:java:expansion-service:classes UP-TO-DATE > Task :sdks:java:expansion-service:jar > Task :runners:direct-java:compileJava FROM-CACHE > Task :runners:direct-java:classes UP-TO-DATE > Task :sdks:java:io:kafka:compileJava FROM-CACHE > Task :sdks:java:io:kafka:classes UP-TO-DATE > Task :sdks:java:io:kafka:jar > Task :sdks:java:io:google-cloud-platform:compileJava FROM-CACHE > Task :sdks:java:io:google-cloud-platform:classes UP-TO-DATE > Task :sdks:java:io:google-cloud-platform:jar > Task :runners:google-cloud-dataflow-java:compileJava FROM-CACHE > Task :runners:google-cloud-dataflow-java:classes > Task :runners:google-cloud-dataflow-java:jar > Task :runners:google-cloud-dataflow-java:****:legacy-****:compileJava > FROM-CACHE > Task :runners:google-cloud-dataflow-java:****:legacy-****:classes UP-TO-DATE > Task :runners:direct-java:shadowJar > Task :sdks:java:testing:load-tests:compileJava FROM-CACHE > Task :sdks:java:testing:load-tests:classes UP-TO-DATE > Task :sdks:java:testing:load-tests:jar > Task :runners:google-cloud-dataflow-java:****:legacy-****:shadowJar > Task :sdks:java:testing:load-tests:run Aug 17, 2020 12:05:31 PM org.apache.beam.runners.dataflow.options.DataflowPipelineOptions$StagingLocationFactory create INFO: No stagingLocation provided, falling back to gcpTempLocation Aug 17, 2020 12:05:31 PM org.apache.beam.runners.dataflow.DataflowRunner fromOptions INFO: PipelineOptions.filesToStage was not specified. Defaulting to files from the classpath: will stage 181 files. Enable logging at DEBUG level to see which files will be staged. Aug 17, 2020 12:05:32 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Executing pipeline on the Dataflow Service, which will have billing implications related to Google Compute Engine usage and other Google Cloud Services. Aug 17, 2020 12:05:34 PM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements INFO: Uploading 182 files from PipelineOptions.filesToStage to staging location to prepare for execution. Aug 17, 2020 12:05:34 PM org.apache.beam.runners.dataflow.util.PackageUtil$PackageAttributes forFileToStage INFO: Staging custom dataflow-****.jar as beam-runners-google-cloud-dataflow-java-legacy-****-2.25.0-SNAPSHOT-eaien-ilAfnYso-iJzIRyeaibYQrHnEuO6inqGv2Anw.jar Aug 17, 2020 12:05:34 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <https://ci-beam.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Streaming/ws/src/sdks/java/extensions/protobuf/build/libs/beam-sdks-java-extensions-protobuf-2.25.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/loadtests/staging/beam-sdks-java-extensions-protobuf-2.25.0-SNAPSHOT-DvZ_TWPGLfKH5B9Ay0phSqSgsI68AJCt20BG1hdNMh0.jar Aug 17, 2020 12:05:34 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <https://ci-beam.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Streaming/ws/src/sdks/java/core/build/libs/beam-sdks-java-core-2.25.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/loadtests/staging/beam-sdks-java-core-2.25.0-SNAPSHOT-MtUfsQz94adWwazOotfRLElj9hWA6rTRRn_0lwxMWoY.jar Aug 17, 2020 12:05:34 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <https://ci-beam.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Streaming/ws/src/sdks/java/io/google-cloud-platform/build/libs/beam-sdks-java-io-google-cloud-platform-2.25.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/loadtests/staging/beam-sdks-java-io-google-cloud-platform-2.25.0-SNAPSHOT-lilgtdBgfZ4S4Qp5eD-i9_ztY9NysaWq8_vhnaR7Acg.jar Aug 17, 2020 12:05:34 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <https://ci-beam.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Streaming/ws/src/sdks/java/extensions/google-cloud-platform-core/build/libs/beam-sdks-java-extensions-google-cloud-platform-core-2.25.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/loadtests/staging/beam-sdks-java-extensions-google-cloud-platform-core-2.25.0-SNAPSHOT-l8QKM-bLVdHiIdolL28BykRtAlnVIwibKk6q_4NuA4k.jar Aug 17, 2020 12:05:34 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <https://ci-beam.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Streaming/ws/src/model/pipeline/build/libs/beam-model-pipeline-2.25.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/loadtests/staging/beam-model-pipeline-2.25.0-SNAPSHOT-WHv1u8rJmiYNmZbzEU-SoKDflRAcMe7_-ukprJlK5EI.jar Aug 17, 2020 12:05:34 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <https://ci-beam.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Streaming/ws/src/sdks/java/io/kafka/build/libs/beam-sdks-java-io-kafka-2.25.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/loadtests/staging/beam-sdks-java-io-kafka-2.25.0-SNAPSHOT-HtXPSYEiOos7AWXmUwfB_7PMj3eDPyzKQSJMGzNZRZA.jar Aug 17, 2020 12:05:34 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <https://ci-beam.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Streaming/ws/src/runners/google-cloud-dataflow-java/build/libs/beam-runners-google-cloud-dataflow-java-2.25.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/loadtests/staging/beam-runners-google-cloud-dataflow-java-2.25.0-SNAPSHOT-0y9sHUaG8vFghgcccyIo7og2zdTh8aMsQoV4KXpn5z8.jar Aug 17, 2020 12:05:34 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <https://ci-beam.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Streaming/ws/src/sdks/java/fn-execution/build/libs/beam-sdks-java-fn-execution-2.25.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/loadtests/staging/beam-sdks-java-fn-execution-2.25.0-SNAPSHOT-Cv5eedL5IYUWm8Ba7Xk_NuwiQElBxdNZKOuuseVnVUU.jar Aug 17, 2020 12:05:34 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <https://ci-beam.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Streaming/ws/src/sdks/java/expansion-service/build/libs/beam-sdks-java-expansion-service-2.25.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/loadtests/staging/beam-sdks-java-expansion-service-2.25.0-SNAPSHOT-mqbrAHUiC3A_gHXrW_cLtCMAWuwi5VloG7s4ym7-zCA.jar Aug 17, 2020 12:05:34 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <https://ci-beam.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Streaming/ws/src/sdks/java/io/synthetic/build/libs/beam-sdks-java-io-synthetic-2.25.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/loadtests/staging/beam-sdks-java-io-synthetic-2.25.0-SNAPSHOT-il_0kr_xWT7_yoonRtGNJB7zwLJY5Ab3VnsdBjSiOqQ.jar Aug 17, 2020 12:05:34 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <https://ci-beam.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Streaming/ws/src/sdks/java/testing/load-tests/build/libs/beam-sdks-java-load-tests-2.25.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/loadtests/staging/beam-sdks-java-load-tests-2.25.0-SNAPSHOT-iAxpC98JXe9kXJ7bTXNS8CNl1B3-GK463sBDIr2nMvU.jar Aug 17, 2020 12:05:34 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <https://ci-beam.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Streaming/ws/src/runners/direct-java/build/libs/beam-runners-direct-java-2.25.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/loadtests/staging/beam-runners-direct-java-2.25.0-SNAPSHOT-22x52C4CTextBYZIEkC1GWphhMRXinZIqRk6xftCglU.jar Aug 17, 2020 12:05:34 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <https://ci-beam.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Streaming/ws/src/runners/core-construction-java/build/libs/beam-runners-core-construction-java-2.25.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/loadtests/staging/beam-runners-core-construction-java-2.25.0-SNAPSHOT-nzuPeqDbzMSJNHc80_CHZre9yYyg3EJkuPAGVMawv28.jar Aug 17, 2020 12:05:34 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <https://ci-beam.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Streaming/ws/src/runners/java-fn-execution/build/libs/beam-runners-java-fn-execution-2.25.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/loadtests/staging/beam-runners-java-fn-execution-2.25.0-SNAPSHOT-48GV8ALVBUwU4eSsyKTq_FuT0Ml6KuojNPUJx4aJ_Ps.jar Aug 17, 2020 12:05:34 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <https://ci-beam.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Streaming/ws/src/vendor/sdks-java-extensions-protobuf/build/libs/beam-vendor-sdks-java-extensions-protobuf-2.25.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/loadtests/staging/beam-vendor-sdks-java-extensions-protobuf-2.25.0-SNAPSHOT-v66NOsNcdFpp7z30pU40OcIihhDQdUvfmqQH78q0jaU.jar Aug 17, 2020 12:05:34 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <https://ci-beam.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Streaming/ws/src/sdks/java/testing/test-utils/build/libs/beam-sdks-java-test-utils-2.25.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/loadtests/staging/beam-sdks-java-test-utils-2.25.0-SNAPSHOT-aR-AOxEdoPMSOy4tXeAu8vixLrj3p33M1pT2uQGTs7M.jar Aug 17, 2020 12:05:34 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <https://ci-beam.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Streaming/ws/src/sdks/java/io/kinesis/build/libs/beam-sdks-java-io-kinesis-2.25.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/loadtests/staging/beam-sdks-java-io-kinesis-2.25.0-SNAPSHOT-Jq-YsyqXc1-sxILxYKiUnRqZqmtPyqQZuUUpAV4Vj3I.jar Aug 17, 2020 12:05:34 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <https://ci-beam.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Streaming/ws/src/model/fn-execution/build/libs/beam-model-fn-execution-2.25.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/loadtests/staging/beam-model-fn-execution-2.25.0-SNAPSHOT-O4MBvOb5IfuUQUq6drpkOHpGHsEiRPRhW4eAY-yWs4w.jar Aug 17, 2020 12:05:34 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <https://ci-beam.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Streaming/ws/src/model/job-management/build/libs/beam-model-job-management-2.25.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/loadtests/staging/beam-model-job-management-2.25.0-SNAPSHOT-Fgmq0VsFTABhkCnqERbcYkIau-lsMSqvVtcxohZGr_w.jar Aug 17, 2020 12:05:35 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <https://ci-beam.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Streaming/ws/src/runners/google-cloud-dataflow-java/****/legacy-****/build/libs/beam-runners-google-cloud-dataflow-java-legacy-****-2.25.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/loadtests/staging/beam-runners-google-cloud-dataflow-java-legacy-****-2.25.0-SNAPSHOT-eaien-ilAfnYso-iJzIRyeaibYQrHnEuO6inqGv2Anw.jar Aug 17, 2020 12:05:35 PM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements INFO: Staging files complete: 162 files cached, 20 files newly uploaded in 1 seconds Aug 17, 2020 12:05:35 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Read input/DataflowRunner.StreamingUnboundedRead.ReadWithIds as step s1 Aug 17, 2020 12:05:35 PM org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource split INFO: Split into 20 bundles of sizes: [org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@54f66455, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@65ae095c, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@4c03a37, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@2e140e59, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@49a71302, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@5cd61783, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@59429fac, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@57aa341b, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@26e412ef, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@4ae263bf, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@7103ab0, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@19ccca5, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@769d513, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@5e4fa1da, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@588cd519, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@3f92c349, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@769a58e5, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@55f8669d, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@6be25526, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@42435b98] Aug 17, 2020 12:05:35 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Read input/StripIds as step s2 Aug 17, 2020 12:05:35 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Collect start time metrics as step s3 Aug 17, 2020 12:05:35 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Total bytes monitor as step s4 Aug 17, 2020 12:05:35 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Window.Into()/Window.Assign as step s5 Aug 17, 2020 12:05:35 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Group by key (0) as step s6 Aug 17, 2020 12:05:35 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Ungroup and reiterate (0) as step s7 Aug 17, 2020 12:05:35 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Collect end time metrics (0) as step s8 Aug 17, 2020 12:05:35 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Staging pipeline description to gs://temp-storage-for-perf-tests/loadtests/staging/ Aug 17, 2020 12:05:35 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <79495 bytes, hash 6c0bc3d5dc73a2a2e42b33e89b1e5a7cc70e786753c0493a1d216bce716a66ad> to gs://temp-storage-for-perf-tests/loadtests/staging/pipeline-bAvD1dxzoqLkKzPomx5afMcOeGdTwEk6HSFrznFqZq0.pb Aug 17, 2020 12:05:36 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Dataflow SDK version: 2.25.0-SNAPSHOT Aug 17, 2020 12:05:37 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-17_05_05_36-17486832035498956912?project=apache-beam-testing Aug 17, 2020 12:05:37 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Submitted job: 2020-08-17_05_05_36-17486832035498956912 Aug 17, 2020 12:05:37 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: To cancel the job using the 'gcloud' tool, run: > gcloud dataflow jobs --project=apache-beam-testing cancel > --region=us-central1 2020-08-17_05_05_36-17486832035498956912 Aug 17, 2020 12:05:41 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process WARNING: 2020-08-17T12:05:39.563Z: The workflow name is not a valid Cloud Label. Labels applied to Cloud resources (such as GCE Instances) for monitoring will be labeled with this modified job name: load0tests0java0dataflow0streaming0gbk01-jenkins-081712053-nycn. For the best monitoring experience, please name your job with a valid Cloud Label. For details, see: https://cloud.google.com/compute/docs/labeling-resources#restrictions Aug 17, 2020 12:05:43 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-08-17T12:05:43.453Z: Worker configuration: n1-standard-4 in us-central1-f. Aug 17, 2020 12:05:46 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-08-17T12:05:44.096Z: Expanding CoGroupByKey operations into optimizable parts. Aug 17, 2020 12:05:46 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-08-17T12:05:44.110Z: Expanding SplittableProcessKeyed operations into optimizable parts. Aug 17, 2020 12:05:46 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-08-17T12:05:44.112Z: Expanding GroupByKey operations into streaming Read/Write steps Aug 17, 2020 12:05:46 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-08-17T12:05:44.118Z: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns Aug 17, 2020 12:05:46 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-08-17T12:05:44.141Z: Fusing adjacent ParDo, Read, Write, and Flatten operations Aug 17, 2020 12:05:46 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-08-17T12:05:44.143Z: Fusing consumer Read input/StripIds into Read input/DataflowRunner.StreamingUnboundedRead.ReadWithIds Aug 17, 2020 12:05:46 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-08-17T12:05:44.145Z: Fusing consumer Collect start time metrics into Read input/StripIds Aug 17, 2020 12:05:46 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-08-17T12:05:44.148Z: Fusing consumer Total bytes monitor into Collect start time metrics Aug 17, 2020 12:05:46 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-08-17T12:05:44.151Z: Fusing consumer Window.Into()/Window.Assign into Total bytes monitor Aug 17, 2020 12:05:46 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-08-17T12:05:44.155Z: Fusing consumer Group by key (0)/WriteStream into Window.Into()/Window.Assign Aug 17, 2020 12:05:46 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-08-17T12:05:44.158Z: Fusing consumer Group by key (0)/MergeBuckets into Group by key (0)/ReadStream Aug 17, 2020 12:05:46 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-08-17T12:05:44.161Z: Fusing consumer Ungroup and reiterate (0) into Group by key (0)/MergeBuckets Aug 17, 2020 12:05:46 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-08-17T12:05:44.163Z: Fusing consumer Collect end time metrics (0) into Ungroup and reiterate (0) Aug 17, 2020 12:05:46 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-08-17T12:05:44.531Z: Starting 5 ****s... Aug 17, 2020 12:05:49 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-08-17T12:05:46.811Z: Executing operation Group by key (0)/ReadStream+Group by key (0)/MergeBuckets+Ungroup and reiterate (0)+Collect end time metrics (0) Aug 17, 2020 12:05:49 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-08-17T12:05:46.814Z: Executing operation Read input/DataflowRunner.StreamingUnboundedRead.ReadWithIds+Read input/StripIds+Collect start time metrics+Total bytes monitor+Window.Into()/Window.Assign+Group by key (0)/WriteStream Aug 17, 2020 12:05:51 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process WARNING: 2020-08-17T12:05:50.697Z: Your project already contains 100 Dataflow-created metric descriptors and Stackdriver will not create new Dataflow custom metrics for this job. Each unique user-defined metric name (independent of the DoFn in which it is defined) produces a new metric descriptor. To delete old / unused metric descriptors see https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete Aug 17, 2020 12:06:03 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-08-17T12:06:02.409Z: Unable to bring up enough ****s. Will retry in 5 seconds. Aug 17, 2020 12:06:08 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-08-17T12:06:08.258Z: Unable to bring up enough ****s. Will retry in 5 seconds. Aug 17, 2020 12:06:14 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-08-17T12:06:14.204Z: Unable to bring up enough ****s. Will retry in 5 seconds. Aug 17, 2020 12:06:21 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-08-17T12:06:19.991Z: Unable to bring up enough ****s. Will retry in 5 seconds. Aug 17, 2020 12:06:27 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-08-17T12:06:25.747Z: Unable to bring up enough ****s. Will retry in 5 seconds. Aug 17, 2020 12:06:31 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-08-17T12:06:31.479Z: Unable to bring up enough ****s. Will retry in 5 seconds. Aug 17, 2020 12:06:37 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-08-17T12:06:37.278Z: Unable to bring up enough ****s. Will retry in 5 seconds. Aug 17, 2020 12:06:42 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-08-17T12:06:41.827Z: Workers have started successfully. Aug 17, 2020 12:06:43 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-08-17T12:06:42.964Z: Unable to bring up enough ****s. Will retry in 5 seconds. Aug 17, 2020 12:06:49 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-08-17T12:06:48.779Z: Unable to bring up enough ****s. Will retry in 5 seconds. Aug 17, 2020 12:06:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-08-17T12:06:54.485Z: Unable to bring up enough ****s. Will retry in 5 seconds. Aug 17, 2020 12:08:08 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process SEVERE: 2020-08-17T12:08:07.998Z: Workflow failed. Causes: Error: Message: Quota exceeded for quota group 'ReadGroup' and limit 'Read requests per 100 seconds' of service 'compute.googleapis.com' for consumer 'project_number:844138762903'. HTTP Code: 403 Aug 17, 2020 12:08:08 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-08-17T12:08:08.036Z: Finished operation Read input/DataflowRunner.StreamingUnboundedRead.ReadWithIds+Read input/StripIds+Collect start time metrics+Total bytes monitor+Window.Into()/Window.Assign+Group by key (0)/WriteStream Aug 17, 2020 12:08:08 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-08-17T12:08:08.040Z: Finished operation Group by key (0)/ReadStream+Group by key (0)/MergeBuckets+Ungroup and reiterate (0)+Collect end time metrics (0) Aug 17, 2020 12:08:08 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-08-17T12:08:08.161Z: Cleaning up. Aug 17, 2020 12:08:08 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-08-17T12:08:08.187Z: Worker pool stopped. Aug 17, 2020 12:08:09 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-08-17T12:08:08.204Z: Stopping **** pool... Aug 17, 2020 12:08:09 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-08-17T12:08:08.216Z: Worker pool stopped. Aug 17, 2020 12:08:17 PM org.apache.beam.runners.dataflow.DataflowPipelineJob logTerminalState INFO: Job 2020-08-17_05_05_36-17486832035498956912 failed with status FAILED. Aug 17, 2020 12:08:18 PM org.apache.beam.sdk.testutils.metrics.MetricsReader getCounterMetric SEVERE: Failed to get metric totalBytes.count, from namespace gbk Load test results for test (ID): 21bdea59-0e3f-4af3-8831-4b0ba4ae8fda and timestamp: 2020-08-17T12:05:32.093000000Z: Metric: Value: dataflow_runtime_sec 0.0 dataflow_total_bytes_count -1.0 Exception in thread "main" java.lang.RuntimeException: Invalid job state: FAILED. at org.apache.beam.sdk.loadtests.JobFailure.handleFailure(JobFailure.java:55) at org.apache.beam.sdk.loadtests.LoadTest.run(LoadTest.java:134) at org.apache.beam.sdk.loadtests.GroupByKeyLoadTest.run(GroupByKeyLoadTest.java:57) at org.apache.beam.sdk.loadtests.GroupByKeyLoadTest.main(GroupByKeyLoadTest.java:131) > Task :sdks:java:testing:load-tests:run FAILED FAILURE: Build failed with an exception. * What went wrong: Execution failed for task ':sdks:java:testing:load-tests:run'. > Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with > non-zero exit value 1 * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. * Get more help at https://help.gradle.org Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings BUILD FAILED in 3m 40s 73 actionable tasks: 48 executed, 25 from cache Publishing build scan... https://gradle.com/s/5iqjcyzmacir2 Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
