See <https://ci-beam.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_V2_Streaming_Java11/614/display/redirect>
Changes: ------------------------------------------ [...truncated 50.44 KB...] google.golang.org/grpc,https://github.com/grpc/grpc-go/blob/v1.52.0/LICENSE,Apache-2.0 google.golang.org/protobuf,https://github.com/protocolbuffers/protobuf-go/blob/v1.28.1/LICENSE,BSD-3-Clause + chmod -R a+w /output/licenses > Task :sdks:java:container:java11:copySdkHarnessLauncher Execution optimizations have been disabled for task ':sdks:java:container:java11:copySdkHarnessLauncher' to ensure correctness due to the following reasons: - Gradle detected a problem with the following location: '/home/jenkins/jenkins-slave/workspace/beam_LoadTests_Java_GBK_Dataflow_V2_Streaming_Java11/src/sdks/java/container/build/target'. Reason: Task ':sdks:java:container:java11:copySdkHarnessLauncher' uses this output of task ':sdks:java:container:downloadCloudProfilerAgent' without declaring an explicit or implicit dependency. This can lead to incorrect results being produced, depending on what order the tasks are executed. Please refer to https://docs.gradle.org/7.5.1/userguide/validation_problems.html#implicit_dependency for more details about this problem. - Gradle detected a problem with the following location: '/home/jenkins/jenkins-slave/workspace/beam_LoadTests_Java_GBK_Dataflow_V2_Streaming_Java11/src/sdks/java/container/build/target'. Reason: Task ':sdks:java:container:java11:copySdkHarnessLauncher' uses this output of task ':sdks:java:container:pullLicenses' without declaring an explicit or implicit dependency. This can lead to incorrect results being produced, depending on what order the tasks are executed. Please refer to https://docs.gradle.org/7.5.1/userguide/validation_problems.html#implicit_dependency for more details about this problem. > Task :release:go-licenses:java:createLicenses > Task :sdks:java:container:java11:copyGolangLicenses > Task :sdks:java:container:java11:dockerPrepare > Task :sdks:java:container:java11:docker > Task :runners:google-cloud-dataflow-java:buildAndPushDockerJavaContainer WARNING: `gcloud docker` will not be supported for Docker client versions above 18.03. As an alternative, use `gcloud auth configure-docker` to configure `docker` to use `gcloud` as a credential helper, then use `docker` as you would for non-GCR registries, e.g. `docker pull gcr.io/project-id/my-image`. Add `--verbosity=error` to silence this warning: `gcloud docker --verbosity=error -- pull gcr.io/project-id/my-image`. See: https://cloud.google.com/container-registry/docs/support/deprecation-notices#gcloud-docker The push refers to repository [us.gcr.io/apache-beam-testing/java-postcommit-it/java] 610fb57a2fa0: Preparing 06c05f2ff16e: Preparing b051b83ddc15: Preparing 732a2503d2fc: Preparing 6cb22526c79d: Preparing af1cc1edc241: Preparing e454a01873e6: Preparing cb0d368a8521: Preparing 71037e1e9917: Preparing 9f88ff1e36e7: Preparing 7328e8782822: Preparing 66cddb5cb437: Preparing 22c549729912: Preparing 5e400bcae5ae: Preparing de6cc97e5d43: Preparing 6a78856b640d: Preparing 6f27936daf95: Preparing 2a2545560d49: Preparing 862320036867: Preparing e454a01873e6: Waiting cb0d368a8521: Waiting 5e400bcae5ae: Waiting de6cc97e5d43: Waiting 6f27936daf95: Waiting 6a78856b640d: Waiting 9f88ff1e36e7: Waiting 66cddb5cb437: Waiting 71037e1e9917: Waiting c5ff2d88f679: Preparing 7328e8782822: Waiting 2a2545560d49: Waiting 862320036867: Waiting 732a2503d2fc: Pushed 06c05f2ff16e: Pushed 6cb22526c79d: Pushed b051b83ddc15: Pushed 610fb57a2fa0: Pushed cb0d368a8521: Pushed e454a01873e6: Pushed af1cc1edc241: Pushed 9f88ff1e36e7: Pushed 71037e1e9917: Pushed 66cddb5cb437: Pushed 7328e8782822: Pushed 6f27936daf95: Layer already exists 2a2545560d49: Layer already exists 862320036867: Layer already exists c5ff2d88f679: Layer already exists 5e400bcae5ae: Pushed 22c549729912: Pushed de6cc97e5d43: Pushed 6a78856b640d: Pushed 20230224120458: digest: sha256:8ba084c0746b68f2460236639220eac9644243a8e088c4033b5033c5ea87390d size: 4506 > Task :sdks:java:testing:load-tests:run Feb 24, 2023 12:05:52 PM org.apache.beam.runners.dataflow.options.DataflowPipelineOptions$StagingLocationFactory create INFO: No stagingLocation provided, falling back to gcpTempLocation Feb 24, 2023 12:05:52 PM org.apache.beam.runners.dataflow.DataflowRunner fromOptions INFO: PipelineOptions.filesToStage was not specified. Defaulting to files from the classpath: will stage 233 files. Enable logging at DEBUG level to see which files will be staged. Feb 24, 2023 12:05:53 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Executing pipeline on the Dataflow Service, which will have billing implications related to Google Compute Engine usage and other Google Cloud Services. Feb 24, 2023 12:05:55 PM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements INFO: Uploading 233 files from PipelineOptions.filesToStage to staging location to prepare for execution. Feb 24, 2023 12:05:56 PM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements INFO: Staging files complete: 233 files cached, 0 files newly uploaded in 0 seconds Feb 24, 2023 12:05:56 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Staging portable pipeline proto to gs://temp-storage-for-perf-tests/loadtests/staging/ Feb 24, 2023 12:05:56 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <102193 bytes, hash e72c899ed6962ff5a50b4e6a63701b3f6982c7561d01589c74b913b43988f9a7> to gs://temp-storage-for-perf-tests/loadtests/staging/pipeline-5yyJntaWL_WlC05qY3AbP2mCx1YdAVicdLkTtDmI-ac.pb Feb 24, 2023 12:05:58 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Read input/DataflowRunner.StreamingUnboundedRead.ReadWithIds as step s1 Feb 24, 2023 12:05:58 PM org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource split INFO: Split into 20 bundles: [SyntheticUnboundedSource{startOffset=0, endOffset=10000000}, SyntheticUnboundedSource{startOffset=10000000, endOffset=20000000}, SyntheticUnboundedSource{startOffset=20000000, endOffset=30000000}, SyntheticUnboundedSource{startOffset=30000000, endOffset=40000000}, SyntheticUnboundedSource{startOffset=40000000, endOffset=50000000}, SyntheticUnboundedSource{startOffset=50000000, endOffset=60000000}, SyntheticUnboundedSource{startOffset=60000000, endOffset=70000000}, SyntheticUnboundedSource{startOffset=70000000, endOffset=80000000}, SyntheticUnboundedSource{startOffset=80000000, endOffset=90000000}, SyntheticUnboundedSource{startOffset=90000000, endOffset=100000000}, SyntheticUnboundedSource{startOffset=100000000, endOffset=110000000}, SyntheticUnboundedSource{startOffset=110000000, endOffset=120000000}, SyntheticUnboundedSource{startOffset=120000000, endOffset=130000000}, SyntheticUnboundedSource{startOffset=130000000, endOffset=140000000}, SyntheticUnboundedSource{startOffset=140000000, endOffset=150000000}, SyntheticUnboundedSource{startOffset=150000000, endOffset=160000000}, SyntheticUnboundedSource{startOffset=160000000, endOffset=170000000}, SyntheticUnboundedSource{startOffset=170000000, endOffset=180000000}, SyntheticUnboundedSource{startOffset=180000000, endOffset=190000000}, SyntheticUnboundedSource{startOffset=190000000, endOffset=200000000}] Feb 24, 2023 12:05:58 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Read input/StripIds as step s2 Feb 24, 2023 12:05:58 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Collect start time metrics as step s3 Feb 24, 2023 12:05:58 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Total bytes monitor as step s4 Feb 24, 2023 12:05:58 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Window.Into()/Window.Assign as step s5 Feb 24, 2023 12:05:58 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Group by key (0) as step s6 Feb 24, 2023 12:05:58 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Ungroup and reiterate (0) as step s7 Feb 24, 2023 12:05:58 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Collect end time metrics (0) as step s8 Feb 24, 2023 12:05:58 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Dataflow SDK version: 2.47.0-SNAPSHOT Feb 24, 2023 12:05:59 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2023-02-24_04_05_58-8738926323315600543?project=apache-beam-testing Feb 24, 2023 12:05:59 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Submitted job: 2023-02-24_04_05_58-8738926323315600543 Feb 24, 2023 12:05:59 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: To cancel the job using the 'gcloud' tool, run: > gcloud dataflow jobs --project=apache-beam-testing cancel > --region=us-central1 2023-02-24_04_05_58-8738926323315600543 Feb 24, 2023 12:06:11 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process WARNING: 2023-02-24T12:06:09.692Z: The workflow name is not a valid Cloud Label. Labels applied to Cloud resources (such as GCE Instances) for monitoring will be labeled with this modified job name: load0tests0java110dataflow0v20streaming0gbk01-jenkins-0224-3vmp. For the best monitoring experience, please name your job with a valid Cloud Label. For details, see: https://cloud.google.com/compute/docs/labeling-resources#restrictions Feb 24, 2023 12:06:17 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2023-02-24T12:06:16.976Z: Worker configuration: e2-standard-2 in us-central1-f. Feb 24, 2023 12:06:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2023-02-24T12:06:18.184Z: Expanding SplittableParDo operations into optimizable parts. Feb 24, 2023 12:06:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2023-02-24T12:06:18.216Z: Expanding CollectionToSingleton operations into optimizable parts. Feb 24, 2023 12:06:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2023-02-24T12:06:18.280Z: Expanding CoGroupByKey operations into optimizable parts. Feb 24, 2023 12:06:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2023-02-24T12:06:18.376Z: Expanding SplittableProcessKeyed operations into optimizable parts. Feb 24, 2023 12:06:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2023-02-24T12:06:18.409Z: Expanding GroupByKey operations into streaming Read/Write steps Feb 24, 2023 12:06:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2023-02-24T12:06:18.472Z: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns Feb 24, 2023 12:06:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2023-02-24T12:06:18.573Z: Fusing adjacent ParDo, Read, Write, and Flatten operations Feb 24, 2023 12:06:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2023-02-24T12:06:18.601Z: Fusing consumer Read input/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource) into Read input/Impulse Feb 24, 2023 12:06:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2023-02-24T12:06:18.632Z: Fusing consumer Read-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/PairWithRestriction into Read input/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource) Feb 24, 2023 12:06:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2023-02-24T12:06:18.653Z: Fusing consumer Read-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/SplitWithSizing into Read-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/PairWithRestriction Feb 24, 2023 12:06:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2023-02-24T12:06:18.684Z: Fusing consumer Read input/ParDo(StripIds)/ParMultiDo(StripIds) into Read-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/ProcessElementAndRestrictionWithSizing Feb 24, 2023 12:06:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2023-02-24T12:06:18.705Z: Fusing consumer Collect start time metrics/ParMultiDo(TimeMonitor) into Read input/ParDo(StripIds)/ParMultiDo(StripIds) Feb 24, 2023 12:06:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2023-02-24T12:06:18.733Z: Fusing consumer Total bytes monitor/ParMultiDo(ByteMonitor) into Collect start time metrics/ParMultiDo(TimeMonitor) Feb 24, 2023 12:06:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2023-02-24T12:06:18.765Z: Fusing consumer Window.Into()/Window.Assign into Total bytes monitor/ParMultiDo(ByteMonitor) Feb 24, 2023 12:06:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2023-02-24T12:06:18.800Z: Fusing consumer Group by key (0)/WriteStream into Window.Into()/Window.Assign Feb 24, 2023 12:06:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2023-02-24T12:06:18.835Z: Fusing consumer Group by key (0)/MergeBuckets into Group by key (0)/ReadStream Feb 24, 2023 12:06:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2023-02-24T12:06:18.867Z: Fusing consumer Ungroup and reiterate (0)/ParMultiDo(UngroupAndReiterate) into Group by key (0)/MergeBuckets Feb 24, 2023 12:06:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2023-02-24T12:06:18.904Z: Fusing consumer Collect end time metrics (0)/ParMultiDo(TimeMonitor) into Ungroup and reiterate (0)/ParMultiDo(UngroupAndReiterate) Feb 24, 2023 12:06:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2023-02-24T12:06:18.995Z: Running job using Streaming Engine Feb 24, 2023 12:06:21 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2023-02-24T12:06:20.245Z: Starting 5 ****s in us-central1-f... Feb 24, 2023 12:06:24 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2023-02-24T12:06:23.676Z: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete Feb 24, 2023 12:07:03 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2023-02-24T12:07:01.359Z: Autoscaling: Raised the number of ****s to 5 so that the pipeline can catch up with its backlog and keep up with its input rate. Feb 24, 2023 12:07:49 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2023-02-24T12:07:49.353Z: All ****s have finished the startup processes and began to receive work requests. Feb 24, 2023 12:07:52 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2023-02-24T12:07:50.493Z: Workers have started successfully. Feb 24, 2023 12:20:28 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2023-02-24T12:20:27.712Z: Cleaning up. Feb 24, 2023 12:20:28 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2023-02-24T12:20:27.824Z: Stopping **** pool... Feb 24, 2023 12:20:28 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2023-02-24T12:20:27.879Z: Stopping **** pool... Feb 24, 2023 12:22:39 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2023-02-24T12:22:39.054Z: Autoscaling: Reduced the number of ****s to 0 based on low average **** CPU utilization, and the pipeline having sufficiently low backlog and keeping up with input rate. Feb 24, 2023 12:22:39 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2023-02-24T12:22:39.104Z: Worker pool stopped. Feb 24, 2023 12:22:46 PM org.apache.beam.runners.dataflow.DataflowPipelineJob logTerminalState INFO: Job 2023-02-24_04_05_58-8738926323315600543 finished with status DONE. Load test results for test (ID): ec5ef975-bed3-4d0b-9ab9-f518892d6d69 and timestamp: 2023-02-24T12:05:53.174000000Z: Metric: Value: dataflow_v2_java11_runtime_sec 645.941 dataflow_v2_java11_total_bytes_count 1.9999998E9 > Task :runners:google-cloud-dataflow-java:cleanUpDockerJavaImages Untagged: us.gcr.io/apache-beam-testing/java-postcommit-it/java:20230224120458 Untagged: us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:8ba084c0746b68f2460236639220eac9644243a8e088c4033b5033c5ea87390d WARNING: Successfully resolved tag to sha256, but it is recommended to use sha256 directly. Tag: [us.gcr.io/apache-beam-testing/java-postcommit-it/java:20230224120458] - referencing digest: [us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:8ba084c0746b68f2460236639220eac9644243a8e088c4033b5033c5ea87390d] Deleted [[us.gcr.io/apache-beam-testing/java-postcommit-it/java:20230224120458] (referencing [us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:8ba084c0746b68f2460236639220eac9644243a8e088c4033b5033c5ea87390d])]. Removing untagged image us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:8ba084c0746b68f2460236639220eac9644243a8e088c4033b5033c5ea87390d Digests: - us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:8ba084c0746b68f2460236639220eac9644243a8e088c4033b5033c5ea87390d Deleted [us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:8ba084c0746b68f2460236639220eac9644243a8e088c4033b5033c5ea87390d]. Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0. You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins. See https://docs.gradle.org/7.5.1/userguide/command_line_interface.html#sec:command_line_warnings Execution optimizations have been disabled for 1 invalid unit(s) of work during this build to ensure correctness. Please consult deprecation warnings for more details. BUILD SUCCESSFUL in 18m 11s 117 actionable tasks: 78 executed, 35 from cache, 4 up-to-date Publishing build scan... https://gradle.com/s/qk4qtdfjwe2bw Build step 'Invoke Gradle script' changed build result to SUCCESS [beam_LoadTests_Java_GBK_Dataflow_V2_Streaming_Java11] $ /bin/bash -xe /tmp/jenkins1139036736644465353.sh + echo '*** Load test: 2GB of 100B records ***' *** Load test: 2GB of 100B records *** [Gradle] - Launching build. FATAL: command execution failed java.io.InterruptedIOException at hudson.Launcher$RemoteLauncher.launch(Launcher.java:1126) at hudson.Launcher$ProcStarter.start(Launcher.java:509) at hudson.Launcher$ProcStarter.join(Launcher.java:520) at hudson.plugins.gradle.Gradle.perform(Gradle.java:317) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:814) at hudson.model.Build$BuildExecution.build(Build.java:199) at hudson.model.Build$BuildExecution.doRun(Build.java:164) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:522) at hudson.model.Run.execute(Run.java:1896) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44) at hudson.model.ResourceController.execute(ResourceController.java:101) at hudson.model.Executor.run(Executor.java:442) Caused by: java.lang.InterruptedException at java.base/java.util.concurrent.FutureTask.awaitDone(FutureTask.java:418) at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:190) at hudson.remoting.Request.call(Request.java:185) at hudson.remoting.Channel.call(Channel.java:999) at hudson.Launcher$RemoteLauncher.launch(Launcher.java:1124) ... 12 more Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure ERROR: apache-beam-jenkins-6 is offline; cannot locate jdk_1.8_latest --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
