See <https://ci-beam.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_V2_Batch_Java11/516/display/redirect?page=changes>
Changes: [Moritz Mack] [Metrics] Add new performance dashboard for Java JMH benchmarks (closes [Kenneth Knowles] Fix checkArgument format string in ByteKeyRange [Yi Hu] Unify test parameters for certain IOs based on test row and grafana [noreply] Bump github.com/aws/aws-sdk-go-v2/config from 1.18.0 to 1.18.1 in /sdks [noreply] Add enableGzipHttpCompression option to SplunkIO (#24197) [noreply] [Playground] Examples CI restore (#24155) [noreply] Bump github.com/aws/aws-sdk-go-v2/service/s3 in /sdks (#24220) [noreply] Issue#24161 Updated docstring for Clusters class [Kenneth Knowles] Fix checkArgument format string in BigQueryQueryHelper [Kenneth Knowles] Fix checkArgument calls in BQ dynamic destinations [noreply] Force discarding mode in with_fanout without rewindowing. (#23828) [noreply] Removed trailing whitespaces. [noreply] Clarify that SDF authors need to make the restriction sizing method [noreply] Remove google-cloud-platform-core dependency from harness (#24235) [noreply] Document our benchmarks (#24216) [noreply] Website updates for Beam 2.43.0 release (#24044) [Kenneth Knowles] Add @RunWith annotation to BQ test class [chamikaramj] Fix release date [chamikaramj] Few more fixes to the Website ------------------------------------------ [...truncated 171.61 KB...] f892d183379f: Preparing 938a1fb80088: Preparing 791d2793baa2: Preparing e6b8e97cb540: Preparing a4b779c24351: Preparing 7efc960bdb51: Preparing 6fdc50209b5f: Preparing fa362e0aa3d2: Preparing 7b7f3078e1db: Preparing 826c3ddbb29c: Preparing b626401ef603: Preparing 9b55156abf26: Preparing 293d5db30c9f: Preparing 03127cdb479b: Preparing 9c742cd6c7a5: Preparing 938a1fb80088: Waiting 791d2793baa2: Waiting 7080f8d055d5: Waiting d2911f287f2b: Waiting 826c3ddbb29c: Waiting 03127cdb479b: Waiting 9b55156abf26: Waiting 6fdc50209b5f: Waiting 9c742cd6c7a5: Waiting e6b8e97cb540: Waiting b626401ef603: Waiting 293d5db30c9f: Waiting 7b7f3078e1db: Waiting fa362e0aa3d2: Waiting f892d183379f: Waiting 7efc960bdb51: Waiting 575bf08859ed: Pushed c41a2170483e: Pushed d6d79b5986db: Pushed 453f7253b19a: Pushed e815a98c9af7: Pushed d2911f287f2b: Pushed f892d183379f: Pushed 791d2793baa2: Pushed 7080f8d055d5: Pushed 938a1fb80088: Pushed a4b779c24351: Pushed 7b7f3078e1db: Layer already exists e6b8e97cb540: Pushed 826c3ddbb29c: Layer already exists b626401ef603: Layer already exists 9b55156abf26: Layer already exists 293d5db30c9f: Layer already exists 03127cdb479b: Layer already exists 9c742cd6c7a5: Layer already exists 6fdc50209b5f: Pushed fa362e0aa3d2: Pushed 7efc960bdb51: Pushed 20221118141716: digest: sha256:48f57455d0e07f3674ca0719cb015e58b878c2ed0de0c6c03cdf8becbb6369a7 size: 4935 > Task :sdks:java:testing:load-tests:run Nov 18, 2022 2:17:43 PM org.apache.beam.runners.dataflow.options.DataflowPipelineOptions$StagingLocationFactory create INFO: No stagingLocation provided, falling back to gcpTempLocation Nov 18, 2022 2:17:44 PM org.apache.beam.runners.dataflow.DataflowRunner fromOptions INFO: PipelineOptions.filesToStage was not specified. Defaulting to files from the classpath: will stage 229 files. Enable logging at DEBUG level to see which files will be staged. Nov 18, 2022 2:17:44 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Executing pipeline on the Dataflow Service, which will have billing implications related to Google Compute Engine usage and other Google Cloud Services. Nov 18, 2022 2:17:46 PM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements INFO: Uploading 229 files from PipelineOptions.filesToStage to staging location to prepare for execution. Nov 18, 2022 2:17:47 PM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements INFO: Staging files complete: 229 files cached, 0 files newly uploaded in 0 seconds Nov 18, 2022 2:17:47 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Staging portable pipeline proto to gs://temp-storage-for-perf-tests/loadtests/staging/ Nov 18, 2022 2:17:47 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <97429 bytes, hash a6c83ccf5ccb6ff3c028f354f775f7aa4e4c3204ab8ed506db102b1fb0929219> to gs://temp-storage-for-perf-tests/loadtests/staging/pipeline-psg8z1zLb_PAKPNU93X3qk5MMgSrjtUG2xArH7CSkhk.pb Nov 18, 2022 2:17:49 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Read input as step s1 Nov 18, 2022 2:17:49 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Collect start time metrics as step s2 Nov 18, 2022 2:17:49 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Total bytes monitor as step s3 Nov 18, 2022 2:17:49 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Window.Into()/Window.Assign as step s4 Nov 18, 2022 2:17:49 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Group by key (0) as step s5 Nov 18, 2022 2:17:49 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Ungroup and reiterate (0) as step s6 Nov 18, 2022 2:17:49 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Collect end time metrics (0) as step s7 Nov 18, 2022 2:17:49 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Dataflow SDK version: 2.44.0-SNAPSHOT Nov 18, 2022 2:17:50 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2022-11-18_06_17_49-10959454474318434488?project=apache-beam-testing Nov 18, 2022 2:17:50 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Submitted job: 2022-11-18_06_17_49-10959454474318434488 Nov 18, 2022 2:17:50 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: To cancel the job using the 'gcloud' tool, run: > gcloud dataflow jobs --project=apache-beam-testing cancel > --region=us-central1 2022-11-18_06_17_49-10959454474318434488 Nov 18, 2022 2:17:52 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process WARNING: 2022-11-18T14:17:52.308Z: The workflow name is not a valid Cloud Label. Labels applied to Cloud resources (such as GCE Instances) for monitoring will be labeled with this modified job name: load0tests0java110dataflow0v20batch0gbk03-jenkins-11181417-mm82. For the best monitoring experience, please name your job with a valid Cloud Label. For details, see: https://cloud.google.com/compute/docs/labeling-resources#restrictions Nov 18, 2022 2:18:01 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-11-18T14:17:58.852Z: Worker configuration: e2-standard-2 in us-central1-b. Nov 18, 2022 2:18:01 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-11-18T14:17:59.857Z: Expanding SplittableParDo operations into optimizable parts. Nov 18, 2022 2:18:01 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-11-18T14:17:59.891Z: Expanding CollectionToSingleton operations into optimizable parts. Nov 18, 2022 2:18:01 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-11-18T14:17:59.953Z: Expanding CoGroupByKey operations into optimizable parts. Nov 18, 2022 2:18:01 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-11-18T14:18:00.012Z: Expanding GroupByKey operations into optimizable parts. Nov 18, 2022 2:18:01 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-11-18T14:18:00.093Z: Fusing adjacent ParDo, Read, Write, and Flatten operations Nov 18, 2022 2:18:01 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-11-18T14:18:00.125Z: Fusing consumer Read input/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource) into Read input/Impulse Nov 18, 2022 2:18:01 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-11-18T14:18:00.162Z: Fusing consumer Read-input-ParDo-BoundedSourceAsSDFWrapper--ParMultiDo-BoundedSourceAsSDFWrapper-/PairWithRestriction into Read input/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource) Nov 18, 2022 2:18:01 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-11-18T14:18:00.198Z: Fusing consumer Read-input-ParDo-BoundedSourceAsSDFWrapper--ParMultiDo-BoundedSourceAsSDFWrapper-/SplitWithSizing into Read-input-ParDo-BoundedSourceAsSDFWrapper--ParMultiDo-BoundedSourceAsSDFWrapper-/PairWithRestriction Nov 18, 2022 2:18:01 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-11-18T14:18:00.227Z: Fusing consumer Collect start time metrics/ParMultiDo(TimeMonitor) into Read-input-ParDo-BoundedSourceAsSDFWrapper--ParMultiDo-BoundedSourceAsSDFWrapper-/ProcessElementAndRestrictionWithSizing Nov 18, 2022 2:18:01 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-11-18T14:18:00.267Z: Fusing consumer Total bytes monitor/ParMultiDo(ByteMonitor) into Collect start time metrics/ParMultiDo(TimeMonitor) Nov 18, 2022 2:18:01 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-11-18T14:18:00.299Z: Fusing consumer Window.Into()/Window.Assign into Total bytes monitor/ParMultiDo(ByteMonitor) Nov 18, 2022 2:18:01 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-11-18T14:18:00.328Z: Fusing consumer Group by key (0)/Write into Window.Into()/Window.Assign Nov 18, 2022 2:18:01 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-11-18T14:18:00.363Z: Fusing consumer Ungroup and reiterate (0)/ParMultiDo(UngroupAndReiterate) into Group by key (0)/Read Nov 18, 2022 2:18:01 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-11-18T14:18:00.401Z: Fusing consumer Collect end time metrics (0)/ParMultiDo(TimeMonitor) into Ungroup and reiterate (0)/ParMultiDo(UngroupAndReiterate) Nov 18, 2022 2:18:01 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-11-18T14:18:00.754Z: Executing operation Read input/Impulse+Read input/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource)+Read-input-ParDo-BoundedSourceAsSDFWrapper--ParMultiDo-BoundedSourceAsSDFWrapper-/PairWithRestriction+Read-input-ParDo-BoundedSourceAsSDFWrapper--ParMultiDo-BoundedSourceAsSDFWrapper-/SplitWithSizing Nov 18, 2022 2:18:01 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-11-18T14:18:00.840Z: Starting 5 ****s in us-central1-b... Nov 18, 2022 2:18:14 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-11-18T14:18:13.330Z: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete Nov 18, 2022 2:18:46 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-11-18T14:18:46.087Z: Autoscaling: Raised the number of ****s to 5 based on the rate of progress in the currently running stage(s). > Task :sdks:java:testing:load-tests:run FAILED > Task :runners:google-cloud-dataflow-java:cleanUpDockerJavaImages Untagged: us.gcr.io/apache-beam-testing/java-postcommit-it/java:20221118141716 Untagged: us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:48f57455d0e07f3674ca0719cb015e58b878c2ed0de0c6c03cdf8becbb6369a7 WARNING: Successfully resolved tag to sha256, but it is recommended to use sha256 directly. Tag: [us.gcr.io/apache-beam-testing/java-postcommit-it/java:20221118141716] - referencing digest: [us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:48f57455d0e07f3674ca0719cb015e58b878c2ed0de0c6c03cdf8becbb6369a7] Deleted [[us.gcr.io/apache-beam-testing/java-postcommit-it/java:20221118141716] (referencing [us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:48f57455d0e07f3674ca0719cb015e58b878c2ed0de0c6c03cdf8becbb6369a7])]. Remove shutdown hook failed java.lang.IllegalStateException: Shutdown in progress at java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:82) at java.lang.Runtime.removeShutdownHook(Runtime.java:231) at org.gradle.process.internal.shutdown.ShutdownHooks.removeShutdownHook(ShutdownHooks.java:38) at org.gradle.process.internal.DefaultExecHandle.setEndStateInfo(DefaultExecHandle.java:208) at org.gradle.process.internal.DefaultExecHandle.aborted(DefaultExecHandle.java:366) at org.gradle.process.internal.ExecHandleRunner.completed(ExecHandleRunner.java:108) at org.gradle.process.internal.ExecHandleRunner.run(ExecHandleRunner.java:84) at org.gradle.internal.operations.CurrentBuildOperationPreservingRunnable.run(CurrentBuildOperationPreservingRunnable.java:38) at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64) at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Remove shutdown hook failed java.lang.IllegalStateException: Shutdown in progress at java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:82) at java.lang.Runtime.removeShutdownHook(Runtime.java:231) at org.gradle.process.internal.shutdown.ShutdownHooks.removeShutdownHook(ShutdownHooks.java:38) at org.gradle.process.internal.DefaultExecHandle.setEndStateInfo(DefaultExecHandle.java:208) at org.gradle.process.internal.DefaultExecHandle.aborted(DefaultExecHandle.java:366) at org.gradle.process.internal.ExecHandleRunner.completed(ExecHandleRunner.java:108) at org.gradle.process.internal.ExecHandleRunner.run(ExecHandleRunner.java:84) at org.gradle.internal.operations.CurrentBuildOperationPreservingRunnable.run(CurrentBuildOperationPreservingRunnable.java:38) at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64) at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Remove shutdown hook failed java.lang.IllegalStateException: Shutdown in progress at java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:82) at java.lang.Runtime.removeShutdownHook(Runtime.java:231) at org.gradle.process.internal.shutdown.ShutdownHooks.removeShutdownHook(ShutdownHooks.java:38) at org.gradle.process.internal.DefaultExecHandle.setEndStateInfo(DefaultExecHandle.java:208) at org.gradle.process.internal.DefaultExecHandle.aborted(DefaultExecHandle.java:366) at org.gradle.process.internal.ExecHandleRunner.completed(ExecHandleRunner.java:108) at org.gradle.process.internal.ExecHandleRunner.run(ExecHandleRunner.java:84) at org.gradle.internal.operations.CurrentBuildOperationPreservingRunnable.run(CurrentBuildOperationPreservingRunnable.java:38) at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64) at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) > Task :runners:google-cloud-dataflow-java:cleanUpDockerJavaImages FAILED Remove shutdown hook failed java.lang.IllegalStateException: Shutdown in progress at java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:82) at java.lang.Runtime.removeShutdownHook(Runtime.java:231) at org.gradle.process.internal.shutdown.ShutdownHooks.removeShutdownHook(ShutdownHooks.java:38) at org.gradle.process.internal.DefaultExecHandle.setEndStateInfo(DefaultExecHandle.java:208) at org.gradle.process.internal.DefaultExecHandle.aborted(DefaultExecHandle.java:366) at org.gradle.process.internal.ExecHandleRunner.completed(ExecHandleRunner.java:108) at org.gradle.process.internal.ExecHandleRunner.run(ExecHandleRunner.java:84) at org.gradle.internal.operations.CurrentBuildOperationPreservingRunnable.run(CurrentBuildOperationPreservingRunnable.java:42) at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64) at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) The message received from the daemon indicates that the daemon has disappeared. Build request sent: Build{id=031830c5-4809-4811-850b-2fa372235d8b, currentDir=<https://ci-beam.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_V2_Batch_Java11/ws/src}> Attempting to read last messages from the daemon log... Daemon pid: 1116073 log file: /home/jenkins/.gradle/daemon/7.5.1/daemon-1116073.out.log ----- Last 20 lines from daemon log file - daemon-1116073.out.log ----- at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64) at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Remove shutdown hook failed java.lang.IllegalStateException: Shutdown in progress at java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:82) at java.lang.Runtime.removeShutdownHook(Runtime.java:231) at org.gradle.process.internal.shutdown.ShutdownHooks.removeShutdownHook(ShutdownHooks.java:38) at org.gradle.process.internal.DefaultExecHandle.setEndStateInfo(DefaultExecHandle.java:208) at org.gradle.process.internal.DefaultExecHandle.aborted(DefaultExecHandle.java:366) at org.gradle.process.internal.ExecHandleRunner.completed(ExecHandleRunner.java:108) at org.gradle.process.internal.ExecHandleRunner.run(ExecHandleRunner.java:84) at org.gradle.internal.operations.CurrentBuildOperationPreservingRunnable.run(CurrentBuildOperationPreservingRunnable.java:42) at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64) at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) ----- End of the daemon log ----- FAILURE: Build failed with an exception. * What went wrong: Gradle build daemon disappeared unexpectedly (it may have been killed or may have crashed) * Try: > Run with --stacktrace option to get the stack trace. > Run with --info or --debug option to get more log output. > Run with --scan to get full insights. * Get more help at https://help.gradle.org Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
