See <https://ci-beam.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_V2_Streaming_Java11/323/display/redirect>
Changes: ------------------------------------------ [...truncated 56.54 KB...] May 08, 2022 12:06:58 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process WARNING: 2022-05-08T12:06:57.411Z: The workflow name is not a valid Cloud Label. Labels applied to Cloud resources (such as GCE Instances) for monitoring will be labeled with this modified job name: load0tests0java110dataflow0v20streaming0gbk01-jenkins-0508-j4im. For the best monitoring experience, please name your job with a valid Cloud Label. For details, see: https://cloud.google.com/compute/docs/labeling-resources#restrictions May 08, 2022 12:07:06 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-08T12:07:03.763Z: Worker configuration: e2-standard-2 in us-central1-b. May 08, 2022 12:07:06 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-08T12:07:04.549Z: Expanding SplittableParDo operations into optimizable parts. May 08, 2022 12:07:06 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-08T12:07:04.581Z: Expanding CollectionToSingleton operations into optimizable parts. May 08, 2022 12:07:06 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-08T12:07:04.674Z: Expanding CoGroupByKey operations into optimizable parts. May 08, 2022 12:07:06 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-08T12:07:04.753Z: Expanding SplittableProcessKeyed operations into optimizable parts. May 08, 2022 12:07:06 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-08T12:07:04.834Z: Expanding GroupByKey operations into streaming Read/Write steps May 08, 2022 12:07:06 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-08T12:07:04.978Z: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns May 08, 2022 12:07:06 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-08T12:07:05.278Z: Fusing adjacent ParDo, Read, Write, and Flatten operations May 08, 2022 12:07:06 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-08T12:07:05.314Z: Fusing consumer Read input/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource) into Read input/Impulse May 08, 2022 12:07:06 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-08T12:07:05.337Z: Fusing consumer Read-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/PairWithRestriction into Read input/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource) May 08, 2022 12:07:06 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-08T12:07:05.363Z: Fusing consumer Read-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/SplitWithSizing into Read-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/PairWithRestriction May 08, 2022 12:07:06 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-08T12:07:05.388Z: Fusing consumer Read input/ParDo(StripIds)/ParMultiDo(StripIds) into Read-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/ProcessElementAndRestrictionWithSizing May 08, 2022 12:07:06 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-08T12:07:05.430Z: Fusing consumer Collect start time metrics/ParMultiDo(TimeMonitor) into Read input/ParDo(StripIds)/ParMultiDo(StripIds) May 08, 2022 12:07:06 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-08T12:07:05.449Z: Fusing consumer Total bytes monitor/ParMultiDo(ByteMonitor) into Collect start time metrics/ParMultiDo(TimeMonitor) May 08, 2022 12:07:06 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-08T12:07:05.472Z: Fusing consumer Window.Into()/Window.Assign into Total bytes monitor/ParMultiDo(ByteMonitor) May 08, 2022 12:07:06 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-08T12:07:05.500Z: Fusing consumer Group by key (0)/WriteStream into Window.Into()/Window.Assign May 08, 2022 12:07:06 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-08T12:07:05.528Z: Fusing consumer Group by key (0)/MergeBuckets into Group by key (0)/ReadStream May 08, 2022 12:07:06 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-08T12:07:05.546Z: Fusing consumer Ungroup and reiterate (0)/ParMultiDo(UngroupAndReiterate) into Group by key (0)/MergeBuckets May 08, 2022 12:07:06 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-08T12:07:05.578Z: Fusing consumer Collect end time metrics (0)/ParMultiDo(TimeMonitor) into Ungroup and reiterate (0)/ParMultiDo(UngroupAndReiterate) May 08, 2022 12:07:06 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-08T12:07:05.719Z: Running job using Streaming Engine May 08, 2022 12:07:06 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-08T12:07:05.945Z: Starting 5 ****s in us-central1-b... May 08, 2022 12:07:29 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-08T12:07:28.656Z: Autoscaling: Raised the number of ****s to 5 so that the pipeline can catch up with its backlog and keep up with its input rate. May 08, 2022 12:07:32 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-08T12:07:31.351Z: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete May 08, 2022 12:08:38 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-08T12:08:37.801Z: Workers have started successfully. May 08, 2022 12:24:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process SEVERE: 2022-05-08T12:23:59.281Z: generic::aborted: SDK harness sdk-0-0 disconnected. generic::aborted: SDK harness sdk-0-0 disconnected. passed through: ==> dist_proc/dax/workflow/****/fnapi_service_impl.cc:922 May 08, 2022 12:27:14 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process SEVERE: 2022-05-08T12:27:12.752Z: generic::cancelled: Data channel closed, unable to send additional data to SDK sdk-0-0 generic::cancelled: Data channel closed, unable to send additional data to SDK sdk-0-0 passed through: ==> dist_proc/dax/workflow/****/fnapi_service_impl.cc:922 May 08, 2022 12:33:36 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-08T12:33:34.253Z: Autoscaling: Raised the number of ****s to 5 so that the pipeline can catch up with its backlog and keep up with its input rate. May 08, 2022 12:52:34 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-08T12:52:34.243Z: Autoscaling: Raised the number of ****s to 5 so that the pipeline can catch up with its backlog and keep up with its input rate. May 08, 2022 1:05:39 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-08T13:05:38.740Z: Autoscaling: Raised the number of ****s to 5 so that the pipeline can catch up with its backlog and keep up with its input rate. May 08, 2022 1:14:38 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-08T13:14:37.486Z: Autoscaling: Raised the number of ****s to 5 so that the pipeline can catch up with its backlog and keep up with its input rate. May 08, 2022 1:23:43 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-08T13:23:41.145Z: Autoscaling: Raised the number of ****s to 5 so that the pipeline can catch up with its backlog and keep up with its input rate. May 08, 2022 1:32:44 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-08T13:32:44.098Z: Autoscaling: Raised the number of ****s to 5 so that the pipeline can catch up with its backlog and keep up with its input rate. May 08, 2022 1:41:43 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-08T13:41:43.029Z: Autoscaling: Raised the number of ****s to 5 so that the pipeline can catch up with its backlog and keep up with its input rate. May 08, 2022 1:51:31 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process SEVERE: 2022-05-08T13:51:30.556Z: generic::aborted: SDK harness sdk-0-0 disconnected. generic::aborted: SDK harness sdk-0-0 disconnected. passed through: ==> dist_proc/dax/workflow/****/monitoring_info_view_creator.cc:68 generic::aborted: SDK harness sdk-0-0 disconnected. passed through: ==> dist_proc/dax/workflow/****/fnapi_service_impl.cc:922 May 08, 2022 1:56:48 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-08T13:56:47.282Z: Autoscaling: Raised the number of ****s to 5 so that the pipeline can catch up with its backlog and keep up with its input rate. May 08, 2022 2:05:32 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process SEVERE: 2022-05-08T14:05:31.055Z: generic::aborted: SDK harness sdk-0-0 disconnected. passed through: ==> dist_proc/dax/workflow/****/fnapi_service_impl.cc:922 generic::aborted: SDK harness sdk-0-0 disconnected. passed through: ==> dist_proc/dax/workflow/****/monitoring_info_view_creator.cc:68 generic::aborted: SDK harness sdk-0-0 disconnected. May 08, 2022 2:06:27 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process SEVERE: 2022-05-08T14:06:25.555Z: generic::aborted: SDK harness sdk-0-0 disconnected. passed through: ==> dist_proc/dax/workflow/****/fnapi_service_impl.cc:922 generic::aborted: SDK harness sdk-0-0 disconnected. May 08, 2022 2:07:50 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process SEVERE: 2022-05-08T14:07:49.763Z: generic::aborted: SDK harness sdk-0-0 disconnected. passed through: ==> dist_proc/dax/workflow/****/fnapi_service_impl.cc:922 generic::aborted: SDK harness sdk-0-0 disconnected. May 08, 2022 2:11:47 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-08T14:11:46.956Z: Autoscaling: Raised the number of ****s to 5 so that the pipeline can catch up with its backlog and keep up with its input rate. May 08, 2022 2:20:51 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-08T14:20:49.698Z: Autoscaling: Raised the number of ****s to 5 so that the pipeline can catch up with its backlog and keep up with its input rate. May 08, 2022 2:29:53 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-08T14:29:53.178Z: Autoscaling: Raised the number of ****s to 5 so that the pipeline can catch up with its backlog and keep up with its input rate. May 08, 2022 2:39:24 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process SEVERE: 2022-05-08T14:39:23.139Z: generic::aborted: SDK harness sdk-0-0 disconnected. generic::aborted: SDK harness sdk-0-0 disconnected. passed through: ==> dist_proc/dax/workflow/****/fnapi_service_impl.cc:922 May 08, 2022 2:45:52 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-08T14:45:52.145Z: Autoscaling: Raised the number of ****s to 5 so that the pipeline can catch up with its backlog and keep up with its input rate. May 08, 2022 2:56:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-08T14:56:55.591Z: Autoscaling: Raised the number of ****s to 5 so that the pipeline can catch up with its backlog and keep up with its input rate. May 08, 2022 3:11:59 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-08T15:11:57.078Z: Autoscaling: Raised the number of ****s to 5 so that the pipeline can catch up with its backlog and keep up with its input rate. May 08, 2022 3:20:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process SEVERE: 2022-05-08T15:20:16.204Z: generic::aborted: SDK harness sdk-0-0 disconnected. generic::aborted: SDK harness sdk-0-0 disconnected. passed through: ==> dist_proc/dax/workflow/****/fnapi_service_impl.cc:922 May 08, 2022 3:20:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process SEVERE: 2022-05-08T15:20:22.591Z: generic::aborted: SDK harness sdk-0-0 disconnected. generic::aborted: SDK harness sdk-0-0 disconnected. passed through: ==> dist_proc/dax/workflow/****/fnapi_service_impl.cc:922 May 08, 2022 3:21:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-08T15:21:00.390Z: Autoscaling: Raised the number of ****s to 5 so that the pipeline can catch up with its backlog and keep up with its input rate. May 08, 2022 3:21:23 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process SEVERE: 2022-05-08T15:21:21.942Z: generic::aborted: SDK harness sdk-0-0 disconnected. passed through: ==> dist_proc/dax/workflow/****/monitoring_info_view_creator.cc:68 generic::cancelled: Data channel closed, unable to send additional data to SDK sdk-0-0 passed through: ==> dist_proc/dax/workflow/****/fnapi_service_impl.cc:922 generic::aborted: SDK harness sdk-0-0 disconnected. passed through: ==> dist_proc/dax/workflow/****/fnapi_service_impl.cc:922 generic::aborted: SDK harness sdk-0-0 disconnected. generic::cancelled: Data channel closed, unable to send additional data to SDK sdk-0-0 May 08, 2022 3:31:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-08T15:30:59.386Z: Autoscaling: Raised the number of ****s to 5 so that the pipeline can catch up with its backlog and keep up with its input rate. May 08, 2022 3:36:35 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process SEVERE: 2022-05-08T15:36:32.970Z: generic::aborted: SDK harness sdk-0-0 disconnected. May 08, 2022 3:36:42 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process SEVERE: 2022-05-08T15:36:40.555Z: generic::aborted: SDK harness sdk-0-0 disconnected. generic::aborted: SDK harness sdk-0-0 disconnected. passed through: ==> dist_proc/dax/workflow/****/fnapi_service_impl.cc:922 May 08, 2022 3:43:04 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-08T15:43:03.194Z: Autoscaling: Raised the number of ****s to 5 so that the pipeline can catch up with its backlog and keep up with its input rate. May 08, 2022 3:51:36 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process SEVERE: 2022-05-08T15:51:35.629Z: generic::aborted: SDK harness sdk-0-0 disconnected. generic::aborted: SDK harness sdk-0-0 disconnected. passed through: ==> dist_proc/dax/workflow/****/fnapi_service_impl.cc:922 May 08, 2022 3:56:05 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-08T15:56:03.022Z: Autoscaling: Raised the number of ****s to 5 so that the pipeline can catch up with its backlog and keep up with its input rate. May 08, 2022 4:01:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-08T16:00:59.091Z: Cancel request is committed for workflow job: 2022-05-08_05_06_49-8142284004590900524. May 08, 2022 4:01:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-08T16:00:59.141Z: Cleaning up. May 08, 2022 4:01:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-08T16:00:59.223Z: Stopping **** pool... May 08, 2022 4:01:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-08T16:00:59.266Z: Stopping **** pool... May 08, 2022 4:03:27 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-08T16:03:26.340Z: Autoscaling: Reduced the number of ****s to 0 based on low average **** CPU utilization, and the pipeline having sufficiently low backlog and keeping up with input rate. May 08, 2022 4:03:27 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-05-08T16:03:26.400Z: Worker pool stopped. May 08, 2022 4:03:34 PM org.apache.beam.runners.dataflow.DataflowPipelineJob logTerminalState INFO: Job 2022-05-08_05_06_49-8142284004590900524 finished with status CANCELLED. Load test results for test (ID): b42fa6b2-eea2-4fe9-b7bd-9a56b407fd48 and timestamp: 2022-05-08T12:06:44.558000000Z: Metric: Value: dataflow_v2_java11_runtime_sec 6926.827 dataflow_v2_java11_total_bytes_count 2.89020557E9 Exception in thread "main" java.lang.RuntimeException: Invalid job state: CANCELLED. at org.apache.beam.sdk.loadtests.JobFailure.handleFailure(JobFailure.java:51) at org.apache.beam.sdk.loadtests.LoadTest.run(LoadTest.java:139) at org.apache.beam.sdk.loadtests.GroupByKeyLoadTest.run(GroupByKeyLoadTest.java:57) at org.apache.beam.sdk.loadtests.GroupByKeyLoadTest.main(GroupByKeyLoadTest.java:131) > Task :sdks:java:testing:load-tests:run FAILED > Task :runners:google-cloud-dataflow-java:cleanUpDockerJavaImages Untagged: us.gcr.io/apache-beam-testing/java-postcommit-it/java:20220508120443 Untagged: us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:834a3075739c028d7e7b3cd67bca65d5ce5034c4bf58148ecf5c4dc28e64d192 Tag: [us.gcr.io/apache-beam-testing/java-postcommit-it/java:20220508120443] - referencing digest: [us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:834a3075739c028d7e7b3cd67bca65d5ce5034c4bf58148ecf5c4dc28e64d192] Deleted [[us.gcr.io/apache-beam-testing/java-postcommit-it/java:20220508120443] (referencing [us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:834a3075739c028d7e7b3cd67bca65d5ce5034c4bf58148ecf5c4dc28e64d192])]. Removing untagged image us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:039601af6597ae20bef243145ecfafba320c334a30716f8eb5fc2cfe3a9f5ef3 Digests: - us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:039601af6597ae20bef243145ecfafba320c334a30716f8eb5fc2cfe3a9f5ef3 Deleted [us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:039601af6597ae20bef243145ecfafba320c334a30716f8eb5fc2cfe3a9f5ef3]. Removing untagged image us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:834a3075739c028d7e7b3cd67bca65d5ce5034c4bf58148ecf5c4dc28e64d192 Digests: - us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:834a3075739c028d7e7b3cd67bca65d5ce5034c4bf58148ecf5c4dc28e64d192 Deleted [us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:834a3075739c028d7e7b3cd67bca65d5ce5034c4bf58148ecf5c4dc28e64d192]. FAILURE: Build failed with an exception. * What went wrong: Execution failed for task ':sdks:java:testing:load-tests:run'. > Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with > non-zero exit value 1 * Try: > Run with --stacktrace option to get the stack trace. > Run with --info or --debug option to get more log output. > Run with --scan to get full insights. * Get more help at https://help.gradle.org Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0. You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins. See https://docs.gradle.org/7.4/userguide/command_line_interface.html#sec:command_line_warnings Execution optimizations have been disabled for 1 invalid unit(s) of work during this build to ensure correctness. Please consult deprecation warnings for more details. BUILD FAILED in 3h 59m 15s 109 actionable tasks: 72 executed, 33 from cache, 4 up-to-date Publishing build scan... https://gradle.com/s/2vbdepvsacmga Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
