See <https://ci-beam.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Batch/1344/display/redirect?page=changes>
Changes: [Robert Bradshaw] Move logging to its own module. [Robert Bradshaw] Cleanup worker logging. [Robert Bradshaw] Add basic counter setting and getting to the typescript SDK. [Robert Bradshaw] Support metrics over the portability API. [Robert Bradshaw] Add distribution metric type. [Robert Bradshaw] old prettier change [noreply] [Task]: PR Bot will push commits only if they are non-empty (#23937) [noreply] Bump cloud.google.com/go/datastore from 1.8.0 to 1.9.0 in /sdks (#23916) [Robert Bradshaw] Remove obsolete TODO. [Robert Bradshaw] Only report counters that were actually used. [noreply] Add custom inference fn suport to the sklearn model handlers (#23642) [noreply] removed trailing whitespace (#23987) [noreply] Beam starter projects blog post (#23964) [noreply] Enable more portable-runner requiring tests. (#23970) [noreply] Website add and update logos (#23899) ------------------------------------------ [...truncated 13.39 KB...] > Task :sdks:java:harness:processResources NO-SOURCE > Task :sdks:java:expansion-service:processResources NO-SOURCE > Task :sdks:java:extensions:protobuf:createCheckerFrameworkManifest > Task :sdks:java:extensions:arrow:createCheckerFrameworkManifest > Task :sdks:java:io:google-cloud-platform:createCheckerFrameworkManifest > Task :sdks:java:extensions:arrow:processResources NO-SOURCE > Task :sdks:java:io:kafka:createCheckerFrameworkManifest > Task :sdks:java:io:google-cloud-platform:processResources NO-SOURCE > Task :sdks:java:core:generateGrammarSource FROM-CACHE > Task :sdks:java:io:kafka:processResources NO-SOURCE > Task :sdks:java:extensions:protobuf:extractProto > Task :model:job-management:processResources > Task :model:fn-execution:processResources > Task :sdks:java:extensions:protobuf:processResources NO-SOURCE > Task :sdks:java:io:kinesis:createCheckerFrameworkManifest > Task > :runners:google-cloud-dataflow-java:****:legacy-****:createCheckerFrameworkManifest > Task > :runners:google-cloud-dataflow-java:****:windmill:createCheckerFrameworkManifest > Task :sdks:java:io:kinesis:processResources NO-SOURCE > Task :sdks:java:core:processResources > Task :runners:google-cloud-dataflow-java:****:legacy-****:processResources > NO-SOURCE > Task :runners:google-cloud-dataflow-java:processResources > Task :sdks:java:io:synthetic:createCheckerFrameworkManifest > Task :sdks:java:io:synthetic:processResources NO-SOURCE > Task :sdks:java:testing:load-tests:createCheckerFrameworkManifest > Task :sdks:java:testing:test-utils:createCheckerFrameworkManifest > Task :sdks:java:testing:load-tests:processResources NO-SOURCE > Task :sdks:java:testing:test-utils:processResources NO-SOURCE > Task :runners:google-cloud-dataflow-java:****:windmill:extractIncludeProto > Task :model:pipeline:extractIncludeProto > Task :runners:google-cloud-dataflow-java:****:windmill:extractProto > Task :model:pipeline:extractProto > Task :runners:google-cloud-dataflow-java:****:windmill:generateProto > FROM-CACHE > Task :model:pipeline:generateProto FROM-CACHE > Task :runners:google-cloud-dataflow-java:****:windmill:compileJava FROM-CACHE > Task :runners:google-cloud-dataflow-java:****:windmill:processResources > Task :runners:google-cloud-dataflow-java:****:windmill:classes > Task :runners:google-cloud-dataflow-java:****:windmill:shadowJar FROM-CACHE > Task :model:pipeline:compileJava FROM-CACHE > Task :model:pipeline:processResources > Task :model:pipeline:classes > Task :model:pipeline:jar > Task :model:pipeline:shadowJar FROM-CACHE > Task :model:fn-execution:extractIncludeProto > Task :model:job-management:extractIncludeProto > Task :model:job-management:generateProto FROM-CACHE > Task :model:fn-execution:generateProto FROM-CACHE > Task :model:job-management:compileJava FROM-CACHE > Task :model:job-management:classes > Task :model:fn-execution:compileJava FROM-CACHE > Task :model:fn-execution:classes > Task :model:job-management:shadowJar FROM-CACHE > Task :model:fn-execution:shadowJar FROM-CACHE > Task :sdks:java:core:compileJava FROM-CACHE > Task :sdks:java:core:classes > Task :sdks:java:core:shadowJar FROM-CACHE > Task :sdks:java:extensions:arrow:compileJava FROM-CACHE > Task :sdks:java:extensions:arrow:classes UP-TO-DATE > Task :sdks:java:extensions:arrow:jar > Task :sdks:java:io:kinesis:compileJava FROM-CACHE > Task :sdks:java:io:kinesis:classes UP-TO-DATE > Task :sdks:java:io:kinesis:jar > Task :sdks:java:testing:test-utils:compileJava FROM-CACHE > Task :sdks:java:testing:test-utils:classes UP-TO-DATE > Task :sdks:java:testing:test-utils:jar > Task :sdks:java:io:synthetic:compileJava FROM-CACHE > Task :sdks:java:io:synthetic:classes UP-TO-DATE > Task :sdks:java:io:synthetic:jar > Task :sdks:java:fn-execution:compileJava FROM-CACHE > Task :sdks:java:fn-execution:classes UP-TO-DATE > Task :sdks:java:fn-execution:jar > Task :runners:core-construction-java:compileJava FROM-CACHE > Task :runners:core-construction-java:classes UP-TO-DATE > Task :runners:core-construction-java:jar > Task :sdks:java:extensions:protobuf:extractIncludeProto > Task :sdks:java:extensions:protobuf:generateProto NO-SOURCE > Task :sdks:java:extensions:protobuf:compileJava FROM-CACHE > Task :sdks:java:extensions:protobuf:classes UP-TO-DATE > Task :sdks:java:extensions:protobuf:jar > Task :runners:core-java:compileJava FROM-CACHE > Task :runners:core-java:classes UP-TO-DATE > Task :runners:core-java:jar > Task :sdks:java:extensions:google-cloud-platform-core:compileJava FROM-CACHE > Task :sdks:java:extensions:google-cloud-platform-core:classes UP-TO-DATE > Task :sdks:java:extensions:google-cloud-platform-core:jar > Task :sdks:java:harness:compileJava FROM-CACHE > Task :sdks:java:harness:classes UP-TO-DATE > Task :sdks:java:harness:jar > Task :runners:java-fn-execution:compileJava FROM-CACHE > Task :runners:java-fn-execution:classes UP-TO-DATE > Task :runners:java-fn-execution:jar > Task :sdks:java:expansion-service:compileJava FROM-CACHE > Task :sdks:java:expansion-service:classes UP-TO-DATE > Task :sdks:java:expansion-service:jar > Task :sdks:java:io:kafka:compileJava FROM-CACHE > Task :sdks:java:io:kafka:classes UP-TO-DATE > Task :sdks:java:io:kafka:jar > Task :sdks:java:io:google-cloud-platform:compileJava FROM-CACHE > Task :sdks:java:io:google-cloud-platform:classes UP-TO-DATE > Task :sdks:java:io:google-cloud-platform:jar > Task :sdks:java:testing:load-tests:compileJava FROM-CACHE > Task :sdks:java:testing:load-tests:classes UP-TO-DATE > Task :sdks:java:testing:load-tests:jar > Task :runners:google-cloud-dataflow-java:compileJava FROM-CACHE > Task :runners:google-cloud-dataflow-java:classes > Task :runners:google-cloud-dataflow-java:jar > Task :runners:google-cloud-dataflow-java:****:legacy-****:compileJava > FROM-CACHE > Task :runners:google-cloud-dataflow-java:****:legacy-****:classes UP-TO-DATE > Task :runners:google-cloud-dataflow-java:****:legacy-****:shadowJar FROM-CACHE > Task :sdks:java:testing:load-tests:run Nov 05, 2022 2:13:44 PM org.apache.beam.runners.dataflow.DataflowRunner validateSdkContainerImageOptions WARNING: Prefer --sdkContainerImage over deprecated legacy option --****HarnessContainerImage. Nov 05, 2022 2:13:45 PM org.apache.beam.runners.dataflow.options.DataflowPipelineOptions$StagingLocationFactory create INFO: No stagingLocation provided, falling back to gcpTempLocation Nov 05, 2022 2:13:45 PM org.apache.beam.runners.dataflow.DataflowRunner fromOptions INFO: PipelineOptions.filesToStage was not specified. Defaulting to files from the classpath: will stage 229 files. Enable logging at DEBUG level to see which files will be staged. Nov 05, 2022 2:13:46 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Executing pipeline on the Dataflow Service, which will have billing implications related to Google Compute Engine usage and other Google Cloud Services. Nov 05, 2022 2:13:48 PM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements INFO: Uploading 230 files from PipelineOptions.filesToStage to staging location to prepare for execution. Nov 05, 2022 2:13:49 PM org.apache.beam.runners.dataflow.util.PackageUtil$PackageAttributes forFileToStage INFO: Staging custom dataflow-****.jar as beam-runners-google-cloud-dataflow-java-legacy-****-2.44.0-SNAPSHOT-KtgNvbRUO1de4zqmPTbUq0gqNI3jNGM8Nfg4VYMJa4g.jar Nov 05, 2022 2:13:49 PM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements INFO: Staging files complete: 230 files cached, 0 files newly uploaded in 0 seconds Nov 05, 2022 2:13:49 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Staging portable pipeline proto to gs://temp-storage-for-perf-tests/loadtests/staging/ Nov 05, 2022 2:13:51 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Read input as step s1 Nov 05, 2022 2:13:51 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Collect start time metrics as step s2 Nov 05, 2022 2:13:51 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Total bytes monitor as step s3 Nov 05, 2022 2:13:51 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Window.Into()/Window.Assign as step s4 Nov 05, 2022 2:13:51 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Group by key (0) as step s5 Nov 05, 2022 2:13:51 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Ungroup and reiterate (0) as step s6 Nov 05, 2022 2:13:51 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Collect end time metrics (0) as step s7 Nov 05, 2022 2:13:51 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Dataflow SDK version: 2.44.0-SNAPSHOT Nov 05, 2022 2:13:51 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2022-11-05_07_13_51-9488149486145469014?project=apache-beam-testing Nov 05, 2022 2:13:51 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Submitted job: 2022-11-05_07_13_51-9488149486145469014 Nov 05, 2022 2:13:51 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: To cancel the job using the 'gcloud' tool, run: > gcloud dataflow jobs --project=apache-beam-testing cancel > --region=us-central1 2022-11-05_07_13_51-9488149486145469014 Nov 05, 2022 2:13:54 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process WARNING: 2022-11-05T14:13:53.387Z: The workflow name is not a valid Cloud Label. Labels applied to Cloud resources (such as GCE Instances) for monitoring will be labeled with this modified job name: load0tests0java0dataflow0batch0gbk01-jenkins-1105141345-18-29h4. For the best monitoring experience, please name your job with a valid Cloud Label. For details, see: https://cloud.google.com/compute/docs/labeling-resources#restrictions Nov 05, 2022 2:14:01 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-11-05T14:14:00.353Z: Worker configuration: e2-standard-2 in us-central1-b. Nov 05, 2022 2:14:03 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-11-05T14:14:01.514Z: Expanding CoGroupByKey operations into optimizable parts. Nov 05, 2022 2:14:03 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-11-05T14:14:01.574Z: Expanding GroupByKey operations into optimizable parts. Nov 05, 2022 2:14:03 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-11-05T14:14:01.599Z: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns Nov 05, 2022 2:14:03 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-11-05T14:14:01.675Z: Fusing adjacent ParDo, Read, Write, and Flatten operations Nov 05, 2022 2:14:03 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-11-05T14:14:01.699Z: Fusing consumer Collect start time metrics into Read input Nov 05, 2022 2:14:03 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-11-05T14:14:01.715Z: Fusing consumer Total bytes monitor into Collect start time metrics Nov 05, 2022 2:14:03 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-11-05T14:14:01.735Z: Fusing consumer Window.Into()/Window.Assign into Total bytes monitor Nov 05, 2022 2:14:03 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-11-05T14:14:01.764Z: Fusing consumer Group by key (0)/Reify into Window.Into()/Window.Assign Nov 05, 2022 2:14:03 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-11-05T14:14:01.782Z: Fusing consumer Group by key (0)/Write into Group by key (0)/Reify Nov 05, 2022 2:14:03 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-11-05T14:14:01.806Z: Fusing consumer Group by key (0)/GroupByWindow into Group by key (0)/Read Nov 05, 2022 2:14:03 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-11-05T14:14:01.824Z: Fusing consumer Ungroup and reiterate (0) into Group by key (0)/GroupByWindow Nov 05, 2022 2:14:03 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-11-05T14:14:01.850Z: Fusing consumer Collect end time metrics (0) into Ungroup and reiterate (0) Nov 05, 2022 2:14:03 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-11-05T14:14:02.178Z: Executing operation Group by key (0)/Create Nov 05, 2022 2:14:03 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-11-05T14:14:02.246Z: Starting 5 ****s in us-central1-b... Nov 05, 2022 2:14:03 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-11-05T14:14:02.677Z: Finished operation Group by key (0)/Create Nov 05, 2022 2:14:03 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-11-05T14:14:02.782Z: Executing operation Read input+Collect start time metrics+Total bytes monitor+Window.Into()/Window.Assign+Group by key (0)/Reify+Group by key (0)/Write Nov 05, 2022 2:14:08 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-11-05T14:14:06.988Z: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete Nov 05, 2022 2:14:50 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-11-05T14:14:49.753Z: Autoscaling: Raised the number of ****s to 5 based on the rate of progress in the currently running stage(s). Nov 05, 2022 2:15:21 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-11-05T14:15:20.225Z: Workers have started successfully. Nov 05, 2022 2:16:42 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-11-05T14:16:42.299Z: Finished operation Read input+Collect start time metrics+Total bytes monitor+Window.Into()/Window.Assign+Group by key (0)/Reify+Group by key (0)/Write Nov 05, 2022 2:16:42 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-11-05T14:16:42.367Z: Executing operation Group by key (0)/Close Nov 05, 2022 2:16:42 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-11-05T14:16:42.425Z: Finished operation Group by key (0)/Close Nov 05, 2022 2:16:43 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-11-05T14:16:42.502Z: Executing operation Group by key (0)/Read+Group by key (0)/GroupByWindow+Ungroup and reiterate (0)+Collect end time metrics (0) Nov 05, 2022 2:17:41 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-11-05T14:17:39.258Z: Finished operation Group by key (0)/Read+Group by key (0)/GroupByWindow+Ungroup and reiterate (0)+Collect end time metrics (0) Nov 05, 2022 2:17:41 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-11-05T14:17:39.401Z: Cleaning up. Nov 05, 2022 2:17:41 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-11-05T14:17:39.476Z: Stopping **** pool... > Task :sdks:java:testing:load-tests:run FAILED The message received from the daemon indicates that the daemon has disappeared. Build request sent: Build{id=c82f24c1-bb83-4352-8af3-7b48a84b9d19, currentDir=<https://ci-beam.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Batch/ws/src}> Attempting to read last messages from the daemon log... Daemon pid: 3220154 log file: /home/jenkins/.gradle/daemon/7.5.1/daemon-3220154.out.log ----- Last 20 lines from daemon log file - daemon-3220154.out.log ----- * What went wrong: Execution failed for task ':sdks:java:testing:load-tests:run'. > Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with > non-zero exit value 143 * Try: > Run with --stacktrace option to get the stack trace. > Run with --info or --debug option to get more log output. * Get more help at https://help.gradle.org Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0. You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins. See https://docs.gradle.org/7.5.1/userguide/command_line_interface.html#sec:command_line_warnings BUILD FAILED in 5m 5s 99 actionable tasks: 60 executed, 37 from cache, 2 up-to-date Daemon vm is shutting down... The daemon has exited normally or was terminated in response to a user interrupt. ----- End of the daemon log ----- FAILURE: Build failed with an exception. * What went wrong: Gradle build daemon disappeared unexpectedly (it may have been killed or may have crashed) * Try: > Run with --stacktrace option to get the stack trace. > Run with --info or --debug option to get more log output. > Run with --scan to get full insights. * Get more help at https://help.gradle.org Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
