See
<https://ci-beam.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Batch/817/display/redirect?page=changes>
Changes:
[Ismaël Mejía] [BEAM-12342] Upgrade Spark 2 to version 2.4.8
[Udi Meiri] [BEAM-12352] Skip
GcsIOIntegrationTest.test_copy{,_batch}_rewrite_token
[noreply] [BEAM-3713] Move validatesRunnerBatchTests and
validatesRunnerStreaming
------------------------------------------
[...truncated 340.13 KB...]
> Task :sdks:java:io:synthetic:createCheckerFrameworkManifest UP-TO-DATE
> Task :runners:google-cloud-dataflow-java:processResources UP-TO-DATE
> Task :sdks:java:testing:test-utils:createCheckerFrameworkManifest UP-TO-DATE
> Task :sdks:java:io:kinesis:processResources NO-SOURCE
> Task :sdks:java:testing:load-tests:createCheckerFrameworkManifest UP-TO-DATE
> Task :sdks:java:core:generateGrammarSource UP-TO-DATE
> Task :sdks:java:io:synthetic:processResources NO-SOURCE
> Task :sdks:java:testing:test-utils:processResources NO-SOURCE
> Task :sdks:java:testing:load-tests:processResources NO-SOURCE
> Task :runners:google-cloud-dataflow-java:****:legacy-****:processResources
> NO-SOURCE
> Task :sdks:java:io:kafka:processResources NO-SOURCE
> Task :sdks:java:core:processResources UP-TO-DATE
> Task :runners:google-cloud-dataflow-java:****:windmill:extractIncludeProto
> UP-TO-DATE
> Task :model:pipeline:extractIncludeProto UP-TO-DATE
> Task :runners:google-cloud-dataflow-java:****:windmill:extractProto UP-TO-DATE
> Task :model:pipeline:extractProto UP-TO-DATE
> Task :model:pipeline:generateProto UP-TO-DATE
> Task :runners:google-cloud-dataflow-java:****:windmill:generateProto
> UP-TO-DATE
> Task :model:pipeline:compileJava UP-TO-DATE
> Task :runners:google-cloud-dataflow-java:****:windmill:compileJava UP-TO-DATE
> Task :model:pipeline:processResources UP-TO-DATE
> Task :runners:google-cloud-dataflow-java:****:windmill:processResources
> UP-TO-DATE
> Task :model:pipeline:classes UP-TO-DATE
> Task :runners:google-cloud-dataflow-java:****:windmill:classes UP-TO-DATE
> Task :model:pipeline:jar UP-TO-DATE
> Task :runners:google-cloud-dataflow-java:****:windmill:shadowJar UP-TO-DATE
> Task :model:pipeline:shadowJar UP-TO-DATE
> Task :model:fn-execution:extractIncludeProto UP-TO-DATE
> Task :model:job-management:extractIncludeProto UP-TO-DATE
> Task :model:fn-execution:generateProto UP-TO-DATE
> Task :model:job-management:generateProto UP-TO-DATE
> Task :model:fn-execution:compileJava UP-TO-DATE
> Task :model:job-management:compileJava UP-TO-DATE
> Task :model:fn-execution:classes UP-TO-DATE
> Task :model:job-management:classes UP-TO-DATE
> Task :model:job-management:shadowJar UP-TO-DATE
> Task :model:fn-execution:shadowJar UP-TO-DATE
> Task :sdks:java:core:compileJava UP-TO-DATE
> Task :sdks:java:core:classes UP-TO-DATE
> Task :sdks:java:core:shadowJar UP-TO-DATE
> Task :sdks:java:extensions:protobuf:extractIncludeProto UP-TO-DATE
> Task :sdks:java:extensions:protobuf:generateProto NO-SOURCE
> Task :sdks:java:io:synthetic:compileJava UP-TO-DATE
> Task :sdks:java:io:synthetic:classes UP-TO-DATE
> Task :sdks:java:fn-execution:compileJava UP-TO-DATE
> Task :sdks:java:fn-execution:classes UP-TO-DATE
> Task :sdks:java:io:synthetic:jar UP-TO-DATE
> Task :sdks:java:fn-execution:jar UP-TO-DATE
> Task :sdks:java:extensions:protobuf:compileJava UP-TO-DATE
> Task :sdks:java:extensions:protobuf:classes UP-TO-DATE
> Task :sdks:java:io:kinesis:compileJava UP-TO-DATE
> Task :sdks:java:io:kinesis:classes UP-TO-DATE
> Task :sdks:java:extensions:protobuf:jar UP-TO-DATE
> Task :sdks:java:testing:test-utils:compileJava UP-TO-DATE
> Task :sdks:java:testing:test-utils:classes UP-TO-DATE
> Task :sdks:java:io:kinesis:jar UP-TO-DATE
> Task :sdks:java:testing:test-utils:jar UP-TO-DATE
> Task :runners:core-construction-java:compileJava UP-TO-DATE
> Task :runners:core-construction-java:classes UP-TO-DATE
> Task :runners:core-construction-java:jar UP-TO-DATE
> Task :runners:core-java:compileJava UP-TO-DATE
> Task :runners:core-java:classes UP-TO-DATE
> Task :runners:core-java:jar UP-TO-DATE
> Task :sdks:java:extensions:google-cloud-platform-core:compileJava UP-TO-DATE
> Task :sdks:java:extensions:google-cloud-platform-core:classes UP-TO-DATE
> Task :sdks:java:extensions:google-cloud-platform-core:jar UP-TO-DATE
> Task :sdks:java:harness:compileJava UP-TO-DATE
> Task :sdks:java:harness:classes UP-TO-DATE
> Task :sdks:java:harness:jar UP-TO-DATE
> Task :sdks:java:harness:shadowJar UP-TO-DATE
> Task :runners:java-fn-execution:compileJava UP-TO-DATE
> Task :runners:java-fn-execution:classes UP-TO-DATE
> Task :runners:java-fn-execution:jar UP-TO-DATE
> Task :sdks:java:expansion-service:compileJava UP-TO-DATE
> Task :sdks:java:expansion-service:classes UP-TO-DATE
> Task :sdks:java:expansion-service:jar UP-TO-DATE
> Task :sdks:java:io:kafka:compileJava UP-TO-DATE
> Task :sdks:java:io:kafka:classes UP-TO-DATE
> Task :sdks:java:io:kafka:jar UP-TO-DATE
> Task :sdks:java:io:google-cloud-platform:compileJava UP-TO-DATE
> Task :sdks:java:io:google-cloud-platform:classes UP-TO-DATE
> Task :sdks:java:io:google-cloud-platform:jar UP-TO-DATE
> Task :sdks:java:testing:load-tests:compileJava UP-TO-DATE
> Task :sdks:java:testing:load-tests:classes UP-TO-DATE
> Task :sdks:java:testing:load-tests:jar UP-TO-DATE
> Task :runners:google-cloud-dataflow-java:compileJava UP-TO-DATE
> Task :runners:google-cloud-dataflow-java:classes UP-TO-DATE
> Task :runners:google-cloud-dataflow-java:jar UP-TO-DATE
> Task :runners:google-cloud-dataflow-java:****:legacy-****:compileJava
> UP-TO-DATE
> Task :runners:google-cloud-dataflow-java:****:legacy-****:classes UP-TO-DATE
> Task :runners:google-cloud-dataflow-java:****:legacy-****:shadowJar UP-TO-DATE
> Task :sdks:java:testing:load-tests:run
May 21, 2021 2:51:41 PM org.apache.beam.runners.dataflow.DataflowRunner
validateSdkContainerImageOptions
WARNING: Prefer --sdkContainerImage over deprecated legacy option
--****HarnessContainerImage.
May 21, 2021 2:51:42 PM
org.apache.beam.runners.dataflow.options.DataflowPipelineOptions$StagingLocationFactory
create
INFO: No stagingLocation provided, falling back to gcpTempLocation
May 21, 2021 2:51:43 PM org.apache.beam.runners.dataflow.DataflowRunner
fromOptions
INFO: PipelineOptions.filesToStage was not specified. Defaulting to files from
the classpath: will stage 188 files. Enable logging at DEBUG level to see which
files will be staged.
May 21, 2021 2:51:44 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Executing pipeline on the Dataflow Service, which will have billing
implications related to Google Compute Engine usage and other Google Cloud
Services.
May 21, 2021 2:51:47 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging portable pipeline proto to
gs://temp-storage-for-perf-tests/loadtests/staging/
May 21, 2021 2:51:50 PM org.apache.beam.runners.dataflow.util.PackageUtil
stageClasspathElements
INFO: Uploading 189 files from PipelineOptions.filesToStage to staging location
to prepare for execution.
May 21, 2021 2:51:50 PM
org.apache.beam.runners.dataflow.util.PackageUtil$PackageAttributes
forFileToStage
INFO: Staging custom dataflow-****.jar as
beam-runners-google-cloud-dataflow-java-legacy-****-2.31.0-SNAPSHOT-p8DZqfSGxUnLeoUu48qOOTbZOXglhdbI2awWBgSdBQs.jar
May 21, 2021 2:51:53 PM org.apache.beam.runners.dataflow.util.PackageUtil
stageClasspathElements
INFO: Staging files complete: 189 files cached, 0 files newly uploaded in 2
seconds
May 21, 2021 2:51:53 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read input as step s1
May 21, 2021 2:51:53 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Collect start time metrics as step s2
May 21, 2021 2:51:53 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Total bytes monitor as step s3
May 21, 2021 2:51:53 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Window.Into()/Window.Assign as step s4
May 21, 2021 2:51:53 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Group by key (0) as step s5
May 21, 2021 2:51:53 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Ungroup and reiterate (0) as step s6
May 21, 2021 2:51:53 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Collect end time metrics (0) as step s7
May 21, 2021 2:51:53 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Dataflow SDK version: 2.31.0-SNAPSHOT
May 21, 2021 2:51:57 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-05-21_07_51_53-13225627573959012564?project=apache-beam-testing
May 21, 2021 2:51:57 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Submitted job: 2021-05-21_07_51_53-13225627573959012564
May 21, 2021 2:51:57 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel
> --region=us-central1 2021-05-21_07_51_53-13225627573959012564
May 21, 2021 2:52:04 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
WARNING: 2021-05-21T14:52:02.428Z: The workflow name is not a valid Cloud
Label. Labels applied to Cloud resources (such as GCE Instances) for monitoring
will be labeled with this modified job name:
load0tests0java0dataflow0batch0gbk07-jenkins-0521145143-69-uk5w. For the best
monitoring experience, please name your job with a valid Cloud Label. For
details, see:
https://cloud.google.com/compute/docs/labeling-resources#restrictions
May 21, 2021 2:52:13 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-05-21T14:52:11.934Z: Worker configuration: n1-standard-1 in
us-central1-b.
May 21, 2021 2:52:13 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-05-21T14:52:12.702Z: Expanding CoGroupByKey operations into
optimizable parts.
May 21, 2021 2:52:13 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-05-21T14:52:12.772Z: Expanding GroupByKey operations into
optimizable parts.
May 21, 2021 2:52:13 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-05-21T14:52:12.801Z: Lifting ValueCombiningMappingFns into
MergeBucketsMappingFns
May 21, 2021 2:52:13 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-05-21T14:52:12.966Z: Fusing adjacent ParDo, Read, Write, and Flatten
operations
May 21, 2021 2:52:13 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-05-21T14:52:12.994Z: Fusing consumer Collect start time metrics into
Read input
May 21, 2021 2:52:13 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-05-21T14:52:13.019Z: Fusing consumer Total bytes monitor into
Collect start time metrics
May 21, 2021 2:52:13 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-05-21T14:52:13.050Z: Fusing consumer Window.Into()/Window.Assign
into Total bytes monitor
May 21, 2021 2:52:13 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-05-21T14:52:13.110Z: Fusing consumer Group by key (0)/Reify into
Window.Into()/Window.Assign
May 21, 2021 2:52:13 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-05-21T14:52:13.146Z: Fusing consumer Group by key (0)/Write into
Group by key (0)/Reify
May 21, 2021 2:52:13 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-05-21T14:52:13.176Z: Fusing consumer Group by key (0)/GroupByWindow
into Group by key (0)/Read
May 21, 2021 2:52:13 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-05-21T14:52:13.207Z: Fusing consumer Ungroup and reiterate (0) into
Group by key (0)/GroupByWindow
May 21, 2021 2:52:13 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-05-21T14:52:13.231Z: Fusing consumer Collect end time metrics (0)
into Ungroup and reiterate (0)
May 21, 2021 2:52:16 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-05-21T14:52:13.581Z: Executing operation Group by key (0)/Create
May 21, 2021 2:52:16 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-05-21T14:52:13.690Z: Starting 5 ****s in us-central1-b...
May 21, 2021 2:52:16 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-05-21T14:52:14.096Z: Finished operation Group by key (0)/Create
May 21, 2021 2:52:16 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-05-21T14:52:14.294Z: Executing operation Read input+Collect start
time metrics+Total bytes monitor+Window.Into()/Window.Assign+Group by key
(0)/Reify+Group by key (0)/Write
May 21, 2021 2:52:23 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-05-21T14:52:22.815Z: Your project already contains 100
Dataflow-created metric descriptors, so new user metrics of the form
custom.googleapis.com/* will not be created. However, all user metrics are also
available in the metric dataflow.googleapis.com/job/user_counter. If you rely
on the custom metrics, you can delete old / unused metric descriptors. See
https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list
and
https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
May 21, 2021 2:52:55 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-05-21T14:52:54.950Z: Autoscaling: Raised the number of ****s to 5
based on the rate of progress in the currently running stage(s).
May 21, 2021 2:53:21 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-05-21T14:53:19.400Z: Workers have started successfully.
May 21, 2021 2:53:21 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-05-21T14:53:19.454Z: Workers have started successfully.
May 21, 2021 2:53:53 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-05-21T14:53:51.811Z: Finished operation Read input+Collect start
time metrics+Total bytes monitor+Window.Into()/Window.Assign+Group by key
(0)/Reify+Group by key (0)/Write
May 21, 2021 2:53:53 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-05-21T14:53:51.876Z: Executing operation Group by key (0)/Close
May 21, 2021 2:53:53 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-05-21T14:53:51.938Z: Finished operation Group by key (0)/Close
May 21, 2021 2:53:53 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-05-21T14:53:52.016Z: Executing operation Group by key (0)/Read+Group
by key (0)/GroupByWindow+Ungroup and reiterate (0)+Collect end time metrics (0)
May 21, 2021 2:54:42 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-05-21T14:54:42.860Z: Finished operation Group by key (0)/Read+Group
by key (0)/GroupByWindow+Ungroup and reiterate (0)+Collect end time metrics (0)
May 21, 2021 2:54:45 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-05-21T14:54:43.001Z: Cleaning up.
May 21, 2021 2:54:45 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-05-21T14:54:43.096Z: Stopping **** pool...
May 21, 2021 2:55:34 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-05-21T14:55:32.726Z: Autoscaling: Resized **** pool from 5 to 0.
May 21, 2021 2:55:34 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-05-21T14:55:32.766Z: Worker pool stopped.
May 21, 2021 3:00:45 PM org.apache.beam.runners.dataflow.DataflowPipelineJob
logTerminalState
INFO: Job 2021-05-21_07_51_53-13225627573959012564 finished with status DONE.
May 21, 2021 3:00:45 PM org.apache.beam.sdk.testutils.metrics.MetricsReader
getCounterMetric
SEVERE: Failed to get metric totalBytes.count, from namespace gbk
Load test results for test (ID): 747433ec-aca7-4e7a-9171-08d79f781853 and
timestamp: 2021-05-21T14:51:43.914000000Z:
Metric: Value:
dataflow_runtime_sec 0.0
dataflow_total_bytes_count -1.0
Exception in thread "main" java.lang.RuntimeException: Invalid test results
at
org.apache.beam.sdk.loadtests.JobFailure.handleFailure(JobFailure.java:55)
at org.apache.beam.sdk.loadtests.LoadTest.run(LoadTest.java:137)
at
org.apache.beam.sdk.loadtests.GroupByKeyLoadTest.run(GroupByKeyLoadTest.java:57)
at
org.apache.beam.sdk.loadtests.GroupByKeyLoadTest.main(GroupByKeyLoadTest.java:131)
> Task :sdks:java:testing:load-tests:run FAILED
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':sdks:java:testing:load-tests:run'.
> Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with
> non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug
option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with
Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See
https://docs.gradle.org/6.8.3/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 9m 11s
87 actionable tasks: 1 executed, 86 up-to-date
Publishing build scan...
Publishing failed.
The response from https://scans-in.gradle.com/scans/publish/gradle/3.4.1/token
was not from the build scan server.
Your network environment may be interfering, or the service may be unavailable.
If you believe this to be in error, please report this problem via
https://gradle.com/help/plugin and include the following via copy/paste:
----------
Gradle version: 6.8.3
Plugin version: 3.4.1
Request URL: https://scans-in.gradle.com/scans/publish/gradle/3.4.1/token
Request ID: 76e87cb0-be5b-4fd6-bc40-74fe18b17203
Response status code: 521
Response content type: text/html; charset=UTF-8
Response server type: cloudflare
----------
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
