See
<https://ci-beam.apache.org/job/beam_LoadTests_Java_CoGBK_Dataflow_V2_Batch_Java17/317/display/redirect?page=changes>
Changes:
[yathu] Fix Beam Sql does not support CHAR, VARCHAR, BINARY, VARBINARY
[yathu] Remove debug leftover
[noreply] Disable `optimizeOuterThis` when building with JDK > 8 (#23902)
[noreply] [Playground] [Backend] Update playground cache component to increase
[noreply] Upgrade Akvelon editor (#23415) (#23900)
[noreply] [Website] update additional case studies layout and scss (#23555)
[noreply] [Website] add shuffle to logos (#23847)
[noreply] Remove Dataflow Portability test suite from mass_comment.py
[noreply] Clean-up DatastoreV1.java (#23892)
[noreply] Add LogElements as a Beam PTransform (#23879)
[noreply] Fix incorrect object size calculation in StateCache (#23000) (#23886)
------------------------------------------
[...truncated 45.57 KB...]
> Task :sdks:java:container:java17:copyGolangLicenses
> Task :sdks:java:container:pullLicenses
Collecting pip
Using cached pip-22.3-py3-none-any.whl (2.1 MB)
Collecting setuptools
Using cached setuptools-65.5.0-py3-none-any.whl (1.2 MB)
Collecting wheel
Using cached wheel-0.37.1-py2.py3-none-any.whl (35 kB)
Installing collected packages: pip, setuptools, wheel
Attempting uninstall: pip
Found existing installation: pip 20.0.2
Uninstalling pip-20.0.2:
Successfully uninstalled pip-20.0.2
Attempting uninstall: setuptools
Found existing installation: setuptools 44.0.0
Uninstalling setuptools-44.0.0:
Successfully uninstalled setuptools-44.0.0
Successfully installed pip-22.3 setuptools-65.5.0 wheel-0.37.1
Collecting beautifulsoup4<5.0,>=4.9.0
Using cached beautifulsoup4-4.11.1-py3-none-any.whl (128 kB)
Collecting pyyaml<6.0.0,>=3.12
Using cached PyYAML-5.4.1-cp38-cp38-manylinux1_x86_64.whl (662 kB)
Collecting tenacity<6.0,>=5.0.2
Using cached tenacity-5.1.5-py2.py3-none-any.whl (34 kB)
Collecting soupsieve>1.2
Using cached soupsieve-2.3.2.post1-py3-none-any.whl (37 kB)
Collecting six>=1.9.0
Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Installing collected packages: soupsieve, six, pyyaml, tenacity, beautifulsoup4
Successfully installed beautifulsoup4-4.11.1 pyyaml-5.4.1 six-1.16.0
soupsieve-2.3.2.post1 tenacity-5.1.5
Executing python
<https://ci-beam.apache.org/job/beam_LoadTests_Java_CoGBK_Dataflow_V2_Batch_Java17/ws/src/sdks/java/container/license_scripts/pull_licenses_java.py>
--license_index=<https://ci-beam.apache.org/job/beam_LoadTests_Java_CoGBK_Dataflow_V2_Batch_Java17/ws/src/sdks/java/container/build/reports/dependency-license/index.json>
--output_dir=<https://ci-beam.apache.org/job/beam_LoadTests_Java_CoGBK_Dataflow_V2_Batch_Java17/ws/src/sdks/java/container/build/target/java_third_party_licenses>
--dep_url_yaml=<https://ci-beam.apache.org/job/beam_LoadTests_Java_CoGBK_Dataflow_V2_Batch_Java17/ws/src/sdks/java/container/license_scripts/dep_urls_java.yaml>
--manual_license_path=<https://ci-beam.apache.org/job/beam_LoadTests_Java_CoGBK_Dataflow_V2_Batch_Java17/ws/src/sdks/java/container/license_scripts/manual_licenses>
INFO:root:Pulling license for 256 dependencies using 16 threads.
INFO:root:Replaced local file URL with
file://<https://ci-beam.apache.org/job/beam_LoadTests_Java_CoGBK_Dataflow_V2_Batch_Java17/ws/src/sdks/java/container/license_scripts/manual_licenses/xz/COPYING>
for xz-1.5
INFO:root:pull_licenses_java.py succeed. It took 14.24912 seconds with 16
threads.
Copying licenses from
<https://ci-beam.apache.org/job/beam_LoadTests_Java_CoGBK_Dataflow_V2_Batch_Java17/ws/src/sdks/java/container/build/target/java_third_party_licenses>
to
<https://ci-beam.apache.org/job/beam_LoadTests_Java_CoGBK_Dataflow_V2_Batch_Java17/ws/src/sdks/java/container/build/target/third_party_licenses.>
Finished license_scripts.sh
> Task :sdks:java:container:goPrepare UP-TO-DATE
> Task :sdks:java:container:java17:copyJavaThirdPartyLicenses
> Task :sdks:java:container:goBuild
/home/jenkins/go/bin/go1.18.1 build -o ./build/target/linux_amd64/boot boot.go
boot_test.go
> Task :sdks:java:container:java17:copySdkHarnessLauncher
Execution optimizations have been disabled for task
':sdks:java:container:java17:copySdkHarnessLauncher' to ensure correctness due
to the following reasons:
- Gradle detected a problem with the following location:
'<https://ci-beam.apache.org/job/beam_LoadTests_Java_CoGBK_Dataflow_V2_Batch_Java17/ws/src/sdks/java/container/build/target'.>
Reason: Task ':sdks:java:container:java17:copySdkHarnessLauncher' uses this
output of task ':sdks:java:container:downloadCloudProfilerAgent' without
declaring an explicit or implicit dependency. This can lead to incorrect
results being produced, depending on what order the tasks are executed. Please
refer to
https://docs.gradle.org/7.5.1/userguide/validation_problems.html#implicit_dependency
for more details about this problem.
- Gradle detected a problem with the following location:
'<https://ci-beam.apache.org/job/beam_LoadTests_Java_CoGBK_Dataflow_V2_Batch_Java17/ws/src/sdks/java/container/build/target'.>
Reason: Task ':sdks:java:container:java17:copySdkHarnessLauncher' uses this
output of task ':sdks:java:container:pullLicenses' without declaring an
explicit or implicit dependency. This can lead to incorrect results being
produced, depending on what order the tasks are executed. Please refer to
https://docs.gradle.org/7.5.1/userguide/validation_problems.html#implicit_dependency
for more details about this problem.
> Task :sdks:java:container:java17:dockerPrepare
> Task :sdks:java:container:java17:docker
> Task :runners:google-cloud-dataflow-java:buildAndPushDockerJavaContainer
WARNING: `gcloud docker` will not be supported for Docker client versions above
18.03.
As an alternative, use `gcloud auth configure-docker` to configure `docker` to
use `gcloud` as a credential helper, then use `docker` as you would for non-GCR
registries, e.g. `docker pull gcr.io/project-id/my-image`. Add
`--verbosity=error` to silence this warning: `gcloud docker
--verbosity=error -- pull gcr.io/project-id/my-image`.
See:
https://cloud.google.com/container-registry/docs/support/deprecation-notices#gcloud-docker
The push refers to repository
[us.gcr.io/apache-beam-testing/java-postcommit-it/java]
742faa9ac1d3: Preparing
3ae227d9c3aa: Preparing
c35e21d23e6c: Preparing
93e4fcb08245: Preparing
9d25bc7e1671: Preparing
89262fe06120: Preparing
8afa2ac809ed: Preparing
7361505ce7c3: Preparing
38654cbd3df0: Preparing
45c68d66ee16: Preparing
8acc410b1612: Preparing
42e717b9396e: Preparing
33dce71897fe: Preparing
1ec8ed606aee: Preparing
afda89aae6c0: Preparing
3bc383470c05: Preparing
e93827457889: Preparing
08fa02ce37eb: Preparing
a037458de4e0: Preparing
bafdbe68e4ae: Preparing
a13c519c6361: Preparing
afda89aae6c0: Waiting
33dce71897fe: Waiting
3bc383470c05: Waiting
89262fe06120: Waiting
bafdbe68e4ae: Waiting
1ec8ed606aee: Waiting
08fa02ce37eb: Waiting
e93827457889: Waiting
8afa2ac809ed: Waiting
a13c519c6361: Waiting
7361505ce7c3: Waiting
45c68d66ee16: Waiting
42e717b9396e: Waiting
38654cbd3df0: Waiting
8acc410b1612: Waiting
c35e21d23e6c: Pushed
3ae227d9c3aa: Pushed
93e4fcb08245: Pushed
9d25bc7e1671: Pushed
742faa9ac1d3: Pushed
8afa2ac809ed: Pushed
7361505ce7c3: Pushed
45c68d66ee16: Pushed
38654cbd3df0: Pushed
89262fe06120: Pushed
42e717b9396e: Pushed
3bc383470c05: Layer already exists
e93827457889: Layer already exists
08fa02ce37eb: Layer already exists
8acc410b1612: Pushed
a037458de4e0: Layer already exists
bafdbe68e4ae: Layer already exists
a13c519c6361: Layer already exists
1ec8ed606aee: Pushed
afda89aae6c0: Pushed
33dce71897fe: Pushed
20221101144843: digest:
sha256:9a4a02dd5046b2761d2d806ab3a9bae4348081658e67f919acd73c26e33974bd size:
4729
> Task :sdks:java:testing:load-tests:run
Nov 01, 2022 2:49:53 PM
org.apache.beam.runners.dataflow.options.DataflowPipelineOptions$StagingLocationFactory
create
INFO: No stagingLocation provided, falling back to gcpTempLocation
Nov 01, 2022 2:49:54 PM org.apache.beam.runners.dataflow.DataflowRunner
fromOptions
INFO: PipelineOptions.filesToStage was not specified. Defaulting to files from
the classpath: will stage 229 files. Enable logging at DEBUG level to see which
files will be staged.
Nov 01, 2022 2:49:55 PM org.apache.beam.sdk.Pipeline validate
WARNING: The following transforms do not have stable unique names: Window.Into()
Nov 01, 2022 2:49:55 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Executing pipeline on the Dataflow Service, which will have billing
implications related to Google Compute Engine usage and other Google Cloud
Services.
Nov 01, 2022 2:49:57 PM org.apache.beam.runners.dataflow.util.PackageUtil
stageClasspathElements
INFO: Uploading 229 files from PipelineOptions.filesToStage to staging location
to prepare for execution.
Nov 01, 2022 2:49:57 PM org.apache.beam.runners.dataflow.util.PackageUtil
stageClasspathElements
INFO: Staging files complete: 229 files cached, 0 files newly uploaded in 0
seconds
Nov 01, 2022 2:49:57 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging portable pipeline proto to
gs://temp-storage-for-perf-tests/loadtests/staging/
Nov 01, 2022 2:49:57 PM org.apache.beam.runners.dataflow.util.PackageUtil
tryStagePackage
INFO: Uploading <117229 bytes, hash
19ee4a5e70538386d5cbf8d1251ca3521d98189ddf6ec321c304a1cbb5b6a42b> to
gs://temp-storage-for-perf-tests/loadtests/staging/pipeline-Ge5KXnBTg4bVy_jRJRyjUh2YGJ3fbsMhwwShy7W2pCs.pb
Nov 01, 2022 2:49:59 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read input as step s1
Nov 01, 2022 2:49:59 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Collect start time metrics (input) as step s2
Nov 01, 2022 2:49:59 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Window.Into()/Window.Assign as step s3
Nov 01, 2022 2:49:59 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read co-input as step s4
Nov 01, 2022 2:49:59 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Collect start time metrics (co-input) as step s5
Nov 01, 2022 2:49:59 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Window.Into()2/Window.Assign as step s6
Nov 01, 2022 2:49:59 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding CoGroupByKey/MakeUnionTable0 as step s7
Nov 01, 2022 2:49:59 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding CoGroupByKey/MakeUnionTable1 as step s8
Nov 01, 2022 2:49:59 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding CoGroupByKey/Flatten as step s9
Nov 01, 2022 2:49:59 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding CoGroupByKey/GBK as step s10
Nov 01, 2022 2:49:59 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding CoGroupByKey/ConstructCoGbkResultFn as step s11
Nov 01, 2022 2:49:59 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Ungroup and reiterate as step s12
Nov 01, 2022 2:49:59 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Collect total bytes as step s13
Nov 01, 2022 2:49:59 PM
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Collect end time metrics as step s14
Nov 01, 2022 2:49:59 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Dataflow SDK version: 2.44.0-SNAPSHOT
Nov 01, 2022 2:50:02 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to
https://console.cloud.google.com/dataflow/jobs/us-central1/2022-11-01_07_49_59-4583948089430459787?project=apache-beam-testing
Nov 01, 2022 2:50:02 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Submitted job: 2022-11-01_07_49_59-4583948089430459787
Nov 01, 2022 2:50:02 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel
> --region=us-central1 2022-11-01_07_49_59-4583948089430459787
Nov 01, 2022 2:50:04 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
WARNING: 2022-11-01T14:50:04.036Z: The workflow name is not a valid Cloud
Label. Labels applied to Cloud resources (such as GCE Instances) for monitoring
will be labeled with this modified job name:
load0tests0java170dataflow0v20batch0cogbk01-jenkins-110114-71n6. For the best
monitoring experience, please name your job with a valid Cloud Label. For
details, see:
https://cloud.google.com/compute/docs/labeling-resources#restrictions
Nov 01, 2022 2:50:04 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
SEVERE: 2022-11-01T14:50:04.298Z: Staged package
amazon-kinesis-client-1.14.2-qtBh3OuKuLE7S26ITTJzYsoYfG-mWjqbGmv3VJPhec8.jar at
location
'gs://temp-storage-for-perf-tests/loadtests/staging/amazon-kinesis-client-1.14.2-qtBh3OuKuLE7S26ITTJzYsoYfG-mWjqbGmv3VJPhec8.jar'
is inaccessible.
Nov 01, 2022 2:50:08 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
SEVERE: 2022-11-01T14:50:08.466Z: Staged package
jakarta.activation-api-1.2.2-oYepORA671hJp6-EvX4nvi0SDEEK8pFDc3X_4GH08J0.jar at
location
'gs://temp-storage-for-perf-tests/loadtests/staging/jakarta.activation-api-1.2.2-oYepORA671hJp6-EvX4nvi0SDEEK8pFDc3X_4GH08J0.jar'
is inaccessible.
Nov 01, 2022 2:50:10 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
SEVERE: 2022-11-01T14:50:10.320Z: Workflow failed. Causes: One or more access
checks for temp location or staged files failed. Please refer to other error
messages for details. For more information on security and permissions, please
see https://cloud.google.com/dataflow/security-and-permissions.
Nov 01, 2022 2:50:10 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-11-01T14:50:10.363Z: Cleaning up.
Nov 01, 2022 2:50:10 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-11-01T14:50:10.421Z: Worker pool stopped.
Nov 01, 2022 2:50:12 PM
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-11-01T14:50:11.638Z: Your project already contains 100
Dataflow-created metric descriptors, so new user metrics of the form
custom.googleapis.com/* will not be created. However, all user metrics are also
available in the metric dataflow.googleapis.com/job/user_counter. If you rely
on the custom metrics, you can delete old / unused metric descriptors. See
https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list
and
https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
Nov 01, 2022 2:50:35 PM org.apache.beam.runners.dataflow.DataflowPipelineJob
logTerminalState
INFO: Job 2022-11-01_07_49_59-4583948089430459787 failed with status FAILED.
Nov 01, 2022 2:50:35 PM org.apache.beam.sdk.testutils.metrics.MetricsReader
getCounterMetric
SEVERE: Failed to get metric totalBytes.count, from namespace co_gbk
Load test results for test (ID): f35299f9-82ca-48fb-90ea-138c43439dc6 and
timestamp: 2022-11-01T14:49:54.798000000Z:
Metric: Value:
dataflow_v2_java17_runtime_sec 0.0
dataflow_v2_java17_total_bytes_count -1.0
Exception in thread "main" java.lang.RuntimeException: Invalid job state:
FAILED.
at
org.apache.beam.sdk.loadtests.JobFailure.handleFailure(JobFailure.java:51)
at org.apache.beam.sdk.loadtests.LoadTest.run(LoadTest.java:139)
at
org.apache.beam.sdk.loadtests.CoGroupByKeyLoadTest.run(CoGroupByKeyLoadTest.java:62)
at
org.apache.beam.sdk.loadtests.CoGroupByKeyLoadTest.main(CoGroupByKeyLoadTest.java:157)
> Task :sdks:java:testing:load-tests:run FAILED
> Task :runners:google-cloud-dataflow-java:cleanUpDockerJavaImages
Untagged: us.gcr.io/apache-beam-testing/java-postcommit-it/java:20221101144843
Untagged:
us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:9a4a02dd5046b2761d2d806ab3a9bae4348081658e67f919acd73c26e33974bd
WARNING: Successfully resolved tag to sha256, but it is recommended to use
sha256 directly.
Tag: [us.gcr.io/apache-beam-testing/java-postcommit-it/java:20221101144843]
- referencing digest:
[us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:9a4a02dd5046b2761d2d806ab3a9bae4348081658e67f919acd73c26e33974bd]
Deleted [[us.gcr.io/apache-beam-testing/java-postcommit-it/java:20221101144843]
(referencing
[us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:9a4a02dd5046b2761d2d806ab3a9bae4348081658e67f919acd73c26e33974bd])].
Removing untagged image
us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:9a4a02dd5046b2761d2d806ab3a9bae4348081658e67f919acd73c26e33974bd
Digests:
-
us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:9a4a02dd5046b2761d2d806ab3a9bae4348081658e67f919acd73c26e33974bd
Deleted
[us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:9a4a02dd5046b2761d2d806ab3a9bae4348081658e67f919acd73c26e33974bd].
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':sdks:java:testing:load-tests:run'.
> Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with
> non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with
Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings
and determine if they come from your own scripts or plugins.
See
https://docs.gradle.org/7.5.1/userguide/command_line_interface.html#sec:command_line_warnings
Execution optimizations have been disabled for 1 invalid unit(s) of work during
this build to ensure correctness.
Please consult deprecation warnings for more details.
BUILD FAILED in 2m 13s
113 actionable tasks: 76 executed, 33 from cache, 4 up-to-date
Publishing build scan...
https://gradle.com/s/urujbkvrcrcxe
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
