See <https://ci-beam.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_V2_Streaming_Java11/455/display/redirect?page=changes>
Changes: [noreply] updated the pydoc for running a custom model on Beam (#23218) [noreply] Add drop_example flag to the RunInference and Model Handler (#23266) ------------------------------------------ [...truncated 43.32 KB...] Collecting beautifulsoup4<5.0,>=4.9.0 Using cached beautifulsoup4-4.11.1-py3-none-any.whl (128 kB) Collecting pyyaml<6.0.0,>=3.12 Using cached PyYAML-5.4.1-cp38-cp38-manylinux1_x86_64.whl (662 kB) Collecting tenacity<6.0,>=5.0.2 Using cached tenacity-5.1.5-py2.py3-none-any.whl (34 kB) Collecting soupsieve>1.2 Using cached soupsieve-2.3.2.post1-py3-none-any.whl (37 kB) Collecting six>=1.9.0 Using cached six-1.16.0-py2.py3-none-any.whl (11 kB) Installing collected packages: soupsieve, six, pyyaml, tenacity, beautifulsoup4 Successfully installed beautifulsoup4-4.11.1 pyyaml-5.4.1 six-1.16.0 soupsieve-2.3.2.post1 tenacity-5.1.5 Executing python <https://ci-beam.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_V2_Streaming_Java11/ws/src/sdks/java/container/license_scripts/pull_licenses_java.py> --license_index=<https://ci-beam.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_V2_Streaming_Java11/ws/src/sdks/java/container/build/reports/dependency-license/index.json> --output_dir=<https://ci-beam.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_V2_Streaming_Java11/ws/src/sdks/java/container/build/target/java_third_party_licenses> --dep_url_yaml=<https://ci-beam.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_V2_Streaming_Java11/ws/src/sdks/java/container/license_scripts/dep_urls_java.yaml> --manual_license_path=<https://ci-beam.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_V2_Streaming_Java11/ws/src/sdks/java/container/license_scripts/manual_licenses> INFO:root:Pulling license for 256 dependencies using 16 threads. INFO:root:Replaced local file URL with file://<https://ci-beam.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_V2_Streaming_Java11/ws/src/sdks/java/container/license_scripts/manual_licenses/xz/COPYING> for xz-1.5 INFO:root:pull_licenses_java.py succeed. It took 8.755874 seconds with 16 threads. Copying licenses from <https://ci-beam.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_V2_Streaming_Java11/ws/src/sdks/java/container/build/target/java_third_party_licenses> to <https://ci-beam.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_V2_Streaming_Java11/ws/src/sdks/java/container/build/target/third_party_licenses.> Finished license_scripts.sh > Task :sdks:java:container:goPrepare UP-TO-DATE > Task :release:go-licenses:java:dockerRun go: added github.com/apache/beam/sdks/v2 v2.41.0 go: added github.com/golang/protobuf v1.5.2 go: added golang.org/x/net v0.0.0-20220624214902-1bab6f366d9e go: added golang.org/x/sys v0.0.0-20220624220833-87e55d714810 go: added golang.org/x/text v0.3.7 go: added google.golang.org/genproto v0.0.0-20220714211235-042d03aeabc9 go: added google.golang.org/grpc v1.48.0 go: added google.golang.org/protobuf v1.28.0 + go-licenses save github.com/apache/beam/sdks/v2/java/container --save_path=/output/licenses W0918 12:04:59.961206 27 library.go:94] "golang.org/x/sys/unix" contains non-Go code that can't be inspected for further dependencies: /go/pkg/mod/golang.org/x/[email protected]/unix/asm_linux_amd64.s + go-licenses csv github.com/apache/beam/sdks/v2/java/container + tee /output/licenses/list.csv W0918 12:05:04.910869 185 library.go:94] "golang.org/x/sys/unix" contains non-Go code that can't be inspected for further dependencies: /go/pkg/mod/golang.org/x/[email protected]/unix/asm_linux_amd64.s github.com/apache/beam/sdks/v2/go/pkg/beam,https://github.com/apache/beam/blob/sdks/v2.41.0/sdks/go/README.md,Apache-2.0 github.com/apache/beam/sdks/v2/java/container,https://github.com/apache/beam/blob/sdks/v2.41.0/sdks/LICENSE,Apache-2.0 github.com/golang/protobuf,https://github.com/golang/protobuf/blob/v1.5.2/LICENSE,BSD-3-Clause golang.org/x/net,https://cs.opensource.google/go/x/net/+/1bab6f36:LICENSE,BSD-3-Clause golang.org/x/sys,https://cs.opensource.google/go/x/sys/+/87e55d71:LICENSE,BSD-3-Clause golang.org/x/text,https://cs.opensource.google/go/x/text/+/v0.3.7:LICENSE,BSD-3-Clause google.golang.org/genproto/googleapis/rpc/status,https://github.com/googleapis/go-genproto/blob/042d03aeabc9/LICENSE,Apache-2.0 google.golang.org/grpc,https://github.com/grpc/grpc-go/blob/v1.48.0/LICENSE,Apache-2.0 google.golang.org/protobuf,https://github.com/protocolbuffers/protobuf-go/blob/v1.28.0/LICENSE,BSD-3-Clause + chmod -R a+w /output/licenses > Task :sdks:java:container:java11:copyJavaThirdPartyLicenses > Task :sdks:java:container:goBuild /home/jenkins/go/bin/go1.18.1 build -o ./build/target/linux_amd64/boot boot.go boot_test.go > Task :sdks:java:container:java11:copySdkHarnessLauncher Execution optimizations have been disabled for task ':sdks:java:container:java11:copySdkHarnessLauncher' to ensure correctness due to the following reasons: - Gradle detected a problem with the following location: '<https://ci-beam.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_V2_Streaming_Java11/ws/src/sdks/java/container/build/target'.> Reason: Task ':sdks:java:container:java11:copySdkHarnessLauncher' uses this output of task ':sdks:java:container:downloadCloudProfilerAgent' without declaring an explicit or implicit dependency. This can lead to incorrect results being produced, depending on what order the tasks are executed. Please refer to https://docs.gradle.org/7.5.1/userguide/validation_problems.html#implicit_dependency for more details about this problem. - Gradle detected a problem with the following location: '<https://ci-beam.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_V2_Streaming_Java11/ws/src/sdks/java/container/build/target'.> Reason: Task ':sdks:java:container:java11:copySdkHarnessLauncher' uses this output of task ':sdks:java:container:pullLicenses' without declaring an explicit or implicit dependency. This can lead to incorrect results being produced, depending on what order the tasks are executed. Please refer to https://docs.gradle.org/7.5.1/userguide/validation_problems.html#implicit_dependency for more details about this problem. > Task :release:go-licenses:java:createLicenses > Task :sdks:java:container:java11:copyGolangLicenses > Task :sdks:java:container:java11:dockerPrepare > Task :sdks:java:container:java11:docker > Task :runners:google-cloud-dataflow-java:buildAndPushDockerJavaContainer WARNING: `gcloud docker` will not be supported for Docker client versions above 18.03. As an alternative, use `gcloud auth configure-docker` to configure `docker` to use `gcloud` as a credential helper, then use `docker` as you would for non-GCR registries, e.g. `docker pull gcr.io/project-id/my-image`. Add `--verbosity=error` to silence this warning: `gcloud docker --verbosity=error -- pull gcr.io/project-id/my-image`. See: https://cloud.google.com/container-registry/docs/support/deprecation-notices#gcloud-docker The push refers to repository [us.gcr.io/apache-beam-testing/java-postcommit-it/java] fa8e93918967: Preparing dd8d5a050115: Preparing 28f4141904d7: Preparing 3807d7d1ed88: Preparing 8bcb96c72dae: Preparing 6f4644dab427: Preparing 9057f81f3e4b: Preparing 1094ca92e7a2: Preparing 10341f4356ed: Preparing 35c61a3de42b: Preparing 3e94d99c28c7: Preparing 8f64fcb925eb: Preparing 57666cd64726: Preparing ad1fc204ba1c: Preparing 380c9e18bf38: Preparing 7b7f3078e1db: Preparing 826c3ddbb29c: Preparing b626401ef603: Preparing 9b55156abf26: Preparing 293d5db30c9f: Preparing 03127cdb479b: Preparing 9c742cd6c7a5: Preparing 9057f81f3e4b: Waiting 380c9e18bf38: Waiting 1094ca92e7a2: Waiting 6f4644dab427: Waiting ad1fc204ba1c: Waiting 7b7f3078e1db: Waiting 8f64fcb925eb: Waiting 57666cd64726: Waiting 826c3ddbb29c: Waiting 293d5db30c9f: Waiting 10341f4356ed: Waiting 9b55156abf26: Waiting 35c61a3de42b: Waiting 3e94d99c28c7: Waiting 03127cdb479b: Waiting dd8d5a050115: Pushed 3807d7d1ed88: Pushed 28f4141904d7: Pushed 8bcb96c72dae: Pushed fa8e93918967: Pushed 9057f81f3e4b: Pushed 35c61a3de42b: Pushed 1094ca92e7a2: Pushed 6f4644dab427: Pushed 10341f4356ed: Pushed 3e94d99c28c7: Pushed 7b7f3078e1db: Layer already exists 826c3ddbb29c: Layer already exists b626401ef603: Layer already exists 9b55156abf26: Layer already exists 293d5db30c9f: Layer already exists 03127cdb479b: Layer already exists 9c742cd6c7a5: Layer already exists 8f64fcb925eb: Pushed ad1fc204ba1c: Pushed 380c9e18bf38: Pushed 57666cd64726: Pushed 20220918120443: digest: sha256:33fbfab05ba5291c07a4aa79ee4c40b38af88648ba93c4f9e9437d4ee3c63819 size: 4935 > Task :sdks:java:testing:load-tests:run Sep 18, 2022 12:05:48 PM org.apache.beam.runners.dataflow.options.DataflowPipelineOptions$StagingLocationFactory create INFO: No stagingLocation provided, falling back to gcpTempLocation Sep 18, 2022 12:05:48 PM org.apache.beam.runners.dataflow.DataflowRunner fromOptions INFO: PipelineOptions.filesToStage was not specified. Defaulting to files from the classpath: will stage 227 files. Enable logging at DEBUG level to see which files will be staged. Sep 18, 2022 12:05:49 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Executing pipeline on the Dataflow Service, which will have billing implications related to Google Compute Engine usage and other Google Cloud Services. Sep 18, 2022 12:05:51 PM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements INFO: Uploading 227 files from PipelineOptions.filesToStage to staging location to prepare for execution. Sep 18, 2022 12:05:52 PM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements INFO: Staging files complete: 227 files cached, 0 files newly uploaded in 0 seconds Sep 18, 2022 12:05:52 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Staging portable pipeline proto to gs://temp-storage-for-perf-tests/loadtests/staging/ Sep 18, 2022 12:05:52 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <100095 bytes, hash b00f99b6933b13ef9b83f6eb45fc3e5a1823053a0158fe95567051c685143965> to gs://temp-storage-for-perf-tests/loadtests/staging/pipeline-sA-ZtpM7E--bg_brRfw-WhgjBToBWP6VVnBRxoUUOWU.pb Sep 18, 2022 12:05:54 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Read input/DataflowRunner.StreamingUnboundedRead.ReadWithIds as step s1 Sep 18, 2022 12:05:54 PM org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource split INFO: Split into 20 bundles: [SyntheticUnboundedSource{startOffset=0, endOffset=10000000}, SyntheticUnboundedSource{startOffset=10000000, endOffset=20000000}, SyntheticUnboundedSource{startOffset=20000000, endOffset=30000000}, SyntheticUnboundedSource{startOffset=30000000, endOffset=40000000}, SyntheticUnboundedSource{startOffset=40000000, endOffset=50000000}, SyntheticUnboundedSource{startOffset=50000000, endOffset=60000000}, SyntheticUnboundedSource{startOffset=60000000, endOffset=70000000}, SyntheticUnboundedSource{startOffset=70000000, endOffset=80000000}, SyntheticUnboundedSource{startOffset=80000000, endOffset=90000000}, SyntheticUnboundedSource{startOffset=90000000, endOffset=100000000}, SyntheticUnboundedSource{startOffset=100000000, endOffset=110000000}, SyntheticUnboundedSource{startOffset=110000000, endOffset=120000000}, SyntheticUnboundedSource{startOffset=120000000, endOffset=130000000}, SyntheticUnboundedSource{startOffset=130000000, endOffset=140000000}, SyntheticUnboundedSource{startOffset=140000000, endOffset=150000000}, SyntheticUnboundedSource{startOffset=150000000, endOffset=160000000}, SyntheticUnboundedSource{startOffset=160000000, endOffset=170000000}, SyntheticUnboundedSource{startOffset=170000000, endOffset=180000000}, SyntheticUnboundedSource{startOffset=180000000, endOffset=190000000}, SyntheticUnboundedSource{startOffset=190000000, endOffset=200000000}] Sep 18, 2022 12:05:54 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Read input/StripIds as step s2 Sep 18, 2022 12:05:54 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Collect start time metrics as step s3 Sep 18, 2022 12:05:54 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Total bytes monitor as step s4 Sep 18, 2022 12:05:54 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Window.Into()/Window.Assign as step s5 Sep 18, 2022 12:05:54 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Group by key (0) as step s6 Sep 18, 2022 12:05:54 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Ungroup and reiterate (0) as step s7 Sep 18, 2022 12:05:54 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Collect end time metrics (0) as step s8 Sep 18, 2022 12:05:54 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Dataflow SDK version: 2.43.0-SNAPSHOT Sep 18, 2022 12:05:55 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2022-09-18_05_05_54-11126304123864579436?project=apache-beam-testing Sep 18, 2022 12:05:55 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Submitted job: 2022-09-18_05_05_54-11126304123864579436 Sep 18, 2022 12:05:55 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: To cancel the job using the 'gcloud' tool, run: > gcloud dataflow jobs --project=apache-beam-testing cancel > --region=us-central1 2022-09-18_05_05_54-11126304123864579436 Sep 18, 2022 12:05:59 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process WARNING: 2022-09-18T12:05:58.243Z: The workflow name is not a valid Cloud Label. Labels applied to Cloud resources (such as GCE Instances) for monitoring will be labeled with this modified job name: load0tests0java110dataflow0v20streaming0gbk01-jenkins-0918-d8p3. For the best monitoring experience, please name your job with a valid Cloud Label. For details, see: https://cloud.google.com/compute/docs/labeling-resources#restrictions Sep 18, 2022 12:05:59 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process SEVERE: 2022-09-18T12:05:58.960Z: Staged package arrow-memory-netty-5.0.0-go167T751tVSdjySwEiyo5BIdHbgHBniwUvnBhvpEHs.jar at location 'gs://temp-storage-for-perf-tests/loadtests/staging/arrow-memory-netty-5.0.0-go167T751tVSdjySwEiyo5BIdHbgHBniwUvnBhvpEHs.jar' is inaccessible. Sep 18, 2022 12:05:59 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process SEVERE: 2022-09-18T12:05:59.172Z: Staged package auto-value-annotations-1.9--lRp9MRO5Zii2PAzqwqdy8ZJigxeDJmN-gwq31E1gEQ.jar at location 'gs://temp-storage-for-perf-tests/loadtests/staging/auto-value-annotations-1.9--lRp9MRO5Zii2PAzqwqdy8ZJigxeDJmN-gwq31E1gEQ.jar' is inaccessible. Sep 18, 2022 12:06:05 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process SEVERE: 2022-09-18T12:06:04.504Z: Workflow failed. Causes: One or more access checks for temp location or staged files failed. Please refer to other error messages for details. For more information on security and permissions, please see https://cloud.google.com/dataflow/security-and-permissions. Sep 18, 2022 12:06:05 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-09-18T12:06:04.543Z: Cleaning up. Sep 18, 2022 12:06:05 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-09-18T12:06:04.591Z: Worker pool stopped. Sep 18, 2022 12:06:05 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2022-09-18T12:06:05.074Z: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete Sep 18, 2022 12:06:15 PM org.apache.beam.runners.dataflow.DataflowPipelineJob logTerminalState INFO: Job 2022-09-18_05_05_54-11126304123864579436 failed with status FAILED. Sep 18, 2022 12:06:15 PM org.apache.beam.sdk.testutils.metrics.MetricsReader getCounterMetric SEVERE: Failed to get metric totalBytes.count, from namespace gbk Load test results for test (ID): 35255607-4e96-4167-935b-f03bec9802fb and timestamp: 2022-09-18T12:05:49.244000000Z: Metric: Value: dataflow_v2_java11_runtime_sec 0.0 dataflow_v2_java11_total_bytes_count -1.0 Exception in thread "main" java.lang.RuntimeException: Invalid job state: FAILED. at org.apache.beam.sdk.loadtests.JobFailure.handleFailure(JobFailure.java:51) at org.apache.beam.sdk.loadtests.LoadTest.run(LoadTest.java:139) at org.apache.beam.sdk.loadtests.GroupByKeyLoadTest.run(GroupByKeyLoadTest.java:57) at org.apache.beam.sdk.loadtests.GroupByKeyLoadTest.main(GroupByKeyLoadTest.java:131) > Task :sdks:java:testing:load-tests:run FAILED > Task :runners:google-cloud-dataflow-java:cleanUpDockerJavaImages Untagged: us.gcr.io/apache-beam-testing/java-postcommit-it/java:20220918120443 Untagged: us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:33fbfab05ba5291c07a4aa79ee4c40b38af88648ba93c4f9e9437d4ee3c63819 Tag: [us.gcr.io/apache-beam-testing/java-postcommit-it/java:20220918120443] - referencing digest: [us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:33fbfab05ba5291c07a4aa79ee4c40b38af88648ba93c4f9e9437d4ee3c63819] Deleted [[us.gcr.io/apache-beam-testing/java-postcommit-it/java:20220918120443] (referencing [us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:33fbfab05ba5291c07a4aa79ee4c40b38af88648ba93c4f9e9437d4ee3c63819])]. Removing untagged image us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:33fbfab05ba5291c07a4aa79ee4c40b38af88648ba93c4f9e9437d4ee3c63819 Digests: - us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:33fbfab05ba5291c07a4aa79ee4c40b38af88648ba93c4f9e9437d4ee3c63819 Deleted [us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:33fbfab05ba5291c07a4aa79ee4c40b38af88648ba93c4f9e9437d4ee3c63819]. FAILURE: Build failed with an exception. * What went wrong: Execution failed for task ':sdks:java:testing:load-tests:run'. > Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with > non-zero exit value 1 * Try: > Run with --stacktrace option to get the stack trace. > Run with --info or --debug option to get more log output. * Get more help at https://help.gradle.org Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0. You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins. See https://docs.gradle.org/7.5.1/userguide/command_line_interface.html#sec:command_line_warnings Execution optimizations have been disabled for 1 invalid unit(s) of work during this build to ensure correctness. Please consult deprecation warnings for more details. BUILD FAILED in 1m 54s 110 actionable tasks: 73 executed, 33 from cache, 4 up-to-date Publishing build scan... https://gradle.com/s/ny7asngqgusx4 Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
