See <https://ci-beam.apache.org/job/beam_LoadTests_Java_ParDo_Dataflow_V2_Streaming_Java11/55/display/redirect?page=changes>
Changes: [andyxu] Add support for python sdk container to turn on google cloud profiler [andyxu] format file [marwant] Update Google Cloud Healthcare API version from v1beta1 to GA [chamikaramj] Fix typo [zyichi] Update beam dataflow python container versions. [noreply] [BEAM-12403][BEAM-12348] Add User State Support for Portable SamzaRunner ------------------------------------------ [...truncated 68.75 KB...] INFO: Executing pipeline on the Dataflow Service, which will have billing implications related to Google Compute Engine usage and other Google Cloud Services. Aug 11, 2021 12:43:04 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Staging portable pipeline proto to gs://temp-storage-for-perf-tests/loadtests/staging/ Aug 11, 2021 12:43:04 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <107841 bytes, hash 01d84f7e5e8263bffe580e14fba687460077d953a582efe8ec6c696d40be7482> to gs://temp-storage-for-perf-tests/loadtests/staging/pipeline-AdhPfl6CY7_-WA4U-6aHRgB32VOlgu_o7GxpbUC-dII.pb Aug 11, 2021 12:43:05 PM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements INFO: Uploading 191 files from PipelineOptions.filesToStage to staging location to prepare for execution. Aug 11, 2021 12:43:07 PM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements INFO: Staging files complete: 191 files cached, 0 files newly uploaded in 2 seconds Aug 11, 2021 12:43:07 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Read input/DataflowRunner.StreamingUnboundedRead.ReadWithIds as step s1 Aug 11, 2021 12:43:07 PM org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource split INFO: Split into 20 bundles of sizes: [org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@6342d610, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@dc4a691, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@784abd3e, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@36c2b646, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@434514d8, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@6b1dc20f, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@4613311f, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@6540cf1d, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@ec8f4b9, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@bc042d5, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@5484117b, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@37df14d1, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@7efb53af, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@7724704f, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@3dfa819, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@4ce94d2f, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@68ab0936, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@3cd9aa64, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@42b84286, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@443effcb] Aug 11, 2021 12:43:07 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Read input/StripIds as step s2 Aug 11, 2021 12:43:07 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding ParDo(TimeMonitor) as step s3 Aug 11, 2021 12:43:07 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding ParDo(ByteMonitor) as step s4 Aug 11, 2021 12:43:07 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 0 as step s5 Aug 11, 2021 12:43:07 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 1 as step s6 Aug 11, 2021 12:43:07 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 2 as step s7 Aug 11, 2021 12:43:07 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 3 as step s8 Aug 11, 2021 12:43:07 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 4 as step s9 Aug 11, 2021 12:43:07 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 5 as step s10 Aug 11, 2021 12:43:07 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 6 as step s11 Aug 11, 2021 12:43:07 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 7 as step s12 Aug 11, 2021 12:43:07 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 8 as step s13 Aug 11, 2021 12:43:07 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Step: 9 as step s14 Aug 11, 2021 12:43:07 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding ParDo(TimeMonitor)2 as step s15 Aug 11, 2021 12:43:08 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Dataflow SDK version: 2.33.0-SNAPSHOT Aug 11, 2021 12:43:08 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2021-08-11_05_43_08-16348139797564905653?project=apache-beam-testing Aug 11, 2021 12:43:08 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Submitted job: 2021-08-11_05_43_08-16348139797564905653 Aug 11, 2021 12:43:08 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: To cancel the job using the 'gcloud' tool, run: > gcloud dataflow jobs --project=apache-beam-testing cancel > --region=us-central1 2021-08-11_05_43_08-16348139797564905653 Aug 11, 2021 12:43:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process WARNING: 2021-08-11T12:43:16.997Z: The workflow name is not a valid Cloud Label. Labels applied to Cloud resources (such as GCE Instances) for monitoring will be labeled with this modified job name: load0tests0java110dataflow0v20streaming0pardo01-jenkins-08-yew8. For the best monitoring experience, please name your job with a valid Cloud Label. For details, see: https://cloud.google.com/compute/docs/labeling-resources#restrictions Aug 11, 2021 12:43:23 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-08-11T12:43:22.531Z: Worker configuration: e2-standard-2 in us-central1-a. Aug 11, 2021 12:43:23 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-08-11T12:43:23.329Z: Expanding SplittableParDo operations into optimizable parts. Aug 11, 2021 12:43:23 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-08-11T12:43:23.359Z: Expanding CollectionToSingleton operations into optimizable parts. Aug 11, 2021 12:43:23 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-08-11T12:43:23.412Z: Expanding CoGroupByKey operations into optimizable parts. Aug 11, 2021 12:43:23 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-08-11T12:43:23.447Z: Expanding SplittableProcessKeyed operations into optimizable parts. Aug 11, 2021 12:43:24 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-08-11T12:43:23.471Z: Expanding GroupByKey operations into streaming Read/Write steps Aug 11, 2021 12:43:24 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-08-11T12:43:23.498Z: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns Aug 11, 2021 12:43:24 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-08-11T12:43:23.571Z: Fusing adjacent ParDo, Read, Write, and Flatten operations Aug 11, 2021 12:43:24 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-08-11T12:43:23.591Z: Fusing consumer Read input/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource) into Read input/Impulse Aug 11, 2021 12:43:24 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-08-11T12:43:23.614Z: Fusing consumer Read-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/PairWithRestriction into Read input/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource) Aug 11, 2021 12:43:24 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-08-11T12:43:23.638Z: Fusing consumer Read-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/SplitWithSizing into Read-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/PairWithRestriction Aug 11, 2021 12:43:24 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-08-11T12:43:23.659Z: Fusing consumer Read input/ParDo(StripIds)/ParMultiDo(StripIds) into Read-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/ProcessElementAndRestrictionWithSizing Aug 11, 2021 12:43:24 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-08-11T12:43:23.687Z: Fusing consumer ParDo(TimeMonitor)/ParMultiDo(TimeMonitor) into Read input/ParDo(StripIds)/ParMultiDo(StripIds) Aug 11, 2021 12:43:24 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-08-11T12:43:23.718Z: Fusing consumer ParDo(ByteMonitor)/ParMultiDo(ByteMonitor) into ParDo(TimeMonitor)/ParMultiDo(TimeMonitor) Aug 11, 2021 12:43:24 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-08-11T12:43:23.746Z: Fusing consumer Step: 0/ParMultiDo(CounterOperation) into ParDo(ByteMonitor)/ParMultiDo(ByteMonitor) Aug 11, 2021 12:43:24 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-08-11T12:43:23.769Z: Fusing consumer Step: 1/ParMultiDo(CounterOperation) into Step: 0/ParMultiDo(CounterOperation) Aug 11, 2021 12:43:24 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-08-11T12:43:23.790Z: Fusing consumer Step: 2/ParMultiDo(CounterOperation) into Step: 1/ParMultiDo(CounterOperation) Aug 11, 2021 12:43:24 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-08-11T12:43:23.817Z: Fusing consumer Step: 3/ParMultiDo(CounterOperation) into Step: 2/ParMultiDo(CounterOperation) Aug 11, 2021 12:43:24 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-08-11T12:43:23.839Z: Fusing consumer Step: 4/ParMultiDo(CounterOperation) into Step: 3/ParMultiDo(CounterOperation) Aug 11, 2021 12:43:24 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-08-11T12:43:23.885Z: Fusing consumer Step: 5/ParMultiDo(CounterOperation) into Step: 4/ParMultiDo(CounterOperation) Aug 11, 2021 12:43:24 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-08-11T12:43:23.912Z: Fusing consumer Step: 6/ParMultiDo(CounterOperation) into Step: 5/ParMultiDo(CounterOperation) Aug 11, 2021 12:43:24 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-08-11T12:43:23.934Z: Fusing consumer Step: 7/ParMultiDo(CounterOperation) into Step: 6/ParMultiDo(CounterOperation) Aug 11, 2021 12:43:24 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-08-11T12:43:23.968Z: Fusing consumer Step: 8/ParMultiDo(CounterOperation) into Step: 7/ParMultiDo(CounterOperation) Aug 11, 2021 12:43:24 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-08-11T12:43:23.995Z: Fusing consumer Step: 9/ParMultiDo(CounterOperation) into Step: 8/ParMultiDo(CounterOperation) Aug 11, 2021 12:43:24 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-08-11T12:43:24.017Z: Fusing consumer ParDo(TimeMonitor)2/ParMultiDo(TimeMonitor) into Step: 9/ParMultiDo(CounterOperation) Aug 11, 2021 12:43:24 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-08-11T12:43:24.336Z: Starting 5 ****s in us-central1-a... Aug 11, 2021 12:43:42 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-08-11T12:43:42.067Z: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete Aug 11, 2021 12:44:11 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-08-11T12:44:11.577Z: Autoscaling: Raised the number of ****s to 5 so that the pipeline can catch up with its backlog and keep up with its input rate. Aug 11, 2021 12:45:09 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-08-11T12:45:07.334Z: Workers have started successfully. Aug 11, 2021 12:45:09 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-08-11T12:45:07.380Z: Workers have started successfully. Aug 11, 2021 2:25:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process SEVERE: 2021-08-11T14:25:24.707Z: Staged package aws-java-sdk-cloudwatch-1.11.974-Ub2yicrz1p27ushPndFcVBigc3SuIZ4G-Yaq45B1iUo.jar at location 'gs://temp-storage-for-perf-tests/loadtests/staging/aws-java-sdk-cloudwatch-1.11.974-Ub2yicrz1p27ushPndFcVBigc3SuIZ4G-Yaq45B1iUo.jar' is inaccessible. Aug 11, 2021 2:25:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process SEVERE: 2021-08-11T14:25:24.799Z: Staged package aws-java-sdk-core-1.11.974-s3lPhICKsq1xExGOyj03VQHRqOsuX5YUc-UUlnhiA64.jar at location 'gs://temp-storage-for-perf-tests/loadtests/staging/aws-java-sdk-core-1.11.974-s3lPhICKsq1xExGOyj03VQHRqOsuX5YUc-UUlnhiA64.jar' is inaccessible. Aug 11, 2021 2:25:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process SEVERE: 2021-08-11T14:25:24.847Z: Staged package aws-java-sdk-dynamodb-1.11.974-RLvIDrX-TDIDN06gNnaHm9dQbuHCD_XSBesOuniSTk4.jar at location 'gs://temp-storage-for-perf-tests/loadtests/staging/aws-java-sdk-dynamodb-1.11.974-RLvIDrX-TDIDN06gNnaHm9dQbuHCD_XSBesOuniSTk4.jar' is inaccessible. Aug 11, 2021 2:25:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process SEVERE: 2021-08-11T14:25:24.890Z: Staged package aws-java-sdk-kinesis-1.11.974-kbU_CjbRpkrhhFeoyIMQA81nTYmniJY77w8Smd7PLpY.jar at location 'gs://temp-storage-for-perf-tests/loadtests/staging/aws-java-sdk-kinesis-1.11.974-kbU_CjbRpkrhhFeoyIMQA81nTYmniJY77w8Smd7PLpY.jar' is inaccessible. Aug 11, 2021 2:25:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process SEVERE: 2021-08-11T14:25:24.930Z: Staged package aws-java-sdk-kms-1.11.974-HWH_mHclw3Wv4_GJ56x62xFDHSPSkZWjeKAlx40XrTk.jar at location 'gs://temp-storage-for-perf-tests/loadtests/staging/aws-java-sdk-kms-1.11.974-HWH_mHclw3Wv4_GJ56x62xFDHSPSkZWjeKAlx40XrTk.jar' is inaccessible. Aug 11, 2021 2:25:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process SEVERE: 2021-08-11T14:25:24.983Z: Staged package aws-java-sdk-s3-1.11.974-Qgd0wgaOuI_uyxxQI4Ulljk9bZEq-3JZArQF0zU5yzA.jar at location 'gs://temp-storage-for-perf-tests/loadtests/staging/aws-java-sdk-s3-1.11.974-Qgd0wgaOuI_uyxxQI4Ulljk9bZEq-3JZArQF0zU5yzA.jar' is inaccessible. Aug 11, 2021 2:25:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process SEVERE: 2021-08-11T14:25:25.814Z: Staged package commons-io-2.6--HfTBGYKwqFC84ZbrfyXHex-1zx0fH-NXS9ROcpzZRM.jar at location 'gs://temp-storage-for-perf-tests/loadtests/staging/commons-io-2.6--HfTBGYKwqFC84ZbrfyXHex-1zx0fH-NXS9ROcpzZRM.jar' is inaccessible. Aug 11, 2021 2:25:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process SEVERE: 2021-08-11T14:25:25.854Z: Staged package commons-lang-2.6-UPEbCfh3wpTVbyRGP0fSj5Kc9QRPZIZhwPDPuumi9Jw.jar at location 'gs://temp-storage-for-perf-tests/loadtests/staging/commons-lang-2.6-UPEbCfh3wpTVbyRGP0fSj5Kc9QRPZIZhwPDPuumi9Jw.jar' is inaccessible. Aug 11, 2021 2:25:27 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process SEVERE: 2021-08-11T14:25:27.324Z: Staged package ion-java-1.0.2-DRJ7IFofzgq8KjdXoEF0hlG8ZsFc9MBZusWDOyfUcaU.jar at location 'gs://temp-storage-for-perf-tests/loadtests/staging/ion-java-1.0.2-DRJ7IFofzgq8KjdXoEF0hlG8ZsFc9MBZusWDOyfUcaU.jar' is inaccessible. Aug 11, 2021 2:25:27 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process SEVERE: 2021-08-11T14:25:27.575Z: Staged package jmespath-java-1.11.974-QaUt5Hnw6wd7ULTQ5Oj4j_zpRr-rnMCdfGwiwkSo8-0.jar at location 'gs://temp-storage-for-perf-tests/loadtests/staging/jmespath-java-1.11.974-QaUt5Hnw6wd7ULTQ5Oj4j_zpRr-rnMCdfGwiwkSo8-0.jar' is inaccessible. Aug 11, 2021 2:25:30 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process WARNING: 2021-08-11T14:25:28.636Z: One or more access checks for temp location or staged files failed. Please refer to other error messages for details. For more information on security and permissions, please see https://cloud.google.com/dataflow/security-and-permissions. Aug 11, 2021 2:28:30 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process WARNING: 2021-08-11T14:28:28.544Z: One or more access checks for temp location or staged files failed. Please refer to other error messages for details. For more information on security and permissions, please see https://cloud.google.com/dataflow/security-and-permissions. Aug 11, 2021 2:31:25 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process SEVERE: 2021-08-11T14:31:24.857Z: Staged package aws-java-sdk-cloudwatch-1.11.974-Ub2yicrz1p27ushPndFcVBigc3SuIZ4G-Yaq45B1iUo.jar at location 'gs://temp-storage-for-perf-tests/loadtests/staging/aws-java-sdk-cloudwatch-1.11.974-Ub2yicrz1p27ushPndFcVBigc3SuIZ4G-Yaq45B1iUo.jar' is inaccessible. Aug 11, 2021 2:31:25 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process SEVERE: 2021-08-11T14:31:24.934Z: Staged package aws-java-sdk-core-1.11.974-s3lPhICKsq1xExGOyj03VQHRqOsuX5YUc-UUlnhiA64.jar at location 'gs://temp-storage-for-perf-tests/loadtests/staging/aws-java-sdk-core-1.11.974-s3lPhICKsq1xExGOyj03VQHRqOsuX5YUc-UUlnhiA64.jar' is inaccessible. Aug 11, 2021 2:31:25 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process SEVERE: 2021-08-11T14:31:24.994Z: Staged package aws-java-sdk-dynamodb-1.11.974-RLvIDrX-TDIDN06gNnaHm9dQbuHCD_XSBesOuniSTk4.jar at location 'gs://temp-storage-for-perf-tests/loadtests/staging/aws-java-sdk-dynamodb-1.11.974-RLvIDrX-TDIDN06gNnaHm9dQbuHCD_XSBesOuniSTk4.jar' is inaccessible. Aug 11, 2021 2:31:25 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process SEVERE: 2021-08-11T14:31:25.061Z: Staged package aws-java-sdk-kinesis-1.11.974-kbU_CjbRpkrhhFeoyIMQA81nTYmniJY77w8Smd7PLpY.jar at location 'gs://temp-storage-for-perf-tests/loadtests/staging/aws-java-sdk-kinesis-1.11.974-kbU_CjbRpkrhhFeoyIMQA81nTYmniJY77w8Smd7PLpY.jar' is inaccessible. Aug 11, 2021 2:31:25 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process SEVERE: 2021-08-11T14:31:25.128Z: Staged package aws-java-sdk-kms-1.11.974-HWH_mHclw3Wv4_GJ56x62xFDHSPSkZWjeKAlx40XrTk.jar at location 'gs://temp-storage-for-perf-tests/loadtests/staging/aws-java-sdk-kms-1.11.974-HWH_mHclw3Wv4_GJ56x62xFDHSPSkZWjeKAlx40XrTk.jar' is inaccessible. Aug 11, 2021 2:31:25 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process SEVERE: 2021-08-11T14:31:25.183Z: Staged package aws-java-sdk-s3-1.11.974-Qgd0wgaOuI_uyxxQI4Ulljk9bZEq-3JZArQF0zU5yzA.jar at location 'gs://temp-storage-for-perf-tests/loadtests/staging/aws-java-sdk-s3-1.11.974-Qgd0wgaOuI_uyxxQI4Ulljk9bZEq-3JZArQF0zU5yzA.jar' is inaccessible. Aug 11, 2021 2:31:28 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process SEVERE: 2021-08-11T14:31:25.995Z: Staged package commons-io-2.6--HfTBGYKwqFC84ZbrfyXHex-1zx0fH-NXS9ROcpzZRM.jar at location 'gs://temp-storage-for-perf-tests/loadtests/staging/commons-io-2.6--HfTBGYKwqFC84ZbrfyXHex-1zx0fH-NXS9ROcpzZRM.jar' is inaccessible. Aug 11, 2021 2:31:28 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process SEVERE: 2021-08-11T14:31:26.041Z: Staged package commons-lang-2.6-UPEbCfh3wpTVbyRGP0fSj5Kc9QRPZIZhwPDPuumi9Jw.jar at location 'gs://temp-storage-for-perf-tests/loadtests/staging/commons-lang-2.6-UPEbCfh3wpTVbyRGP0fSj5Kc9QRPZIZhwPDPuumi9Jw.jar' is inaccessible. Aug 11, 2021 2:31:28 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process SEVERE: 2021-08-11T14:31:27.904Z: Staged package ion-java-1.0.2-DRJ7IFofzgq8KjdXoEF0hlG8ZsFc9MBZusWDOyfUcaU.jar at location 'gs://temp-storage-for-perf-tests/loadtests/staging/ion-java-1.0.2-DRJ7IFofzgq8KjdXoEF0hlG8ZsFc9MBZusWDOyfUcaU.jar' is inaccessible. Aug 11, 2021 2:31:29 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process SEVERE: 2021-08-11T14:31:28.207Z: Staged package jmespath-java-1.11.974-QaUt5Hnw6wd7ULTQ5Oj4j_zpRr-rnMCdfGwiwkSo8-0.jar at location 'gs://temp-storage-for-perf-tests/loadtests/staging/jmespath-java-1.11.974-QaUt5Hnw6wd7ULTQ5Oj4j_zpRr-rnMCdfGwiwkSo8-0.jar' is inaccessible. Aug 11, 2021 2:31:29 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process WARNING: 2021-08-11T14:31:29.499Z: One or more access checks for temp location or staged files failed. Please refer to other error messages for details. For more information on security and permissions, please see https://cloud.google.com/dataflow/security-and-permissions. Aug 11, 2021 4:00:45 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-08-11T16:00:44.128Z: Cancel request is committed for workflow job: 2021-08-11_05_43_08-16348139797564905653. Aug 11, 2021 4:00:45 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-08-11T16:00:44.176Z: Cleaning up. Aug 11, 2021 4:00:45 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-08-11T16:00:44.241Z: Stopping **** pool... Aug 11, 2021 4:00:45 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-08-11T16:00:44.297Z: Stopping **** pool... Aug 11, 2021 4:03:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-08-11T16:03:06.338Z: Autoscaling: Reduced the number of ****s to 0 based on low average **** CPU utilization, and the pipeline having sufficiently low backlog and keeping up with input rate. Aug 11, 2021 4:03:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-08-11T16:03:06.381Z: Worker pool stopped. Aug 11, 2021 4:03:13 PM org.apache.beam.runners.dataflow.DataflowPipelineJob logTerminalState INFO: Job 2021-08-11_05_43_08-16348139797564905653 finished with status CANCELLED. Aug 11, 2021 4:03:13 PM org.apache.beam.sdk.testutils.metrics.MetricsReader getCounterMetric SEVERE: Failed to get metric totalBytes.count, from namespace pardo Load test results for test (ID): e44b22a8-573c-4b9c-bd48-a5a32b5c51f2 and timestamp: 2021-08-11T12:43:00.512000000Z: Metric: Value: dataflow_v2_java11_runtime_sec 0.0 dataflow_v2_java11_total_bytes_count -1.0 Exception in thread "main" java.lang.RuntimeException: Invalid job state: CANCELLED. at org.apache.beam.sdk.loadtests.JobFailure.handleFailure(JobFailure.java:55) at org.apache.beam.sdk.loadtests.LoadTest.run(LoadTest.java:141) at org.apache.beam.sdk.loadtests.ParDoLoadTest.run(ParDoLoadTest.java:53) at org.apache.beam.sdk.loadtests.ParDoLoadTest.main(ParDoLoadTest.java:103) > Task :sdks:java:testing:load-tests:run FAILED > Task :runners:google-cloud-dataflow-java:cleanUpDockerImages Untagged: us.gcr.io/apache-beam-testing/java-postcommit-it/java:20210811123735 Untagged: us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:b3e9b01b7e9f7a0a0103494426627606154ded3dcd26a926e87b1830cdf25981 Deleted: sha256:66c6627c55ba301e94ca741656548295090f3fab70862cc3b8273a54b39b7beb Deleted: sha256:92c66b9ee7749f1a1c97d91cd75c4cde8cc374d5020c5346eb26f88cc2605736 Deleted: sha256:3a1b4e6dd5f7fee7577cdd4995e309a0b6cb6988e915f4488a99a72891f95085 Deleted: sha256:9b699c1e1347a4afa626171e3ff773054ecb6e2f8634229b50b48e0c12a9bbc0 Deleted: sha256:6382cd861933f91a82894c8273e68539c98e5fefeb55e96e4e942840a484f065 Deleted: sha256:808818194bff09d6e1d460f991cd9bb8778f7c75d594aa384e944ca8f6a8984b Deleted: sha256:c408e9c6d281c78f9d30e0516031964427f60aaef3bda1c7cd35ab62e4c6269d Deleted: sha256:6c424a318c0ab8cad53ab4b040d562b701ffc7b34d4c18499584fc62774db9cf Deleted: sha256:2967bbc7b15237f9f5ceb133602bac796f8b89a2cffa2d1f7dad933d820f5bc4 Deleted: sha256:d58e51fc13b24ec8750d6c5dcb57a9c7a7f35438240dc15a8aaffed563890762 Deleted: sha256:262097195bd44f26da35efbc0849e6e0d21493c771efc83eb6607cf62591cd04 Deleted: sha256:0a12a61ed3617dcf12c911e6903ca04b384491f0aeca02bee9cc1be89472167c Deleted: sha256:51e5632ec08fdfd276acc6ca217914694a1480a82bf21dad0b5c56eddee17543 Deleted: sha256:966114a541fde11dc31d3c2ef7893dad4a2b7e9bd394aac93743596e4287a687 Deleted: sha256:a5c70c5b0e5c7d83c6c93c8f1dc0a486a2be0547f92f3822c37f74d4e2695b77 Deleted: sha256:b148b2096c5b8d92bbdc192f0a0493d938c9d1e65812b68a966f1bc1fcb6c807 Deleted: sha256:b31f92e22d28bdcf3e408cb933f02d8c903f7ae6bfc140d8fbfeda03296a47dc Deleted: sha256:f766133d23897123a9c1c0bbe581cbc6c9cb4c89333762a1800bfcb0a7d89b9e Deleted: sha256:be0701f41c56bcd3febff41895bbeba2d5ede67a368a1cb98c8cd570a21b3f5d Deleted: sha256:80449b2c23bf830bb82d363b235c3be963d8a6f609a6fc1d2ecd070f4536f155 Deleted: sha256:fdcda69db5af937de5f5d7de572ca6832d519bd36efaa60b76d93c1a05e1a9f5 Deleted: sha256:f8193691aea946fc19440044b90c8503d73aa52247a502002a0276e9c74e0e65 Deleted: sha256:cd0a4e67aaaa7e8715bc0e1d5defab74cdf59f382939937a099dc11297c238da Deleted: sha256:8d5bb0b0fc1a195145301f8c99891db73836a9521c0f01f055ed6844c78cde56 Deleted: sha256:f24cfb0265722a048f7d2b2b06001bf842fa921c4112c6f51627841f149a1ea5 Deleted: sha256:ede3add900e115ab2f4d5dd4cb6fa8b54b9877c10881e5c5c99c4daaa0ea1c82 Deleted: sha256:88e3ee0ecc14dc594c337eb1c44ee0d66b030ec74f1d159dde2ec0d50347cceb Deleted: sha256:03ac0b20c60f37fca69e5395104ec05650bc5edbe4560c5f548b0a000d32523d Tag: [us.gcr.io/apache-beam-testing/java-postcommit-it/java:20210811123735] - referencing digest: [us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:b3e9b01b7e9f7a0a0103494426627606154ded3dcd26a926e87b1830cdf25981] Deleted [[us.gcr.io/apache-beam-testing/java-postcommit-it/java:20210811123735] (referencing [us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:b3e9b01b7e9f7a0a0103494426627606154ded3dcd26a926e87b1830cdf25981])]. Removing untagged image us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:b3e9b01b7e9f7a0a0103494426627606154ded3dcd26a926e87b1830cdf25981 Digests: - us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:b3e9b01b7e9f7a0a0103494426627606154ded3dcd26a926e87b1830cdf25981 Deleted [us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:b3e9b01b7e9f7a0a0103494426627606154ded3dcd26a926e87b1830cdf25981]. FAILURE: Build failed with an exception. * What went wrong: Execution failed for task ':sdks:java:testing:load-tests:run'. > Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with > non-zero exit value 1 * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. * Get more help at https://help.gradle.org Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/6.8.3/userguide/command_line_interface.html#sec:command_line_warnings BUILD FAILED in 3h 25m 57s 107 actionable tasks: 79 executed, 26 from cache, 2 up-to-date Publishing build scan... https://gradle.com/s/ybvnmxrb6ausg Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
