See <https://ci-beam.apache.org/job/beam_LoadTests_Java_CoGBK_Dataflow_V2_Streaming_Java11/78/display/redirect?page=changes>
Changes: [Steve Niemitz] [BEAM-12767] Add another unit test for PipelineOptions deserialization [aydar.zaynutdinov] [BEAM-11873] Add support for writes with returning values in JdbcIO [Andrew Pilloud] s/org.apache.beam.vendor.calcite.v1_20_0/org.apache.beam.vendor.calcite.v1_26_0/g [Andrew Pilloud] [BEAM-9379] Update to vendored Calcite to 1.26.0 [Andrew Pilloud] Fix flattened rows [Andrew Pilloud] Fix DDL [Andrew Pilloud] Handle BeamRelNode in RelSubset [Andrew Pilloud] Fix BeamIOPushDown [Andrew Pilloud] [BEAM-9190] Update BeamBigQuerySqlDialect [Andrew Pilloud] Remap IN to Search [Andrew Pilloud] [BEAM-9379] Use byte[] instead of ByteString for (VAR)BINARY in UDFs. [Andrew Pilloud] [BEAM-9379] Update UDF NULL type mismatch test since there is stricter [Andrew Pilloud] Fix ZetaSQL window function mapping [Andrew Pilloud] Fix Bigtable tests that depend on SQL types [Andrew Pilloud] Workaround CALCITE-4759 in JoinPushThroughJoinRule [Andrew Pilloud] Disable nested bytes tests, sorry! [Andrew Pilloud] SqlLine is rotting, Just CAST types for now [Andrew Pilloud] Update CHANGES.md [Andrew Pilloud] Up spotbug stack size [Andrew Pilloud] Fix BeamMatchRel copy [Andrew Pilloud] partitionKey everywhere [Andrew Pilloud] Make it functional [Andrew Pilloud] Update CreateFunction [Andrew Pilloud] No tpcds dependency [Andrew Pilloud] Fix default time types [Ankur Goenka] fixing release date [Andrew Pilloud] [BEAM-12680] Calcite SqlTransform no longer experimental [noreply] [BEAM-12823] TestStream Support in Samza Portable Runner (#15421) ------------------------------------------ [...truncated 48.20 KB...] 8555e663f65b: Waiting d00da3cd7763: Waiting 799760671c38: Preparing 799760671c38: Waiting d402f4f1b906: Waiting 00ef5416d927: Waiting 3891808a925b: Waiting cbd1847397a6: Pushed 93d8904b4346: Pushed dcc60227efea: Pushed b24966251fc9: Pushed ce552a63504e: Pushed 90a3717dea9a: Pushed ae7896080c89: Pushed a736df8d5d8a: Pushed 3891808a925b: Layer already exists d402f4f1b906: Layer already exists 00ef5416d927: Layer already exists 8555e663f65b: Layer already exists d00da3cd7763: Layer already exists 4e61e63529c2: Layer already exists 799760671c38: Layer already exists d459ac429b69: Pushed efe909e58895: Pushed 7ef68bd23097: Pushed 7c8d8b857bee: Pushed 20210903124334: digest: sha256:32234880a5257393275e5a13bbd8b69d5627be9b45e07ce70f389530435c806d size: 4311 > Task :sdks:java:testing:load-tests:run Sep 03, 2021 12:46:06 PM org.apache.beam.runners.dataflow.options.DataflowPipelineOptions$StagingLocationFactory create INFO: No stagingLocation provided, falling back to gcpTempLocation Sep 03, 2021 12:46:06 PM org.apache.beam.runners.dataflow.DataflowRunner fromOptions INFO: PipelineOptions.filesToStage was not specified. Defaulting to files from the classpath: will stage 191 files. Enable logging at DEBUG level to see which files will be staged. Sep 03, 2021 12:46:07 PM org.apache.beam.sdk.Pipeline validate WARNING: The following transforms do not have stable unique names: Window.Into() Sep 03, 2021 12:46:07 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Executing pipeline on the Dataflow Service, which will have billing implications related to Google Compute Engine usage and other Google Cloud Services. Sep 03, 2021 12:46:10 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Staging portable pipeline proto to gs://temp-storage-for-perf-tests/loadtests/staging/ Sep 03, 2021 12:46:10 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <112622 bytes, hash af91d0cf048a02657fed8ce508db571cdf75562e10530f09e7961d362ee2130c> to gs://temp-storage-for-perf-tests/loadtests/staging/pipeline-r5HQzwSKAmV_7YzlCNtXHN91Vi4QUw8J55YdNi7iEww.pb Sep 03, 2021 12:46:12 PM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements INFO: Uploading 191 files from PipelineOptions.filesToStage to staging location to prepare for execution. Sep 03, 2021 12:46:12 PM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements INFO: Staging files complete: 191 files cached, 0 files newly uploaded in 0 seconds Sep 03, 2021 12:46:12 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Read input/DataflowRunner.StreamingUnboundedRead.ReadWithIds as step s1 Sep 03, 2021 12:46:12 PM org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource split INFO: Split into 20 bundles of sizes: [org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@7afc4db9, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@1a1f5f71, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@63884e4, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@524270b8, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@4acb7ecc, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@2a4f5433, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@6812fa3a, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@29149030, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@38b8b6c0, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@68868328, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@173a6728, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@1a22e0ef, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@67514bdd, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@3b920bdc, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@7f5538a1, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@72fd8a3c, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@1f9f381d, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@34780cd9, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@623ebac7, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@7c8c70d6] Sep 03, 2021 12:46:12 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Read input/StripIds as step s2 Sep 03, 2021 12:46:12 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Collect start time metrics (input) as step s3 Sep 03, 2021 12:46:12 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Window.Into()/Window.Assign as step s4 Sep 03, 2021 12:46:12 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Read co-input/DataflowRunner.StreamingUnboundedRead.ReadWithIds as step s5 Sep 03, 2021 12:46:12 PM org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource split INFO: Split into 20 bundles of sizes: [org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@75120e58, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@48976e6d, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@2a367e93, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@7f6874f2, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@1a6dc589, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@697a34af, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@70211df5, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@4c5228e7, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@38e7ed69, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@806996, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@78b612c6, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@257e0827, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@22752544, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@21ba2445, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@69d23296, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@3c3820bb, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@376c7d7d, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@4784efd9, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@3fba233d, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@427ae189] Sep 03, 2021 12:46:12 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Read co-input/StripIds as step s6 Sep 03, 2021 12:46:12 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Collect start time metrics (co-input) as step s7 Sep 03, 2021 12:46:12 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Window.Into()2/Window.Assign as step s8 Sep 03, 2021 12:46:12 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding CoGroupByKey/MakeUnionTable0 as step s9 Sep 03, 2021 12:46:12 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding CoGroupByKey/MakeUnionTable1 as step s10 Sep 03, 2021 12:46:12 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding CoGroupByKey/Flatten as step s11 Sep 03, 2021 12:46:12 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding CoGroupByKey/GBK as step s12 Sep 03, 2021 12:46:12 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding CoGroupByKey/ConstructCoGbkResultFn as step s13 Sep 03, 2021 12:46:12 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Ungroup and reiterate as step s14 Sep 03, 2021 12:46:12 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Collect total bytes as step s15 Sep 03, 2021 12:46:12 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Collect end time metrics as step s16 Sep 03, 2021 12:46:12 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Dataflow SDK version: 2.34.0-SNAPSHOT Sep 03, 2021 12:46:13 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2021-09-03_05_46_12-16976215867526938241?project=apache-beam-testing Sep 03, 2021 12:46:13 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Submitted job: 2021-09-03_05_46_12-16976215867526938241 Sep 03, 2021 12:46:13 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: To cancel the job using the 'gcloud' tool, run: > gcloud dataflow jobs --project=apache-beam-testing cancel > --region=us-central1 2021-09-03_05_46_12-16976215867526938241 Sep 03, 2021 12:46:24 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process WARNING: 2021-09-03T12:46:24.131Z: The workflow name is not a valid Cloud Label. Labels applied to Cloud resources (such as GCE Instances) for monitoring will be labeled with this modified job name: load0tests0java110dataflow0v20streaming0cogbk01-jenkins-09-2ohq. For the best monitoring experience, please name your job with a valid Cloud Label. For details, see: https://cloud.google.com/compute/docs/labeling-resources#restrictions Sep 03, 2021 12:46:31 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-03T12:46:28.548Z: Worker configuration: e2-standard-2 in us-central1-a. Sep 03, 2021 12:46:31 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-03T12:46:29.337Z: Expanding SplittableParDo operations into optimizable parts. Sep 03, 2021 12:46:31 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-03T12:46:29.356Z: Expanding CollectionToSingleton operations into optimizable parts. Sep 03, 2021 12:46:31 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-03T12:46:29.427Z: Expanding CoGroupByKey operations into optimizable parts. Sep 03, 2021 12:46:31 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-03T12:46:29.499Z: Expanding SplittableProcessKeyed operations into optimizable parts. Sep 03, 2021 12:46:31 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-03T12:46:29.521Z: Expanding GroupByKey operations into streaming Read/Write steps Sep 03, 2021 12:46:31 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-03T12:46:29.575Z: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns Sep 03, 2021 12:46:31 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-03T12:46:29.687Z: Fusing adjacent ParDo, Read, Write, and Flatten operations Sep 03, 2021 12:46:31 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-03T12:46:29.722Z: Unzipping flatten CoGroupByKey-Flatten for input CoGroupByKey-MakeUnionTable0-ParMultiDo-ConstructUnionTable-.output Sep 03, 2021 12:46:31 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-03T12:46:29.754Z: Fusing unzipped copy of CoGroupByKey/GBK/WriteStream, through flatten CoGroupByKey/Flatten, into producer CoGroupByKey/MakeUnionTable0/ParMultiDo(ConstructUnionTable) Sep 03, 2021 12:46:31 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-03T12:46:29.782Z: Fusing consumer CoGroupByKey/GBK/WriteStream into CoGroupByKey/MakeUnionTable1/ParMultiDo(ConstructUnionTable) Sep 03, 2021 12:46:31 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-03T12:46:29.807Z: Fusing consumer Read input/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource) into Read input/Impulse Sep 03, 2021 12:46:31 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-03T12:46:29.832Z: Fusing consumer Read-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/PairWithRestriction into Read input/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource) Sep 03, 2021 12:46:31 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-03T12:46:29.864Z: Fusing consumer Read-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/SplitWithSizing into Read-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/PairWithRestriction Sep 03, 2021 12:46:31 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-03T12:46:29.889Z: Fusing consumer Read input/ParDo(StripIds)/ParMultiDo(StripIds) into Read-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/ProcessElementAndRestrictionWithSizing Sep 03, 2021 12:46:31 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-03T12:46:29.932Z: Fusing consumer Collect start time metrics (input)/ParMultiDo(TimeMonitor) into Read input/ParDo(StripIds)/ParMultiDo(StripIds) Sep 03, 2021 12:46:31 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-03T12:46:29.948Z: Fusing consumer Window.Into()/Window.Assign into Collect start time metrics (input)/ParMultiDo(TimeMonitor) Sep 03, 2021 12:46:31 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-03T12:46:29.974Z: Fusing consumer CoGroupByKey/MakeUnionTable0/ParMultiDo(ConstructUnionTable) into Window.Into()/Window.Assign Sep 03, 2021 12:46:31 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-03T12:46:30.008Z: Fusing consumer Read co-input/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource) into Read co-input/Impulse Sep 03, 2021 12:46:31 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-03T12:46:30.042Z: Fusing consumer Read-co-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/PairWithRestriction into Read co-input/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource) Sep 03, 2021 12:46:31 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-03T12:46:30.065Z: Fusing consumer Read-co-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/SplitWithSizing into Read-co-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/PairWithRestriction Sep 03, 2021 12:46:31 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-03T12:46:30.097Z: Fusing consumer Read co-input/ParDo(StripIds)/ParMultiDo(StripIds) into Read-co-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/ProcessElementAndRestrictionWithSizing Sep 03, 2021 12:46:31 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-03T12:46:30.128Z: Fusing consumer Collect start time metrics (co-input)/ParMultiDo(TimeMonitor) into Read co-input/ParDo(StripIds)/ParMultiDo(StripIds) Sep 03, 2021 12:46:31 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-03T12:46:30.164Z: Fusing consumer Window.Into()2/Window.Assign into Collect start time metrics (co-input)/ParMultiDo(TimeMonitor) Sep 03, 2021 12:46:31 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-03T12:46:30.194Z: Fusing consumer CoGroupByKey/MakeUnionTable1/ParMultiDo(ConstructUnionTable) into Window.Into()2/Window.Assign Sep 03, 2021 12:46:31 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-03T12:46:30.217Z: Fusing consumer CoGroupByKey/GBK/MergeBuckets into CoGroupByKey/GBK/ReadStream Sep 03, 2021 12:46:31 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-03T12:46:30.252Z: Fusing consumer CoGroupByKey/ConstructCoGbkResultFn/ParMultiDo(ConstructCoGbkResult) into CoGroupByKey/GBK/MergeBuckets Sep 03, 2021 12:46:31 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-03T12:46:30.282Z: Fusing consumer Ungroup and reiterate/ParMultiDo(UngroupAndReiterate) into CoGroupByKey/ConstructCoGbkResultFn/ParMultiDo(ConstructCoGbkResult) Sep 03, 2021 12:46:31 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-03T12:46:30.312Z: Fusing consumer Collect total bytes/ParMultiDo(ByteMonitor) into Ungroup and reiterate/ParMultiDo(UngroupAndReiterate) Sep 03, 2021 12:46:31 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-03T12:46:30.340Z: Fusing consumer Collect end time metrics/ParMultiDo(TimeMonitor) into Collect total bytes/ParMultiDo(ByteMonitor) Sep 03, 2021 12:46:31 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-03T12:46:30.712Z: Starting 5 ****s in us-central1-a... Sep 03, 2021 12:46:59 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-03T12:46:58.600Z: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete Sep 03, 2021 12:47:15 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-03T12:47:14.442Z: Autoscaling: Raised the number of ****s to 1 so that the pipeline can catch up with its backlog and keep up with its input rate. Sep 03, 2021 12:47:15 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-03T12:47:14.478Z: Resized **** pool to 1, though goal was 5. This could be a quota issue. Sep 03, 2021 12:47:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-03T12:47:24.748Z: Autoscaling: Raised the number of ****s to 5 so that the pipeline can catch up with its backlog and keep up with its input rate. Sep 03, 2021 12:47:54 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-03T12:47:52.761Z: Workers have started successfully. Sep 03, 2021 12:47:54 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-03T12:47:52.796Z: Workers have started successfully. Sep 03, 2021 2:13:33 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process SEVERE: 2021-09-03T14:13:31.587Z: Staged package beam-vendor-grpc-1_36_0-0.2-cBdYf-JPXEZ1XnZqk0m4-VlBsCKU_yOoV0qLHUFCl2c.jar at location 'gs://temp-storage-for-perf-tests/loadtests/staging/beam-vendor-grpc-1_36_0-0.2-cBdYf-JPXEZ1XnZqk0m4-VlBsCKU_yOoV0qLHUFCl2c.jar' is inaccessible. Sep 03, 2021 2:13:35 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process WARNING: 2021-09-03T14:13:34.633Z: One or more access checks for temp location or staged files failed. Please refer to other error messages for details. For more information on security and permissions, please see https://cloud.google.com/dataflow/security-and-permissions. Sep 03, 2021 2:16:34 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process WARNING: 2021-09-03T14:16:34.525Z: One or more access checks for temp location or staged files failed. Please refer to other error messages for details. For more information on security and permissions, please see https://cloud.google.com/dataflow/security-and-permissions. Sep 03, 2021 2:19:31 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process SEVERE: 2021-09-03T14:19:31.531Z: Staged package beam-vendor-grpc-1_36_0-0.2-cBdYf-JPXEZ1XnZqk0m4-VlBsCKU_yOoV0qLHUFCl2c.jar at location 'gs://temp-storage-for-perf-tests/loadtests/staging/beam-vendor-grpc-1_36_0-0.2-cBdYf-JPXEZ1XnZqk0m4-VlBsCKU_yOoV0qLHUFCl2c.jar' is inaccessible. Sep 03, 2021 2:19:36 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process WARNING: 2021-09-03T14:19:34.439Z: One or more access checks for temp location or staged files failed. Please refer to other error messages for details. For more information on security and permissions, please see https://cloud.google.com/dataflow/security-and-permissions. Sep 03, 2021 2:22:35 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process WARNING: 2021-09-03T14:22:34.194Z: One or more access checks for temp location or staged files failed. Please refer to other error messages for details. For more information on security and permissions, please see https://cloud.google.com/dataflow/security-and-permissions. Sep 03, 2021 2:25:34 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process SEVERE: 2021-09-03T14:25:31.835Z: Staged package beam-vendor-grpc-1_36_0-0.2-cBdYf-JPXEZ1XnZqk0m4-VlBsCKU_yOoV0qLHUFCl2c.jar at location 'gs://temp-storage-for-perf-tests/loadtests/staging/beam-vendor-grpc-1_36_0-0.2-cBdYf-JPXEZ1XnZqk0m4-VlBsCKU_yOoV0qLHUFCl2c.jar' is inaccessible. Sep 03, 2021 2:25:37 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process WARNING: 2021-09-03T14:25:34.477Z: One or more access checks for temp location or staged files failed. Please refer to other error messages for details. For more information on security and permissions, please see https://cloud.google.com/dataflow/security-and-permissions. Sep 03, 2021 2:28:36 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process WARNING: 2021-09-03T14:28:34.175Z: One or more access checks for temp location or staged files failed. Please refer to other error messages for details. For more information on security and permissions, please see https://cloud.google.com/dataflow/security-and-permissions. Sep 03, 2021 2:31:32 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process SEVERE: 2021-09-03T14:31:31.514Z: Staged package beam-vendor-grpc-1_36_0-0.2-cBdYf-JPXEZ1XnZqk0m4-VlBsCKU_yOoV0qLHUFCl2c.jar at location 'gs://temp-storage-for-perf-tests/loadtests/staging/beam-vendor-grpc-1_36_0-0.2-cBdYf-JPXEZ1XnZqk0m4-VlBsCKU_yOoV0qLHUFCl2c.jar' is inaccessible. Sep 03, 2021 2:31:35 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process WARNING: 2021-09-03T14:31:34.276Z: One or more access checks for temp location or staged files failed. Please refer to other error messages for details. For more information on security and permissions, please see https://cloud.google.com/dataflow/security-and-permissions. Sep 03, 2021 4:00:40 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-03T16:00:38.701Z: Cancel request is committed for workflow job: 2021-09-03_05_46_12-16976215867526938241. Sep 03, 2021 4:00:40 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-03T16:00:38.744Z: Cleaning up. Sep 03, 2021 4:00:40 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-03T16:00:38.809Z: Stopping **** pool... Sep 03, 2021 4:00:40 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-03T16:00:38.859Z: Stopping **** pool... Sep 03, 2021 4:03:13 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-03T16:03:12.716Z: Autoscaling: Reduced the number of ****s to 0 based on low average **** CPU utilization, and the pipeline having sufficiently low backlog and keeping up with input rate. Sep 03, 2021 4:03:13 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2021-09-03T16:03:12.742Z: Worker pool stopped. Sep 03, 2021 4:03:18 PM org.apache.beam.runners.dataflow.DataflowPipelineJob logTerminalState INFO: Job 2021-09-03_05_46_12-16976215867526938241 finished with status CANCELLED. Load test results for test (ID): bcf151dc-c01a-4d95-9d1f-cb49973589ba and timestamp: 2021-09-03T12:46:07.243000000Z: Metric: Value: dataflow_v2_java11_runtime_sec 11513.842 dataflow_v2_java11_total_bytes_count 2.5381647E10 Exception in thread "main" java.lang.RuntimeException: Invalid job state: CANCELLED. at org.apache.beam.sdk.loadtests.JobFailure.handleFailure(JobFailure.java:55) at org.apache.beam.sdk.loadtests.LoadTest.run(LoadTest.java:141) at org.apache.beam.sdk.loadtests.CoGroupByKeyLoadTest.run(CoGroupByKeyLoadTest.java:62) at org.apache.beam.sdk.loadtests.CoGroupByKeyLoadTest.main(CoGroupByKeyLoadTest.java:157) > Task :sdks:java:testing:load-tests:run FAILED > Task :runners:google-cloud-dataflow-java:cleanUpDockerImages Untagged: us.gcr.io/apache-beam-testing/java-postcommit-it/java:20210903124334 Untagged: us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:32234880a5257393275e5a13bbd8b69d5627be9b45e07ce70f389530435c806d Tag: [us.gcr.io/apache-beam-testing/java-postcommit-it/java:20210903124334] - referencing digest: [us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:32234880a5257393275e5a13bbd8b69d5627be9b45e07ce70f389530435c806d] Deleted [[us.gcr.io/apache-beam-testing/java-postcommit-it/java:20210903124334] (referencing [us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:32234880a5257393275e5a13bbd8b69d5627be9b45e07ce70f389530435c806d])]. Removing untagged image us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:32234880a5257393275e5a13bbd8b69d5627be9b45e07ce70f389530435c806d Digests: - us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:32234880a5257393275e5a13bbd8b69d5627be9b45e07ce70f389530435c806d Deleted [us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:32234880a5257393275e5a13bbd8b69d5627be9b45e07ce70f389530435c806d]. FAILURE: Build failed with an exception. * What went wrong: Execution failed for task ':sdks:java:testing:load-tests:run'. > Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with > non-zero exit value 1 * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. * Get more help at https://help.gradle.org Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/6.8.3/userguide/command_line_interface.html#sec:command_line_warnings BUILD FAILED in 3h 20m 3s 101 actionable tasks: 72 executed, 27 from cache, 2 up-to-date Publishing build scan... https://gradle.com/s/rh53tfhycxrzw Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
