See <https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/9888/display/redirect?page=changes>
Changes: [noreply] RunInference Benchmarks UI (#23371) [noreply] set upper bound on google-cloud-profiler (#23354) ------------------------------------------ [...truncated 233.58 KB...] [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:751 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0927213101-283620-otkribgs.1664314261.283945/torchvision-0.13.1-cp37-cp37m-manylinux1_x86_64.whl in 0 seconds. [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0927213101-283620-otkribgs.1664314261.283945/matplotlib-3.5.2-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl... [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:751 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0927213101-283620-otkribgs.1664314261.283945/matplotlib-3.5.2-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl in 0 seconds. [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0927213101-283620-otkribgs.1664314261.283945/matplotlib-3.5.2-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.whl... [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:751 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0927213101-283620-otkribgs.1664314261.283945/matplotlib-3.5.2-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.whl in 0 seconds. [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0927213101-283620-otkribgs.1664314261.283945/Pillow-9.2.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl... [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:751 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0927213101-283620-otkribgs.1664314261.283945/Pillow-9.2.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl in 0 seconds. [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0927213101-283620-otkribgs.1664314261.283945/matplotlib-3.5.2-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl... [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:751 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0927213101-283620-otkribgs.1664314261.283945/matplotlib-3.5.2-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl in 0 seconds. [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0927213101-283620-otkribgs.1664314261.283945/matplotlib-3.6.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl... [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:751 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0927213101-283620-otkribgs.1664314261.283945/matplotlib-3.6.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl in 0 seconds. [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0927213101-283620-otkribgs.1664314261.283945/matplotlib-3.5.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl... [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:751 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0927213101-283620-otkribgs.1664314261.283945/matplotlib-3.5.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl in 0 seconds. [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0927213101-283620-otkribgs.1664314261.283945/dataflow_python_sdk.tar... [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:751 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0927213101-283620-otkribgs.1664314261.283945/dataflow_python_sdk.tar in 0 seconds. [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0927213101-283620-otkribgs.1664314261.283945/dataflow-worker.jar... [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:751 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0927213101-283620-otkribgs.1664314261.283945/dataflow-worker.jar in 5 seconds. [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0927213101-283620-otkribgs.1664314261.283945/pipeline.pb... [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:751 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0927213101-283620-otkribgs.1664314261.283945/pipeline.pb in 0 seconds. [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:911 Create job: <Job clientRequestId: '20220927213101285887-2161' createTime: '2022-09-27T21:31:46.848491Z' currentStateTime: '1970-01-01T00:00:00Z' id: '2022-09-27_14_31_46-1052856712156872999' location: 'us-central1' name: 'beamapp-jenkins-0927213101-283620-otkribgs' projectId: 'apache-beam-testing' stageStates: [] startTime: '2022-09-27T21:31:46.848491Z' steps: [] tempFiles: [] type: TypeValueValuesEnum(JOB_TYPE_STREAMING, 2)> [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:913 Created job with id: [2022-09-27_14_31_46-1052856712156872999] [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:914 Submitted job: 2022-09-27_14_31_46-1052856712156872999 [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:920 To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2022-09-27_14_31_46-1052856712156872999?project=apache-beam-testing [32mINFO [0m apache_beam.runners.dataflow.test_dataflow_runner:test_dataflow_runner.py:58 Console log: [32mINFO [0m apache_beam.runners.dataflow.test_dataflow_runner:test_dataflow_runner.py:59 https://console.cloud.google.com/dataflow/jobs/us-central1/2022-09-27_14_31_46-1052856712156872999?project=apache-beam-testing [33mWARNING [0m apache_beam.runners.dataflow.test_dataflow_runner:test_dataflow_runner.py:65 Waiting indefinitely for streaming job. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:197 Job 2022-09-27_14_31_46-1052856712156872999 is in state JOB_STATE_RUNNING [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T21:31:46.375Z: JOB_MESSAGE_BASIC: Streaming Engine auto-enabled. Use --experiments=disable_streaming_engine to opt out. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T21:31:46.378Z: JOB_MESSAGE_BASIC: Dataflow Runner V2 auto-enabled. Use --experiments=disable_runner_v2 to opt out. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T21:31:47.446Z: JOB_MESSAGE_WARNING: Autoscaling is enabled for Dataflow Streaming Engine. Workers will scale between 1 and 100 unless maxNumWorkers is specified. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T21:31:47.539Z: JOB_MESSAGE_DETAILED: Autoscaling is enabled for job 2022-09-27_14_31_46-1052856712156872999. The number of workers will be between 1 and 100. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T21:31:47.572Z: JOB_MESSAGE_DETAILED: Autoscaling was automatically enabled for job 2022-09-27_14_31_46-1052856712156872999. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T21:36:58.698Z: JOB_MESSAGE_WARNING: Autoscaling is enabled for Dataflow Streaming Engine. Workers will scale between 1 and 100 unless maxNumWorkers is specified. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T21:36:58.731Z: JOB_MESSAGE_DETAILED: Autoscaling is enabled for job 2022-09-27_14_31_46-1052856712156872999. The number of workers will be between 1 and 100. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T21:36:58.759Z: JOB_MESSAGE_DETAILED: Autoscaling was automatically enabled for job 2022-09-27_14_31_46-1052856712156872999. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T21:37:01.860Z: JOB_MESSAGE_WARNING: Autoscaling is enabled for Dataflow Streaming Engine. Workers will scale between 1 and 100 unless maxNumWorkers is specified. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T21:37:01.906Z: JOB_MESSAGE_DETAILED: Autoscaling is enabled for job 2022-09-27_14_31_46-1052856712156872999. The number of workers will be between 1 and 100. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T21:37:01.942Z: JOB_MESSAGE_DETAILED: Autoscaling was automatically enabled for job 2022-09-27_14_31_46-1052856712156872999. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T21:37:06.930Z: JOB_MESSAGE_WARNING: Autoscaling is enabled for Dataflow Streaming Engine. Workers will scale between 1 and 100 unless maxNumWorkers is specified. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T21:37:06.964Z: JOB_MESSAGE_DETAILED: Autoscaling is enabled for job 2022-09-27_14_31_46-1052856712156872999. The number of workers will be between 1 and 100. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T21:37:06.998Z: JOB_MESSAGE_DETAILED: Autoscaling was automatically enabled for job 2022-09-27_14_31_46-1052856712156872999. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T21:37:09.238Z: JOB_MESSAGE_WARNING: Autoscaling is enabled for Dataflow Streaming Engine. Workers will scale between 1 and 100 unless maxNumWorkers is specified. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T21:37:09.266Z: JOB_MESSAGE_DETAILED: Autoscaling is enabled for job 2022-09-27_14_31_46-1052856712156872999. The number of workers will be between 1 and 100. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T21:37:09.301Z: JOB_MESSAGE_DETAILED: Autoscaling was automatically enabled for job 2022-09-27_14_31_46-1052856712156872999. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T21:37:12.730Z: JOB_MESSAGE_WARNING: Autoscaling is enabled for Dataflow Streaming Engine. Workers will scale between 1 and 100 unless maxNumWorkers is specified. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T21:37:12.756Z: JOB_MESSAGE_DETAILED: Autoscaling is enabled for job 2022-09-27_14_31_46-1052856712156872999. The number of workers will be between 1 and 100. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T21:37:12.786Z: JOB_MESSAGE_DETAILED: Autoscaling was automatically enabled for job 2022-09-27_14_31_46-1052856712156872999. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T21:37:14.449Z: JOB_MESSAGE_WARNING: Autoscaling is enabled for Dataflow Streaming Engine. Workers will scale between 1 and 100 unless maxNumWorkers is specified. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T21:37:14.481Z: JOB_MESSAGE_DETAILED: Autoscaling is enabled for job 2022-09-27_14_31_46-1052856712156872999. The number of workers will be between 1 and 100. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T21:37:14.512Z: JOB_MESSAGE_DETAILED: Autoscaling was automatically enabled for job 2022-09-27_14_31_46-1052856712156872999. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T21:37:16.805Z: JOB_MESSAGE_WARNING: Autoscaling is enabled for Dataflow Streaming Engine. Workers will scale between 1 and 100 unless maxNumWorkers is specified. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T21:37:16.859Z: JOB_MESSAGE_DETAILED: Autoscaling is enabled for job 2022-09-27_14_31_46-1052856712156872999. The number of workers will be between 1 and 100. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T21:37:16.892Z: JOB_MESSAGE_DETAILED: Autoscaling was automatically enabled for job 2022-09-27_14_31_46-1052856712156872999. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T21:37:17.813Z: JOB_MESSAGE_WARNING: Autoscaling is enabled for Dataflow Streaming Engine. Workers will scale between 1 and 100 unless maxNumWorkers is specified. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T21:37:17.872Z: JOB_MESSAGE_DETAILED: Autoscaling is enabled for job 2022-09-27_14_31_46-1052856712156872999. The number of workers will be between 1 and 100. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T21:37:17.906Z: JOB_MESSAGE_DETAILED: Autoscaling was automatically enabled for job 2022-09-27_14_31_46-1052856712156872999. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T21:37:18.253Z: JOB_MESSAGE_WARNING: Autoscaling is enabled for Dataflow Streaming Engine. Workers will scale between 1 and 100 unless maxNumWorkers is specified. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T21:37:18.279Z: JOB_MESSAGE_DETAILED: Autoscaling is enabled for job 2022-09-27_14_31_46-1052856712156872999. The number of workers will be between 1 and 100. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T21:37:18.313Z: JOB_MESSAGE_DETAILED: Autoscaling was automatically enabled for job 2022-09-27_14_31_46-1052856712156872999. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T21:37:21.484Z: JOB_MESSAGE_WARNING: Autoscaling is enabled for Dataflow Streaming Engine. Workers will scale between 1 and 100 unless maxNumWorkers is specified. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T21:37:21.507Z: JOB_MESSAGE_DETAILED: Autoscaling is enabled for job 2022-09-27_14_31_46-1052856712156872999. The number of workers will be between 1 and 100. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T21:37:21.534Z: JOB_MESSAGE_DETAILED: Autoscaling was automatically enabled for job 2022-09-27_14_31_46-1052856712156872999. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T21:37:24.498Z: JOB_MESSAGE_WARNING: Autoscaling is enabled for Dataflow Streaming Engine. Workers will scale between 1 and 100 unless maxNumWorkers is specified. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T21:37:24.529Z: JOB_MESSAGE_DETAILED: Autoscaling is enabled for job 2022-09-27_14_31_46-1052856712156872999. The number of workers will be between 1 and 100. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T21:37:24.565Z: JOB_MESSAGE_DETAILED: Autoscaling was automatically enabled for job 2022-09-27_14_31_46-1052856712156872999. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T21:37:35.516Z: JOB_MESSAGE_WARNING: Autoscaling is enabled for Dataflow Streaming Engine. Workers will scale between 1 and 100 unless maxNumWorkers is specified. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T21:37:35.546Z: JOB_MESSAGE_DETAILED: Autoscaling is enabled for job 2022-09-27_14_31_46-1052856712156872999. The number of workers will be between 1 and 100. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T21:37:35.569Z: JOB_MESSAGE_DETAILED: Autoscaling was automatically enabled for job 2022-09-27_14_31_46-1052856712156872999. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T21:37:45.473Z: JOB_MESSAGE_WARNING: Autoscaling is enabled for Dataflow Streaming Engine. Workers will scale between 1 and 100 unless maxNumWorkers is specified. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T21:37:45.498Z: JOB_MESSAGE_DETAILED: Autoscaling is enabled for job 2022-09-27_14_31_46-1052856712156872999. The number of workers will be between 1 and 100. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T21:37:45.525Z: JOB_MESSAGE_DETAILED: Autoscaling was automatically enabled for job 2022-09-27_14_31_46-1052856712156872999. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T21:37:48.963Z: JOB_MESSAGE_WARNING: Autoscaling is enabled for Dataflow Streaming Engine. Workers will scale between 1 and 100 unless maxNumWorkers is specified. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T21:37:48.994Z: JOB_MESSAGE_DETAILED: Autoscaling is enabled for job 2022-09-27_14_31_46-1052856712156872999. The number of workers will be between 1 and 100. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T21:37:49.025Z: JOB_MESSAGE_DETAILED: Autoscaling was automatically enabled for job 2022-09-27_14_31_46-1052856712156872999. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T21:37:49.884Z: JOB_MESSAGE_ERROR: Error processing pipeline. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T21:37:55.195Z: JOB_MESSAGE_BASIC: Worker configuration: e2-standard-2 in us-central1-b. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T21:37:56.370Z: JOB_MESSAGE_DETAILED: Expanding SplittableParDo operations into optimizable parts. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T21:37:56.394Z: JOB_MESSAGE_DETAILED: Expanding CollectionToSingleton operations into optimizable parts. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T21:37:56.471Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T21:37:56.504Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step assert_that/Group/CoGroupByKeyImpl/GroupByKey: GroupByKey not followed by a combiner. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T21:37:56.534Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step assert:odd/Group/CoGroupByKeyImpl/GroupByKey: GroupByKey not followed by a combiner. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T21:37:56.565Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step assert:even/Group/CoGroupByKeyImpl/GroupByKey: GroupByKey not followed by a combiner. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T21:37:56.595Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step Start/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey: GroupByKey not followed by a combiner. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:197 Job 2022-09-27_14_31_46-1052856712156872999 is in state JOB_STATE_FAILED [1m[31mERROR [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:1672 Console URL: https://console.cloud.google.com/dataflow/jobs/<RegionId>/2022-09-27_14_31_46-1052856712156872999?project=<ProjectId> [33m=============================== warnings summary ===============================[0m ../../build/gradleenv/-1734967052/lib/python3.7/site-packages/hdfs/config.py:15 ../../build/gradleenv/-1734967052/lib/python3.7/site-packages/hdfs/config.py:15 ../../build/gradleenv/-1734967052/lib/python3.7/site-packages/hdfs/config.py:15 ../../build/gradleenv/-1734967052/lib/python3.7/site-packages/hdfs/config.py:15 ../../build/gradleenv/-1734967052/lib/python3.7/site-packages/hdfs/config.py:15 ../../build/gradleenv/-1734967052/lib/python3.7/site-packages/hdfs/config.py:15 ../../build/gradleenv/-1734967052/lib/python3.7/site-packages/hdfs/config.py:15 ../../build/gradleenv/-1734967052/lib/python3.7/site-packages/hdfs/config.py:15 ../../build/gradleenv/-1734967052/lib/python3.7/site-packages/hdfs/config.py:15 <https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/build/gradleenv/-1734967052/lib/python3.7/site-packages/hdfs/config.py>:15: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses from imp import load_source ../../build/gradleenv/-1734967052/lib/python3.7/site-packages/google/api_core/grpc_helpers.py:42 ../../build/gradleenv/-1734967052/lib/python3.7/site-packages/google/api_core/grpc_helpers.py:42 ../../build/gradleenv/-1734967052/lib/python3.7/site-packages/google/api_core/grpc_helpers.py:42 ../../build/gradleenv/-1734967052/lib/python3.7/site-packages/google/api_core/grpc_helpers.py:42 ../../build/gradleenv/-1734967052/lib/python3.7/site-packages/google/api_core/grpc_helpers.py:42 ../../build/gradleenv/-1734967052/lib/python3.7/site-packages/google/api_core/grpc_helpers.py:42 ../../build/gradleenv/-1734967052/lib/python3.7/site-packages/google/api_core/grpc_helpers.py:42 ../../build/gradleenv/-1734967052/lib/python3.7/site-packages/google/api_core/grpc_helpers.py:42 ../../build/gradleenv/-1734967052/lib/python3.7/site-packages/google/api_core/grpc_helpers.py:42 <https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/build/gradleenv/-1734967052/lib/python3.7/site-packages/google/api_core/grpc_helpers.py>:42: DeprecationWarning: Support for grpcio-gcp is deprecated. This feature will be removed from `google-api-core` after January 1, 2024. If you need to continue to use this feature, please pin to a specific version of `google-api-core`. DeprecationWarning, -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html - generated xml file: <https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/pytest_validatesRunnerStreamingTests-df-py37-xdist.xml> - =========================== short test summary info ============================ FAILED apache_beam/transforms/ptransform_test.py::PTransformTest::test_flatten_multiple_pcollections_having_multiple_consumers [31m====== [31m[1m1 failed[0m, [32m29 passed[0m, [33m6 skipped[0m, [33m18 warnings[0m[31m in 3184.48s (0:53:04)[0m[31m =======[0m > Task :sdks:python:test-suites:dataflow:py37:validatesRunnerStreamingTests > FAILED > Task :sdks:python:test-suites:dataflow:py39:validatesRunnerStreamingTests [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:748 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0927220840-800075-nthvafm2.1664316520.800294/dataflow-worker.jar in 5 seconds. [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0927220840-800075-nthvafm2.1664316520.800294/pipeline.pb... [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:748 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0927220840-800075-nthvafm2.1664316520.800294/pipeline.pb in 0 seconds. [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:911 Create job: <Job clientRequestId: '20220927220840801808-2535' createTime: '2022-09-27T22:09:31.386626Z' currentStateTime: '1970-01-01T00:00:00Z' id: '2022-09-27_15_09_30-11586625375109801111' location: 'us-central1' name: 'beamapp-jenkins-0927220840-800075-nthvafm2' projectId: 'apache-beam-testing' stageStates: [] startTime: '2022-09-27T22:09:31.386626Z' steps: [] tempFiles: [] type: TypeValueValuesEnum(JOB_TYPE_STREAMING, 2)> [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:913 Created job with id: [2022-09-27_15_09_30-11586625375109801111] [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:914 Submitted job: 2022-09-27_15_09_30-11586625375109801111 [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:915 To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2022-09-27_15_09_30-11586625375109801111?project=apache-beam-testing Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2022-09-27_15_09_30-11586625375109801111?project=apache-beam-testing [32mINFO [0m apache_beam.runners.dataflow.test_dataflow_runner:test_dataflow_runner.py:58 Console log: [32mINFO [0m apache_beam.runners.dataflow.test_dataflow_runner:test_dataflow_runner.py:59 https://console.cloud.google.com/dataflow/jobs/us-central1/2022-09-27_15_09_30-11586625375109801111?project=apache-beam-testing [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:197 Job 2022-09-27_15_09_30-11586625375109801111 is in state JOB_STATE_RUNNING [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T22:09:30.888Z: JOB_MESSAGE_BASIC: Streaming Engine auto-enabled. Use --experiments=disable_streaming_engine to opt out. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T22:09:30.889Z: JOB_MESSAGE_BASIC: Dataflow Runner V2 auto-enabled. Use --experiments=disable_runner_v2 to opt out. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T22:09:31.900Z: JOB_MESSAGE_WARNING: Autoscaling is enabled for Dataflow Streaming Engine. Workers will scale between 1 and 100 unless maxNumWorkers is specified. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T22:09:32.074Z: JOB_MESSAGE_DETAILED: Autoscaling is enabled for job 2022-09-27_15_09_30-11586625375109801111. The number of workers will be between 1 and 100. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T22:09:32.128Z: JOB_MESSAGE_DETAILED: Autoscaling was automatically enabled for job 2022-09-27_15_09_30-11586625375109801111. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T22:09:41.527Z: JOB_MESSAGE_BASIC: Worker configuration: e2-standard-2 in us-central1-a. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T22:09:42.515Z: JOB_MESSAGE_DETAILED: Expanding SplittableParDo operations into optimizable parts. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T22:09:42.549Z: JOB_MESSAGE_DETAILED: Expanding CollectionToSingleton operations into optimizable parts. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T22:09:42.662Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T22:09:42.700Z: JOB_MESSAGE_DETAILED: Expanding SplittableProcessKeyed operations into optimizable parts. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T22:09:42.807Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into streaming Read/Write steps [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T22:09:42.833Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T22:09:42.862Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T22:09:42.903Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T22:09:42.933Z: JOB_MESSAGE_DETAILED: Fusing consumer generate_metrics into ReadFromPubSub/Read [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T22:09:42.955Z: JOB_MESSAGE_DETAILED: Fusing consumer dump_to_pub/ToProtobuf into generate_metrics [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T22:09:42.980Z: JOB_MESSAGE_DETAILED: Fusing consumer dump_to_pub/Write into dump_to_pub/ToProtobuf [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T22:09:43.022Z: JOB_MESSAGE_BASIC: Running job using Streaming Engine [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T22:09:43.050Z: JOB_MESSAGE_BASIC: Using cloud KMLS key to protect persistent state. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T22:09:43.140Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T22:09:43.174Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T22:09:43.211Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T22:09:43.245Z: JOB_MESSAGE_DEBUG: Assigning stage ids. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T22:09:43.300Z: JOB_MESSAGE_DEBUG: Starting worker pool setup. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T22:09:43.335Z: JOB_MESSAGE_BASIC: Starting 1 workers in us-central1-a... [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T22:09:43.379Z: JOB_MESSAGE_DEBUG: Starting worker pool setup. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T22:10:00.164Z: JOB_MESSAGE_BASIC: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-27T22:10:24.999Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of workers to 1 so that the pipeline can catch up with its backlog and keep up with its input rate. [33mWARNING [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:253 Timing out on waiting for job 2022-09-27_15_09_30-11586625375109801111 after 60 seconds [32mPASSED[0m [33m=============================== warnings summary ===============================[0m ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py:15 <https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py>:15: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses from imp import load_source ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/google/api_core/grpc_helpers.py:37 <https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/google/api_core/grpc_helpers.py>:37: DeprecationWarning: Support for grpcio-gcp is deprecated. This feature will be removed from `google-api-core` after January 1, 2024. If you need to continue to use this feature, please pin to a specific version of `google-api-core`. warnings.warn( ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/tenacity/_asyncio.py:42 <https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/tenacity/_asyncio.py>:42: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead def call(self, fn, *args, **kwargs): apache_beam/typehints/pandas_type_compatibility_test.py:66 <https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:66: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead. }).set_index(pd.Int64Index(range(123, 223), name='an_index')), apache_beam/typehints/pandas_type_compatibility_test.py:89 <https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:89: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead. pd.Int64Index(range(123, 223), name='an_index'), apache_beam/typehints/pandas_type_compatibility_test.py:90 <https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:90: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead. pd.Int64Index(range(475, 575), name='another_index'), -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html - generated xml file: <https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/pytest_validatesRunnerStreamingTests-df-py39-noxdist.xml> - [33m==== [32m1 passed[0m, [33m[1m3 skipped[0m, [33m[1m6668 deselected[0m, [33m[1m6 warnings[0m[33m in 799.41s (0:13:19)[0m[33m =====[0m FAILURE: Build failed with an exception. * Where: Script '<https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/test-suites/dataflow/common.gradle'> line: 264 * What went wrong: Execution failed for task ':sdks:python:test-suites:dataflow:py37:validatesRunnerStreamingTests'. > Process 'command 'sh'' finished with non-zero exit value 1 * Try: > Run with --stacktrace option to get the stack trace. > Run with --info or --debug option to get more log output. * Get more help at https://help.gradle.org Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0. You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins. See https://docs.gradle.org/7.5.1/userguide/command_line_interface.html#sec:command_line_warnings BUILD FAILED in 2h 2m 12s 100 actionable tasks: 65 executed, 33 from cache, 2 up-to-date Publishing build scan... https://gradle.com/s/hk7hr3botpvlq Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
