See <https://ci-beam.apache.org/job/beam_python_mongoio_load_test/3890/display/redirect?page=changes>
Changes: [noreply] Pass user specified destination type to UpdateSchemaDestination [noreply] [Go SDK] Stream decode values in single iterations (#22904) [noreply] Enable autosharding for BQ: #22818 [noreply] Update wordcount_minimal.py by removing pipeline_args.extend (#22786) [noreply] Add map state in the Go Sdk (#22897) [noreply] [BEAM-12164] Feat: Added support to Cloud Spanner Change Streams ------------------------------------------ [...truncated 127.25 KB...] INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:15:24.695Z: JOB_MESSAGE_DETAILED: Fusing consumer Flatten/InputIdentity into Count/CombineGlobally(CountCombineFn)/InjectDefault INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:15:24.727Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Create/FlatMap(<lambda at core.py:3481>) into assert_that/Create/Impulse INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:15:24.761Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Create/Map(decode) into assert_that/Create/FlatMap(<lambda at core.py:3481>) INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:15:24.834Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Group/CoGroupByKeyImpl/Tag[0] into assert_that/Create/Map(decode) INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:15:24.895Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec. INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:15:24.925Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph. INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:15:24.977Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps. INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:15:25.029Z: JOB_MESSAGE_DEBUG: Assigning stage ids. INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:15:25.203Z: JOB_MESSAGE_DEBUG: Executing wait step start73 INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:15:25.271Z: JOB_MESSAGE_BASIC: Executing operation ReadFromMongoDB/Read/Impulse+ReadFromMongoDB/Read/Map(<lambda at iobase.py:908>)+ref_AppliedPTransform_ReadFromMongoDB-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_7/PairWithRestriction+ref_AppliedPTransform_ReadFromMongoDB-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_7/SplitWithSizing INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:15:25.321Z: JOB_MESSAGE_DEBUG: Starting worker pool setup. INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:15:25.350Z: JOB_MESSAGE_BASIC: Starting 5 workers in us-central1-a... INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-08-26_14_15_18-13436952191853648136 is in state JOB_STATE_RUNNING INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:15:31.691Z: JOB_MESSAGE_BASIC: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:16:02.909Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of workers to 5 based on the rate of progress in the currently running stage(s). INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:16:35.598Z: JOB_MESSAGE_DETAILED: Workers have started successfully. INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:18:54.975Z: JOB_MESSAGE_DETAILED: Autoscaling: Reduced the number of workers to 1 based on the rate of progress in the currently running stage(s). INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:18:55.027Z: JOB_MESSAGE_DETAILED: Autoscaling: Would further reduce the number of workers but reached the minimum number allowed for the job. INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:19:00.070Z: JOB_MESSAGE_DETAILED: Autoscaling: Resizing worker pool from 5 to 1. INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:22:04.114Z: JOB_MESSAGE_BASIC: Finished operation ReadFromMongoDB/Read/Impulse+ReadFromMongoDB/Read/Map(<lambda at iobase.py:908>)+ref_AppliedPTransform_ReadFromMongoDB-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_7/PairWithRestriction+ref_AppliedPTransform_ReadFromMongoDB-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_7/SplitWithSizing INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:22:04.187Z: JOB_MESSAGE_DEBUG: Value "ref_AppliedPTransform_ReadFromMongoDB-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_7-split-with-sizing-out3" materialized. INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:22:04.267Z: JOB_MESSAGE_BASIC: Executing operation Combine/CombinePerKey/GroupByKey/Create INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:22:04.300Z: JOB_MESSAGE_BASIC: Executing operation Count/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Create INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:22:05.807Z: JOB_MESSAGE_BASIC: Finished operation Combine/CombinePerKey/GroupByKey/Create INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:22:05.874Z: JOB_MESSAGE_DEBUG: Value "Combine/CombinePerKey/GroupByKey/Session" materialized. INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:22:07.001Z: JOB_MESSAGE_BASIC: Finished operation Count/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Create INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:22:07.072Z: JOB_MESSAGE_DEBUG: Value "Count/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Session" materialized. INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:22:07.167Z: JOB_MESSAGE_BASIC: Executing operation ref_AppliedPTransform_ReadFromMongoDB-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_7/ProcessElementAndRestrictionWithSizing+Map+Combine/KeyWithVoid+Count/CombineGlobally(CountCombineFn)/KeyWithVoid+Combine/CombinePerKey/GroupByKey+Combine/CombinePerKey/Combine/Partial+Combine/CombinePerKey/GroupByKey/Write+Count/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey+Count/CombineGlobally(CountCombineFn)/CombinePerKey/Combine/Partial+Count/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Write INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:22:32.643Z: JOB_MESSAGE_BASIC: Finished operation ref_AppliedPTransform_ReadFromMongoDB-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_7/ProcessElementAndRestrictionWithSizing+Map+Combine/KeyWithVoid+Count/CombineGlobally(CountCombineFn)/KeyWithVoid+Combine/CombinePerKey/GroupByKey+Combine/CombinePerKey/Combine/Partial+Combine/CombinePerKey/GroupByKey/Write+Count/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey+Count/CombineGlobally(CountCombineFn)/CombinePerKey/Combine/Partial+Count/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Write INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:22:32.715Z: JOB_MESSAGE_BASIC: Executing operation Combine/CombinePerKey/GroupByKey/Close INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:22:32.748Z: JOB_MESSAGE_BASIC: Executing operation Count/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Close INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:22:33.758Z: JOB_MESSAGE_BASIC: Finished operation Count/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Close INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:22:33.776Z: JOB_MESSAGE_BASIC: Finished operation Combine/CombinePerKey/GroupByKey/Close INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:22:33.836Z: JOB_MESSAGE_BASIC: Executing operation Count/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Read+Count/CombineGlobally(CountCombineFn)/CombinePerKey/Combine+Count/CombineGlobally(CountCombineFn)/CombinePerKey/Combine/Extract+Count/CombineGlobally(CountCombineFn)/UnKey INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:22:33.880Z: JOB_MESSAGE_BASIC: Executing operation Combine/CombinePerKey/GroupByKey/Read+Combine/CombinePerKey/Combine+Combine/CombinePerKey/Combine/Extract+Combine/UnKey INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:22:35.147Z: JOB_MESSAGE_BASIC: Finished operation Combine/CombinePerKey/GroupByKey/Read+Combine/CombinePerKey/Combine+Combine/CombinePerKey/Combine/Extract+Combine/UnKey INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:22:35.222Z: JOB_MESSAGE_DEBUG: Value "Combine/UnKey.None" materialized. INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:22:35.288Z: JOB_MESSAGE_BASIC: Executing operation Combine/InjectDefault/View-python_side_input0-Combine/InjectDefault INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:22:35.336Z: JOB_MESSAGE_BASIC: Finished operation Combine/InjectDefault/View-python_side_input0-Combine/InjectDefault INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:22:35.406Z: JOB_MESSAGE_DEBUG: Value "Combine/InjectDefault/View-python_side_input0-Combine/InjectDefault.out" materialized. INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:22:37.528Z: JOB_MESSAGE_BASIC: Finished operation Count/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Read+Count/CombineGlobally(CountCombineFn)/CombinePerKey/Combine+Count/CombineGlobally(CountCombineFn)/CombinePerKey/Combine/Extract+Count/CombineGlobally(CountCombineFn)/UnKey INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:22:37.591Z: JOB_MESSAGE_DEBUG: Value "Count/CombineGlobally(CountCombineFn)/UnKey.None" materialized. INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:22:37.705Z: JOB_MESSAGE_BASIC: Executing operation Count/CombineGlobally(CountCombineFn)/InjectDefault/View-python_side_input0-Count/CombineGlobally(CountCombineFn)/InjectDefault INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:22:37.756Z: JOB_MESSAGE_BASIC: Finished operation Count/CombineGlobally(CountCombineFn)/InjectDefault/View-python_side_input0-Count/CombineGlobally(CountCombineFn)/InjectDefault INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:22:37.814Z: JOB_MESSAGE_DEBUG: Value "Count/CombineGlobally(CountCombineFn)/InjectDefault/View-python_side_input0-Count/CombineGlobally(CountCombineFn)/InjectDefault.out" materialized. INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:22:37.897Z: JOB_MESSAGE_BASIC: Executing operation assert_that/Group/CoGroupByKeyImpl/GroupByKey/Create INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:22:38.052Z: JOB_MESSAGE_BASIC: Finished operation assert_that/Group/CoGroupByKeyImpl/GroupByKey/Create INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:22:38.121Z: JOB_MESSAGE_DEBUG: Value "assert_that/Group/CoGroupByKeyImpl/GroupByKey/Session" materialized. INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:22:38.193Z: JOB_MESSAGE_BASIC: Executing operation Combine/DoOnce/Impulse+Combine/DoOnce/FlatMap(<lambda at core.py:3481>)+Combine/DoOnce/Map(decode)+Combine/InjectDefault+assert_that/WindowInto(WindowIntoFn)+assert_that/ToVoidKey+assert_that/Group/CoGroupByKeyImpl/Tag[1]+assert_that/Group/CoGroupByKeyImpl/Flatten/OutputIdentity+assert_that/Group/CoGroupByKeyImpl/GroupByKey/Write INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:22:38.232Z: JOB_MESSAGE_BASIC: Executing operation assert_that/Create/Impulse+assert_that/Create/FlatMap(<lambda at core.py:3481>)+assert_that/Create/Map(decode)+assert_that/Group/CoGroupByKeyImpl/Tag[0]+assert_that/Group/CoGroupByKeyImpl/Flatten/OutputIdentity+assert_that/Group/CoGroupByKeyImpl/GroupByKey/Write INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:22:38.285Z: JOB_MESSAGE_BASIC: Executing operation Count/CombineGlobally(CountCombineFn)/DoOnce/Impulse+Count/CombineGlobally(CountCombineFn)/DoOnce/FlatMap(<lambda at core.py:3481>)+Count/CombineGlobally(CountCombineFn)/DoOnce/Map(decode)+Count/CombineGlobally(CountCombineFn)/InjectDefault+Flatten/InputIdentity+assert_that/WindowInto(WindowIntoFn)+assert_that/ToVoidKey+assert_that/Group/CoGroupByKeyImpl/Tag[1]+assert_that/Group/CoGroupByKeyImpl/Flatten/OutputIdentity+assert_that/Group/CoGroupByKeyImpl/GroupByKey/Write INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:22:39.408Z: JOB_MESSAGE_BASIC: Finished operation assert_that/Create/Impulse+assert_that/Create/FlatMap(<lambda at core.py:3481>)+assert_that/Create/Map(decode)+assert_that/Group/CoGroupByKeyImpl/Tag[0]+assert_that/Group/CoGroupByKeyImpl/Flatten/OutputIdentity+assert_that/Group/CoGroupByKeyImpl/GroupByKey/Write INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:22:39.682Z: JOB_MESSAGE_BASIC: Finished operation Combine/DoOnce/Impulse+Combine/DoOnce/FlatMap(<lambda at core.py:3481>)+Combine/DoOnce/Map(decode)+Combine/InjectDefault+assert_that/WindowInto(WindowIntoFn)+assert_that/ToVoidKey+assert_that/Group/CoGroupByKeyImpl/Tag[1]+assert_that/Group/CoGroupByKeyImpl/Flatten/OutputIdentity+assert_that/Group/CoGroupByKeyImpl/GroupByKey/Write INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:22:40.707Z: JOB_MESSAGE_BASIC: Finished operation Count/CombineGlobally(CountCombineFn)/DoOnce/Impulse+Count/CombineGlobally(CountCombineFn)/DoOnce/FlatMap(<lambda at core.py:3481>)+Count/CombineGlobally(CountCombineFn)/DoOnce/Map(decode)+Count/CombineGlobally(CountCombineFn)/InjectDefault+Flatten/InputIdentity+assert_that/WindowInto(WindowIntoFn)+assert_that/ToVoidKey+assert_that/Group/CoGroupByKeyImpl/Tag[1]+assert_that/Group/CoGroupByKeyImpl/Flatten/OutputIdentity+assert_that/Group/CoGroupByKeyImpl/GroupByKey/Write INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:22:40.792Z: JOB_MESSAGE_BASIC: Executing operation assert_that/Group/CoGroupByKeyImpl/GroupByKey/Close INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:22:41.075Z: JOB_MESSAGE_BASIC: Finished operation assert_that/Group/CoGroupByKeyImpl/GroupByKey/Close INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:22:41.175Z: JOB_MESSAGE_BASIC: Executing operation assert_that/Group/CoGroupByKeyImpl/GroupByKey/Read+assert_that/Group/CoGroupByKeyImpl/MapTuple(collect_values)+assert_that/Group/RestoreTags+assert_that/Unkey+assert_that/Match INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:22:42.480Z: JOB_MESSAGE_BASIC: Finished operation assert_that/Group/CoGroupByKeyImpl/GroupByKey/Read+assert_that/Group/CoGroupByKeyImpl/MapTuple(collect_values)+assert_that/Group/RestoreTags+assert_that/Unkey+assert_that/Match INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:22:42.553Z: JOB_MESSAGE_DEBUG: Executing success step success71 INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:22:42.635Z: JOB_MESSAGE_DETAILED: Cleaning up. INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:22:42.690Z: JOB_MESSAGE_DEBUG: Starting worker pool teardown. INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:22:42.725Z: JOB_MESSAGE_BASIC: Stopping worker pool... INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:23:15.142Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized worker pool from 1 to 0. INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:23:15.194Z: JOB_MESSAGE_BASIC: Worker pool stopped. INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:23:15.228Z: JOB_MESSAGE_DEBUG: Tearing down pending resources... INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-08-26_14_15_18-13436952191853648136 is in state JOB_STATE_DONE INFO:__main__:Reading documents from mongodb finished in 488.775 seconds INFO:__main__:================================================================================ INFO:__main__:Reading from mongodb beam_mongodbio_it_db:integration_test_1661546722 INFO:__main__:reader params : {'filter': {'number_mod_3': 0}, 'projection': ['number'], 'bucket_auto': True} INFO:__main__:expected results: {'number_sum': 166666833333, 'docs_count': 333334} INFO:apache_beam.runners.portability.stager:Copying Beam SDK "/home/jenkins/jenkins-slave/workspace/beam_python_mongoio_load_test/src/sdks/python/build/apache-beam.tar.gz" to staging location. INFO:root:Default Python SDK image for environment is apache/beam_python3.7_sdk:2.42.0.dev INFO:root:Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python37:beam-master-20220811 INFO:root:Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python37:beam-master-20220811" for Docker environment INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function pack_combiners at 0x7f2e5493ccb0> ==================== INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function sort_stages at 0x7f2e5493e4d0> ==================== INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/beamapp-jenkins-0826212327-271810-crsl8yzt.1661549007.272503/dataflow_python_sdk.tar... INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/beamapp-jenkins-0826212327-271810-crsl8yzt.1661549007.272503/dataflow_python_sdk.tar in 0 seconds. INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/beamapp-jenkins-0826212327-271810-crsl8yzt.1661549007.272503/pipeline.pb... INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/beamapp-jenkins-0826212327-271810-crsl8yzt.1661549007.272503/pipeline.pb in 0 seconds. INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job clientRequestId: '20220826212327274048-5384' createTime: '2022-08-26T21:23:29.311224Z' currentStateTime: '1970-01-01T00:00:00Z' id: '2022-08-26_14_23_28-6044344456592736829' location: 'us-central1' name: 'beamapp-jenkins-0826212327-271810-crsl8yzt' projectId: 'apache-beam-testing' stageStates: [] startTime: '2022-08-26T21:23:29.311224Z' steps: [] tempFiles: [] type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)> INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2022-08-26_14_23_28-6044344456592736829] INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2022-08-26_14_23_28-6044344456592736829 INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2022-08-26_14_23_28-6044344456592736829?project=apache-beam-testing INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-08-26_14_23_28-6044344456592736829 is in state JOB_STATE_PENDING INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:23:28.921Z: JOB_MESSAGE_BASIC: Dataflow Runner V2 auto-enabled. Use --experiments=disable_runner_v2 to opt out. INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:23:29.933Z: JOB_MESSAGE_DETAILED: Autoscaling is enabled for job 2022-08-26_14_23_28-6044344456592736829. The number of workers will be between 1 and 1000. INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:23:30.097Z: JOB_MESSAGE_DETAILED: Autoscaling was automatically enabled for job 2022-08-26_14_23_28-6044344456592736829. INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:23:32.154Z: JOB_MESSAGE_BASIC: Worker configuration: e2-standard-2 in us-central1-a. INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:23:32.963Z: JOB_MESSAGE_DETAILED: Expanding SplittableParDo operations into optimizable parts. INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:23:33.002Z: JOB_MESSAGE_DETAILED: Expanding CollectionToSingleton operations into optimizable parts. INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:23:33.066Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts. INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:23:33.104Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step assert_that/Group/CoGroupByKeyImpl/GroupByKey: GroupByKey not followed by a combiner. INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:23:33.216Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts. INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:23:33.246Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information. INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:23:33.293Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:23:33.323Z: JOB_MESSAGE_DEBUG: Inserted coder converter before flatten ref_AppliedPTransform_Flatten_34 INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:23:33.414Z: JOB_MESSAGE_DEBUG: Inserted coder converter after flatten ref_AppliedPTransform_assert_that-Group-CoGroupByKeyImpl-Flatten_47 INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:23:33.449Z: JOB_MESSAGE_DETAILED: Unzipping flatten ref_AppliedPTransform_assert_that-Group-CoGroupByKeyImpl-Flatten_47 for input ref_AppliedPTransform_assert_that-Group-CoGroupByKeyImpl-Tag-0-_45.None INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:23:33.481Z: JOB_MESSAGE_DETAILED: Fusing unzipped copy of assert_that/Group/CoGroupByKeyImpl/Flatten/OutputIdentity, through flatten assert_that/Group/CoGroupByKeyImpl/Flatten, into producer assert_that/Group/CoGroupByKeyImpl/Tag[0] INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:23:33.507Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Group/CoGroupByKeyImpl/MapTuple(collect_values) into assert_that/Group/CoGroupByKeyImpl/GroupByKey/Read INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:23:33.544Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Group/RestoreTags into assert_that/Group/CoGroupByKeyImpl/MapTuple(collect_values) INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:23:33.579Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Unkey into assert_that/Group/RestoreTags INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:23:33.624Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Match into assert_that/Unkey INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:23:33.661Z: JOB_MESSAGE_DETAILED: Unzipping flatten ref_AppliedPTransform_assert_that-Group-CoGroupByKeyImpl-Flatten_47-u47 for input ref_AppliedPTransform_assert_that-Group-CoGroupByKeyImpl-Flatten_47.None-c45 INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:23:33.697Z: JOB_MESSAGE_DETAILED: Fusing unzipped copy of assert_that/Group/CoGroupByKeyImpl/GroupByKey/Write, through flatten assert_that/Group/CoGroupByKeyImpl/Flatten/Unzipped-1, into producer assert_that/Group/CoGroupByKeyImpl/Flatten/OutputIdentity INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:23:33.730Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Group/CoGroupByKeyImpl/Flatten/OutputIdentity into assert_that/Group/CoGroupByKeyImpl/Tag[1] INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:23:33.763Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Group/CoGroupByKeyImpl/GroupByKey/Write into assert_that/Group/CoGroupByKeyImpl/Flatten/OutputIdentity INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:23:33.796Z: JOB_MESSAGE_DETAILED: Unzipping flatten ref_AppliedPTransform_Flatten_34 for input ref_AppliedPTransform_Combine-InjectDefault_20.None INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:23:33.830Z: JOB_MESSAGE_DETAILED: Fusing unzipped copy of assert_that/WindowInto(WindowIntoFn), through flatten Flatten, into producer Combine/InjectDefault INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:23:33.863Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Group/CoGroupByKeyImpl/Tag[1] into assert_that/ToVoidKey INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:23:33.904Z: JOB_MESSAGE_DETAILED: Unzipping flatten ref_AppliedPTransform_Flatten_34-u54 for input ref_AppliedPTransform_assert_that-WindowInto-WindowIntoFn-_41.None-c52 INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:23:33.980Z: JOB_MESSAGE_DETAILED: Fusing unzipped copy of assert_that/ToVoidKey, through flatten Flatten/Unzipped-1, into producer assert_that/WindowInto(WindowIntoFn) INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:23:34.013Z: JOB_MESSAGE_DETAILED: Fusing consumer Combine/KeyWithVoid into Map INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:23:34.046Z: JOB_MESSAGE_DETAILED: Fusing consumer Count/CombineGlobally(CountCombineFn)/KeyWithVoid into Map INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:23:34.111Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/WindowInto(WindowIntoFn) into Flatten/InputIdentity INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:23:34.148Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/ToVoidKey into assert_that/WindowInto(WindowIntoFn) INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:23:34.183Z: JOB_MESSAGE_DETAILED: Fusing consumer ReadFromMongoDB/Read/Map(<lambda at iobase.py:908>) into ReadFromMongoDB/Read/Impulse INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:23:34.220Z: JOB_MESSAGE_DETAILED: Fusing consumer ref_AppliedPTransform_ReadFromMongoDB-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_7/PairWithRestriction into ReadFromMongoDB/Read/Map(<lambda at iobase.py:908>) INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:23:34.256Z: JOB_MESSAGE_DETAILED: Fusing consumer ref_AppliedPTransform_ReadFromMongoDB-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_7/SplitWithSizing into ref_AppliedPTransform_ReadFromMongoDB-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_7/PairWithRestriction INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:23:34.290Z: JOB_MESSAGE_DETAILED: Fusing consumer Map into ref_AppliedPTransform_ReadFromMongoDB-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_7/ProcessElementAndRestrictionWithSizing INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:23:34.323Z: JOB_MESSAGE_DETAILED: Fusing consumer Combine/CombinePerKey/GroupByKey+Combine/CombinePerKey/Combine/Partial into Combine/KeyWithVoid INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:23:34.412Z: JOB_MESSAGE_DETAILED: Fusing consumer Combine/CombinePerKey/GroupByKey/Write into Combine/CombinePerKey/GroupByKey+Combine/CombinePerKey/Combine/Partial INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:23:34.447Z: JOB_MESSAGE_DETAILED: Fusing consumer Combine/CombinePerKey/Combine into Combine/CombinePerKey/GroupByKey/Read INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:23:34.481Z: JOB_MESSAGE_DETAILED: Fusing consumer Combine/CombinePerKey/Combine/Extract into Combine/CombinePerKey/Combine INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:23:34.515Z: JOB_MESSAGE_DETAILED: Fusing consumer Combine/UnKey into Combine/CombinePerKey/Combine/Extract INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:23:34.547Z: JOB_MESSAGE_DETAILED: Fusing consumer Count/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey+Count/CombineGlobally(CountCombineFn)/CombinePerKey/Combine/Partial into Count/CombineGlobally(CountCombineFn)/KeyWithVoid INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:23:34.576Z: JOB_MESSAGE_DETAILED: Fusing consumer Count/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Write into Count/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey+Count/CombineGlobally(CountCombineFn)/CombinePerKey/Combine/Partial INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:23:34.624Z: JOB_MESSAGE_DETAILED: Fusing consumer Count/CombineGlobally(CountCombineFn)/CombinePerKey/Combine into Count/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Read INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:23:34.651Z: JOB_MESSAGE_DETAILED: Fusing consumer Count/CombineGlobally(CountCombineFn)/CombinePerKey/Combine/Extract into Count/CombineGlobally(CountCombineFn)/CombinePerKey/Combine INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:23:34.689Z: JOB_MESSAGE_DETAILED: Fusing consumer Count/CombineGlobally(CountCombineFn)/UnKey into Count/CombineGlobally(CountCombineFn)/CombinePerKey/Combine/Extract INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:23:34.720Z: JOB_MESSAGE_DETAILED: Fusing consumer Combine/DoOnce/FlatMap(<lambda at core.py:3481>) into Combine/DoOnce/Impulse INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:23:34.755Z: JOB_MESSAGE_DETAILED: Fusing consumer Combine/DoOnce/Map(decode) into Combine/DoOnce/FlatMap(<lambda at core.py:3481>) INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:23:34.792Z: JOB_MESSAGE_DETAILED: Fusing consumer Combine/InjectDefault into Combine/DoOnce/Map(decode) INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:23:34.824Z: JOB_MESSAGE_DETAILED: Fusing consumer Count/CombineGlobally(CountCombineFn)/DoOnce/FlatMap(<lambda at core.py:3481>) into Count/CombineGlobally(CountCombineFn)/DoOnce/Impulse INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:23:34.861Z: JOB_MESSAGE_DETAILED: Fusing consumer Count/CombineGlobally(CountCombineFn)/DoOnce/Map(decode) into Count/CombineGlobally(CountCombineFn)/DoOnce/FlatMap(<lambda at core.py:3481>) INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:23:34.884Z: JOB_MESSAGE_DETAILED: Fusing consumer Count/CombineGlobally(CountCombineFn)/InjectDefault into Count/CombineGlobally(CountCombineFn)/DoOnce/Map(decode) INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:23:34.916Z: JOB_MESSAGE_DETAILED: Fusing consumer Flatten/InputIdentity into Count/CombineGlobally(CountCombineFn)/InjectDefault INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:23:34.971Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Create/FlatMap(<lambda at core.py:3481>) into assert_that/Create/Impulse INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:23:35.023Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Create/Map(decode) into assert_that/Create/FlatMap(<lambda at core.py:3481>) INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:23:35.053Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Group/CoGroupByKeyImpl/Tag[0] into assert_that/Create/Map(decode) INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:23:35.114Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec. INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:23:35.150Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph. INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:23:35.172Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps. INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:23:35.211Z: JOB_MESSAGE_DEBUG: Assigning stage ids. INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:23:35.365Z: JOB_MESSAGE_DEBUG: Executing wait step start73 INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:23:35.461Z: JOB_MESSAGE_BASIC: Executing operation ReadFromMongoDB/Read/Impulse+ReadFromMongoDB/Read/Map(<lambda at iobase.py:908>)+ref_AppliedPTransform_ReadFromMongoDB-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_7/PairWithRestriction+ref_AppliedPTransform_ReadFromMongoDB-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_7/SplitWithSizing INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:23:35.514Z: JOB_MESSAGE_DEBUG: Starting worker pool setup. INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:23:35.558Z: JOB_MESSAGE_BASIC: Starting 5 workers in us-central1-a... INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-08-26_14_23_28-6044344456592736829 is in state JOB_STATE_RUNNING INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:24:02.473Z: JOB_MESSAGE_BASIC: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:24:10.495Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of workers to 5 based on the rate of progress in the currently running stage(s). INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:24:37.949Z: JOB_MESSAGE_DETAILED: Workers have started successfully. INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:27:07.624Z: JOB_MESSAGE_DETAILED: Autoscaling: Reduced the number of workers to 1 based on the rate of progress in the currently running stage(s). INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:27:07.669Z: JOB_MESSAGE_DETAILED: Autoscaling: Would further reduce the number of workers but reached the minimum number allowed for the job. INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:27:12.710Z: JOB_MESSAGE_DETAILED: Autoscaling: Resizing worker pool from 5 to 1. INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:30:04.086Z: JOB_MESSAGE_BASIC: Finished operation ReadFromMongoDB/Read/Impulse+ReadFromMongoDB/Read/Map(<lambda at iobase.py:908>)+ref_AppliedPTransform_ReadFromMongoDB-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_7/PairWithRestriction+ref_AppliedPTransform_ReadFromMongoDB-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_7/SplitWithSizing INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:30:04.147Z: JOB_MESSAGE_DEBUG: Value "ref_AppliedPTransform_ReadFromMongoDB-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_7-split-with-sizing-out3" materialized. INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:30:04.225Z: JOB_MESSAGE_BASIC: Executing operation Combine/CombinePerKey/GroupByKey/Create INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:30:04.268Z: JOB_MESSAGE_BASIC: Executing operation Count/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Create INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:30:05.689Z: JOB_MESSAGE_BASIC: Finished operation Combine/CombinePerKey/GroupByKey/Create INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:30:05.768Z: JOB_MESSAGE_DEBUG: Value "Combine/CombinePerKey/GroupByKey/Session" materialized. INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:30:06.896Z: JOB_MESSAGE_BASIC: Finished operation Count/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Create INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:30:07.016Z: JOB_MESSAGE_DEBUG: Value "Count/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Session" materialized. INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:30:07.079Z: JOB_MESSAGE_BASIC: Executing operation ref_AppliedPTransform_ReadFromMongoDB-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_7/ProcessElementAndRestrictionWithSizing+Map+Combine/KeyWithVoid+Count/CombineGlobally(CountCombineFn)/KeyWithVoid+Combine/CombinePerKey/GroupByKey+Combine/CombinePerKey/Combine/Partial+Combine/CombinePerKey/GroupByKey/Write+Count/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey+Count/CombineGlobally(CountCombineFn)/CombinePerKey/Combine/Partial+Count/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Write INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:30:22.977Z: JOB_MESSAGE_BASIC: Finished operation ref_AppliedPTransform_ReadFromMongoDB-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_7/ProcessElementAndRestrictionWithSizing+Map+Combine/KeyWithVoid+Count/CombineGlobally(CountCombineFn)/KeyWithVoid+Combine/CombinePerKey/GroupByKey+Combine/CombinePerKey/Combine/Partial+Combine/CombinePerKey/GroupByKey/Write+Count/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey+Count/CombineGlobally(CountCombineFn)/CombinePerKey/Combine/Partial+Count/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Write INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:30:23.068Z: JOB_MESSAGE_BASIC: Executing operation Combine/CombinePerKey/GroupByKey/Close INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:30:23.202Z: JOB_MESSAGE_BASIC: Executing operation Count/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Close INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:30:23.571Z: JOB_MESSAGE_BASIC: Finished operation Combine/CombinePerKey/GroupByKey/Close INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:30:23.678Z: JOB_MESSAGE_BASIC: Executing operation Combine/CombinePerKey/GroupByKey/Read+Combine/CombinePerKey/Combine+Combine/CombinePerKey/Combine/Extract+Combine/UnKey INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:30:23.988Z: JOB_MESSAGE_BASIC: Finished operation Count/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Close INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:30:24.068Z: JOB_MESSAGE_BASIC: Executing operation Count/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Read+Count/CombineGlobally(CountCombineFn)/CombinePerKey/Combine+Count/CombineGlobally(CountCombineFn)/CombinePerKey/Combine/Extract+Count/CombineGlobally(CountCombineFn)/UnKey INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:30:25.618Z: JOB_MESSAGE_BASIC: Finished operation Count/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Read+Count/CombineGlobally(CountCombineFn)/CombinePerKey/Combine+Count/CombineGlobally(CountCombineFn)/CombinePerKey/Combine/Extract+Count/CombineGlobally(CountCombineFn)/UnKey INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:30:25.765Z: JOB_MESSAGE_DEBUG: Value "Count/CombineGlobally(CountCombineFn)/UnKey.None" materialized. INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:30:25.829Z: JOB_MESSAGE_BASIC: Executing operation Count/CombineGlobally(CountCombineFn)/InjectDefault/View-python_side_input0-Count/CombineGlobally(CountCombineFn)/InjectDefault INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:30:25.849Z: JOB_MESSAGE_BASIC: Finished operation Combine/CombinePerKey/GroupByKey/Read+Combine/CombinePerKey/Combine+Combine/CombinePerKey/Combine/Extract+Combine/UnKey INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:30:25.892Z: JOB_MESSAGE_BASIC: Finished operation Count/CombineGlobally(CountCombineFn)/InjectDefault/View-python_side_input0-Count/CombineGlobally(CountCombineFn)/InjectDefault INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:30:25.924Z: JOB_MESSAGE_DEBUG: Value "Combine/UnKey.None" materialized. INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:30:25.966Z: JOB_MESSAGE_DEBUG: Value "Count/CombineGlobally(CountCombineFn)/InjectDefault/View-python_side_input0-Count/CombineGlobally(CountCombineFn)/InjectDefault.out" materialized. INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:30:26.031Z: JOB_MESSAGE_BASIC: Executing operation Combine/InjectDefault/View-python_side_input0-Combine/InjectDefault INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:30:26.096Z: JOB_MESSAGE_BASIC: Finished operation Combine/InjectDefault/View-python_side_input0-Combine/InjectDefault INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:30:26.239Z: JOB_MESSAGE_DEBUG: Value "Combine/InjectDefault/View-python_side_input0-Combine/InjectDefault.out" materialized. INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:30:26.309Z: JOB_MESSAGE_BASIC: Executing operation assert_that/Group/CoGroupByKeyImpl/GroupByKey/Create INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:30:26.476Z: JOB_MESSAGE_BASIC: Finished operation assert_that/Group/CoGroupByKeyImpl/GroupByKey/Create INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:30:26.545Z: JOB_MESSAGE_DEBUG: Value "assert_that/Group/CoGroupByKeyImpl/GroupByKey/Session" materialized. INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:30:26.614Z: JOB_MESSAGE_BASIC: Executing operation Combine/DoOnce/Impulse+Combine/DoOnce/FlatMap(<lambda at core.py:3481>)+Combine/DoOnce/Map(decode)+Combine/InjectDefault+assert_that/WindowInto(WindowIntoFn)+assert_that/ToVoidKey+assert_that/Group/CoGroupByKeyImpl/Tag[1]+assert_that/Group/CoGroupByKeyImpl/Flatten/OutputIdentity+assert_that/Group/CoGroupByKeyImpl/GroupByKey/Write INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:30:26.681Z: JOB_MESSAGE_BASIC: Executing operation assert_that/Create/Impulse+assert_that/Create/FlatMap(<lambda at core.py:3481>)+assert_that/Create/Map(decode)+assert_that/Group/CoGroupByKeyImpl/Tag[0]+assert_that/Group/CoGroupByKeyImpl/Flatten/OutputIdentity+assert_that/Group/CoGroupByKeyImpl/GroupByKey/Write INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:30:26.779Z: JOB_MESSAGE_BASIC: Executing operation Count/CombineGlobally(CountCombineFn)/DoOnce/Impulse+Count/CombineGlobally(CountCombineFn)/DoOnce/FlatMap(<lambda at core.py:3481>)+Count/CombineGlobally(CountCombineFn)/DoOnce/Map(decode)+Count/CombineGlobally(CountCombineFn)/InjectDefault+Flatten/InputIdentity+assert_that/WindowInto(WindowIntoFn)+assert_that/ToVoidKey+assert_that/Group/CoGroupByKeyImpl/Tag[1]+assert_that/Group/CoGroupByKeyImpl/Flatten/OutputIdentity+assert_that/Group/CoGroupByKeyImpl/GroupByKey/Write INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:30:26.897Z: JOB_MESSAGE_BASIC: Finished operation assert_that/Create/Impulse+assert_that/Create/FlatMap(<lambda at core.py:3481>)+assert_that/Create/Map(decode)+assert_that/Group/CoGroupByKeyImpl/Tag[0]+assert_that/Group/CoGroupByKeyImpl/Flatten/OutputIdentity+assert_that/Group/CoGroupByKeyImpl/GroupByKey/Write INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:30:27.171Z: JOB_MESSAGE_BASIC: Finished operation Combine/DoOnce/Impulse+Combine/DoOnce/FlatMap(<lambda at core.py:3481>)+Combine/DoOnce/Map(decode)+Combine/InjectDefault+assert_that/WindowInto(WindowIntoFn)+assert_that/ToVoidKey+assert_that/Group/CoGroupByKeyImpl/Tag[1]+assert_that/Group/CoGroupByKeyImpl/Flatten/OutputIdentity+assert_that/Group/CoGroupByKeyImpl/GroupByKey/Write INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:30:29.168Z: JOB_MESSAGE_BASIC: Finished operation Count/CombineGlobally(CountCombineFn)/DoOnce/Impulse+Count/CombineGlobally(CountCombineFn)/DoOnce/FlatMap(<lambda at core.py:3481>)+Count/CombineGlobally(CountCombineFn)/DoOnce/Map(decode)+Count/CombineGlobally(CountCombineFn)/InjectDefault+Flatten/InputIdentity+assert_that/WindowInto(WindowIntoFn)+assert_that/ToVoidKey+assert_that/Group/CoGroupByKeyImpl/Tag[1]+assert_that/Group/CoGroupByKeyImpl/Flatten/OutputIdentity+assert_that/Group/CoGroupByKeyImpl/GroupByKey/Write INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:30:29.246Z: JOB_MESSAGE_BASIC: Executing operation assert_that/Group/CoGroupByKeyImpl/GroupByKey/Close INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:30:29.307Z: JOB_MESSAGE_BASIC: Finished operation assert_that/Group/CoGroupByKeyImpl/GroupByKey/Close INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:30:29.382Z: JOB_MESSAGE_BASIC: Executing operation assert_that/Group/CoGroupByKeyImpl/GroupByKey/Read+assert_that/Group/CoGroupByKeyImpl/MapTuple(collect_values)+assert_that/Group/RestoreTags+assert_that/Unkey+assert_that/Match INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:30:30.663Z: JOB_MESSAGE_BASIC: Finished operation assert_that/Group/CoGroupByKeyImpl/GroupByKey/Read+assert_that/Group/CoGroupByKeyImpl/MapTuple(collect_values)+assert_that/Group/RestoreTags+assert_that/Unkey+assert_that/Match INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:30:30.763Z: JOB_MESSAGE_DEBUG: Executing success step success71 INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:30:30.839Z: JOB_MESSAGE_DETAILED: Cleaning up. INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:30:30.891Z: JOB_MESSAGE_DEBUG: Starting worker pool teardown. INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:30:30.923Z: JOB_MESSAGE_BASIC: Stopping worker pool... INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:32:39.607Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized worker pool from 1 to 0. INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:32:39.698Z: JOB_MESSAGE_BASIC: Worker pool stopped. INFO:apache_beam.runners.dataflow.dataflow_runner:2022-08-26T21:32:39.742Z: JOB_MESSAGE_DEBUG: Tearing down pending resources... FATAL: command execution failed hudson.remoting.ChannelClosedException: Channel "hudson.remoting.Channel@643e8700:apache-beam-jenkins-14": Remote call on apache-beam-jenkins-14 failed. The channel is closing down or has closed down at hudson.remoting.Channel.call(Channel.java:993) at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:285) at com.sun.proxy.$Proxy138.isAlive(Unknown Source) at hudson.Launcher$RemoteLauncher$ProcImpl.isAlive(Launcher.java:1215) at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:1207) at hudson.Launcher$ProcStarter.join(Launcher.java:524) at hudson.plugins.gradle.Gradle.perform(Gradle.java:317) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:814) at hudson.model.Build$BuildExecution.build(Build.java:199) at hudson.model.Build$BuildExecution.doRun(Build.java:164) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:522) at hudson.model.Run.execute(Run.java:1896) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44) at hudson.model.ResourceController.execute(ResourceController.java:101) at hudson.model.Executor.run(Executor.java:442) Caused by: hudson.remoting.Channel$OrderlyShutdown: Command Close created at at hudson.remoting.Channel$CloseCommand.execute(Channel.java:1313) at hudson.remoting.Channel$1.handle(Channel.java:606) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:81) Caused by: Command Close created at at hudson.remoting.Command.<init>(Command.java:70) at hudson.remoting.Channel$CloseCommand.<init>(Channel.java:1306) at hudson.remoting.Channel$CloseCommand.<init>(Channel.java:1304) at hudson.remoting.Channel.close(Channel.java:1480) at hudson.remoting.Channel.close(Channel.java:1447) at hudson.remoting.Channel$CloseCommand.execute(Channel.java:1312) ... 2 more Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure ERROR: apache-beam-jenkins-14 is offline; cannot locate jdk_1.8_latest --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
