See <https://ci-beam.apache.org/job/beam_python_mongoio_load_test/4607/display/redirect?page=changes>
Changes: [noreply] Bump github.com/testcontainers/testcontainers-go in /sdks (#25517) ------------------------------------------ [...truncated 91.05 KB...] INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:42:58.454Z: JOB_MESSAGE_DETAILED: Expanding SplittableParDo operations into optimizable parts. INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:42:58.492Z: JOB_MESSAGE_DETAILED: Expanding CollectionToSingleton operations into optimizable parts. INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:42:58.552Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts. INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:42:58.585Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step assert_that/Group/CoGroupByKeyImpl/GroupByKey: GroupByKey not followed by a combiner. INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:42:58.677Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts. INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:42:58.707Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information. INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:42:58.751Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:42:58.787Z: JOB_MESSAGE_DEBUG: Inserted coder converter before flatten ref_AppliedPTransform_Flatten_36 INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:42:58.814Z: JOB_MESSAGE_DEBUG: Inserted coder converter after flatten ref_AppliedPTransform_assert_that-Group-CoGroupByKeyImpl-Flatten_49 INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:42:58.839Z: JOB_MESSAGE_DETAILED: Unzipping flatten ref_AppliedPTransform_assert_that-Group-CoGroupByKeyImpl-Flatten_49 for input ref_AppliedPTransform_assert_that-Group-CoGroupByKeyImpl-Tag-0-_47.None INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:42:58.873Z: JOB_MESSAGE_DETAILED: Fusing unzipped copy of assert_that/Group/CoGroupByKeyImpl/Flatten/OutputIdentity, through flatten assert_that/Group/CoGroupByKeyImpl/Flatten, into producer assert_that/Group/CoGroupByKeyImpl/Tag[0] INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:42:58.903Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Group/CoGroupByKeyImpl/MapTuple(collect_values) into assert_that/Group/CoGroupByKeyImpl/GroupByKey/Read INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:42:58.940Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Group/RestoreTags into assert_that/Group/CoGroupByKeyImpl/MapTuple(collect_values) INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:42:58.978Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Unkey into assert_that/Group/RestoreTags INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:42:59.006Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Match into assert_that/Unkey INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:42:59.041Z: JOB_MESSAGE_DETAILED: Unzipping flatten ref_AppliedPTransform_assert_that-Group-CoGroupByKeyImpl-Flatten_49-u47 for input ref_AppliedPTransform_assert_that-Group-CoGroupByKeyImpl-Flatten_49.None-c45 INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:42:59.074Z: JOB_MESSAGE_DETAILED: Fusing unzipped copy of assert_that/Group/CoGroupByKeyImpl/GroupByKey/Write, through flatten assert_that/Group/CoGroupByKeyImpl/Flatten/Unzipped-1, into producer assert_that/Group/CoGroupByKeyImpl/Flatten/OutputIdentity INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:42:59.105Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Group/CoGroupByKeyImpl/Flatten/OutputIdentity into assert_that/Group/CoGroupByKeyImpl/Tag[1] INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:42:59.135Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Group/CoGroupByKeyImpl/GroupByKey/Write into assert_that/Group/CoGroupByKeyImpl/Flatten/OutputIdentity INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:42:59.160Z: JOB_MESSAGE_DETAILED: Unzipping flatten ref_AppliedPTransform_Flatten_36 for input ref_AppliedPTransform_Combine-InjectDefault_21.None INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:42:59.184Z: JOB_MESSAGE_DETAILED: Fusing unzipped copy of assert_that/WindowInto(WindowIntoFn), through flatten Flatten, into producer Combine/InjectDefault INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:42:59.210Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Group/CoGroupByKeyImpl/Tag[1] into assert_that/ToVoidKey INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:42:59.244Z: JOB_MESSAGE_DETAILED: Unzipping flatten ref_AppliedPTransform_Flatten_36-u54 for input ref_AppliedPTransform_assert_that-WindowInto-WindowIntoFn-_43.None-c52 INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:42:59.273Z: JOB_MESSAGE_DETAILED: Fusing unzipped copy of assert_that/ToVoidKey, through flatten Flatten/Unzipped-1, into producer assert_that/WindowInto(WindowIntoFn) INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:42:59.304Z: JOB_MESSAGE_DETAILED: Fusing consumer Combine/KeyWithVoid into Map INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:42:59.324Z: JOB_MESSAGE_DETAILED: Fusing consumer Count/CombineGlobally(CountCombineFn)/KeyWithVoid into Map INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:42:59.349Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/WindowInto(WindowIntoFn) into Flatten/InputIdentity INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:42:59.377Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/ToVoidKey into assert_that/WindowInto(WindowIntoFn) INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:42:59.406Z: JOB_MESSAGE_DETAILED: Fusing consumer ReadFromMongoDB/Read/Map(<lambda at iobase.py:908>) into ReadFromMongoDB/Read/Impulse INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:42:59.424Z: JOB_MESSAGE_DETAILED: Fusing consumer ref_AppliedPTransform_ReadFromMongoDB-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_7/PairWithRestriction into ReadFromMongoDB/Read/Map(<lambda at iobase.py:908>) INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:42:59.448Z: JOB_MESSAGE_DETAILED: Fusing consumer ref_AppliedPTransform_ReadFromMongoDB-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_7/SplitWithSizing into ref_AppliedPTransform_ReadFromMongoDB-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_7/PairWithRestriction INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:42:59.467Z: JOB_MESSAGE_DETAILED: Fusing consumer Map into ref_AppliedPTransform_ReadFromMongoDB-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_7/ProcessElementAndRestrictionWithSizing INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:42:59.493Z: JOB_MESSAGE_DETAILED: Fusing consumer Combine/CombinePerKey/GroupByKey+Combine/CombinePerKey/Combine/Partial into Combine/KeyWithVoid INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:42:59.514Z: JOB_MESSAGE_DETAILED: Fusing consumer Combine/CombinePerKey/GroupByKey/Write into Combine/CombinePerKey/GroupByKey+Combine/CombinePerKey/Combine/Partial INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:42:59.542Z: JOB_MESSAGE_DETAILED: Fusing consumer Combine/CombinePerKey/Combine into Combine/CombinePerKey/GroupByKey/Read INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:42:59.570Z: JOB_MESSAGE_DETAILED: Fusing consumer Combine/CombinePerKey/Combine/Extract into Combine/CombinePerKey/Combine INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:42:59.605Z: JOB_MESSAGE_DETAILED: Fusing consumer Combine/UnKey into Combine/CombinePerKey/Combine/Extract INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:42:59.627Z: JOB_MESSAGE_DETAILED: Fusing consumer Count/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey+Count/CombineGlobally(CountCombineFn)/CombinePerKey/Combine/Partial into Count/CombineGlobally(CountCombineFn)/KeyWithVoid INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:42:59.655Z: JOB_MESSAGE_DETAILED: Fusing consumer Count/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Write into Count/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey+Count/CombineGlobally(CountCombineFn)/CombinePerKey/Combine/Partial INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:42:59.692Z: JOB_MESSAGE_DETAILED: Fusing consumer Count/CombineGlobally(CountCombineFn)/CombinePerKey/Combine into Count/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Read INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:42:59.739Z: JOB_MESSAGE_DETAILED: Fusing consumer Count/CombineGlobally(CountCombineFn)/CombinePerKey/Combine/Extract into Count/CombineGlobally(CountCombineFn)/CombinePerKey/Combine INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:42:59.767Z: JOB_MESSAGE_DETAILED: Fusing consumer Count/CombineGlobally(CountCombineFn)/UnKey into Count/CombineGlobally(CountCombineFn)/CombinePerKey/Combine/Extract INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:42:59.799Z: JOB_MESSAGE_DETAILED: Fusing consumer Combine/DoOnce/FlatMap(<lambda at core.py:3507>) into Combine/DoOnce/Impulse INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:42:59.828Z: JOB_MESSAGE_DETAILED: Fusing consumer Combine/DoOnce/Map(decode) into Combine/DoOnce/FlatMap(<lambda at core.py:3507>) INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:42:59.857Z: JOB_MESSAGE_DETAILED: Fusing consumer Combine/InjectDefault into Combine/DoOnce/Map(decode) INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:42:59.888Z: JOB_MESSAGE_DETAILED: Fusing consumer Count/CombineGlobally(CountCombineFn)/DoOnce/FlatMap(<lambda at core.py:3507>) into Count/CombineGlobally(CountCombineFn)/DoOnce/Impulse INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:42:59.923Z: JOB_MESSAGE_DETAILED: Fusing consumer Count/CombineGlobally(CountCombineFn)/DoOnce/Map(decode) into Count/CombineGlobally(CountCombineFn)/DoOnce/FlatMap(<lambda at core.py:3507>) INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:42:59.947Z: JOB_MESSAGE_DETAILED: Fusing consumer Count/CombineGlobally(CountCombineFn)/InjectDefault into Count/CombineGlobally(CountCombineFn)/DoOnce/Map(decode) INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:42:59.971Z: JOB_MESSAGE_DETAILED: Fusing consumer Flatten/InputIdentity into Count/CombineGlobally(CountCombineFn)/InjectDefault INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:43:00.003Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Create/FlatMap(<lambda at core.py:3507>) into assert_that/Create/Impulse INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:43:00.031Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Create/Map(decode) into assert_that/Create/FlatMap(<lambda at core.py:3507>) INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:43:00.055Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Group/CoGroupByKeyImpl/Tag[0] into assert_that/Create/Map(decode) INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:43:00.107Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec. INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:43:00.132Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph. INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:43:00.166Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps. INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:43:00.190Z: JOB_MESSAGE_DEBUG: Assigning stage ids. INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:43:00.332Z: JOB_MESSAGE_DEBUG: Executing wait step start73 INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:43:00.397Z: JOB_MESSAGE_BASIC: Executing operation ReadFromMongoDB/Read/Impulse+ReadFromMongoDB/Read/Map(<lambda at iobase.py:908>)+ref_AppliedPTransform_ReadFromMongoDB-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_7/PairWithRestriction+ref_AppliedPTransform_ReadFromMongoDB-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_7/SplitWithSizing INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:43:00.447Z: JOB_MESSAGE_DEBUG: Starting worker pool setup. INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:43:00.471Z: JOB_MESSAGE_BASIC: Starting 5 workers in us-central1-b... INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2023-02-22_07_42_53-6683275953297707361 is in state JOB_STATE_RUNNING INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:43:17.662Z: JOB_MESSAGE_BASIC: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:43:40.090Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of workers to 5 based on the rate of progress in the currently running stage(s). INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:44:07.068Z: JOB_MESSAGE_DETAILED: Workers have started successfully. INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:46:31.441Z: JOB_MESSAGE_DETAILED: Autoscaling: Reduced the number of workers to 1 based on the rate of progress in the currently running stage(s). INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:46:31.469Z: JOB_MESSAGE_DETAILED: Autoscaling: Would further reduce the number of workers but reached the minimum number allowed for the job. INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:46:36.501Z: JOB_MESSAGE_DETAILED: Autoscaling: Resizing worker pool from 5 to 1. INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:49:10.288Z: JOB_MESSAGE_DETAILED: All workers have finished the startup processes and began to receive work requests. INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:49:12.702Z: JOB_MESSAGE_BASIC: Finished operation ReadFromMongoDB/Read/Impulse+ReadFromMongoDB/Read/Map(<lambda at iobase.py:908>)+ref_AppliedPTransform_ReadFromMongoDB-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_7/PairWithRestriction+ref_AppliedPTransform_ReadFromMongoDB-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_7/SplitWithSizing INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:49:13.326Z: JOB_MESSAGE_DEBUG: Value "ref_AppliedPTransform_ReadFromMongoDB-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_7-split-with-sizing-out3" materialized. INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:49:13.390Z: JOB_MESSAGE_BASIC: Executing operation Combine/CombinePerKey/GroupByKey/Create INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:49:13.413Z: JOB_MESSAGE_BASIC: Executing operation Count/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Create INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:49:14.796Z: JOB_MESSAGE_BASIC: Finished operation Combine/CombinePerKey/GroupByKey/Create INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:49:14.863Z: JOB_MESSAGE_DEBUG: Value "Combine/CombinePerKey/GroupByKey/Session" materialized. INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:49:15.612Z: JOB_MESSAGE_BASIC: Finished operation Count/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Create INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:49:15.671Z: JOB_MESSAGE_DEBUG: Value "Count/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Session" materialized. INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:49:15.758Z: JOB_MESSAGE_BASIC: Executing operation ref_AppliedPTransform_ReadFromMongoDB-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_7/ProcessElementAndRestrictionWithSizing+Map+Combine/KeyWithVoid+Count/CombineGlobally(CountCombineFn)/KeyWithVoid+Combine/CombinePerKey/GroupByKey+Combine/CombinePerKey/Combine/Partial+Combine/CombinePerKey/GroupByKey/Write+Count/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey+Count/CombineGlobally(CountCombineFn)/CombinePerKey/Combine/Partial+Count/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Write INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:49:32.401Z: JOB_MESSAGE_BASIC: Finished operation ref_AppliedPTransform_ReadFromMongoDB-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_7/ProcessElementAndRestrictionWithSizing+Map+Combine/KeyWithVoid+Count/CombineGlobally(CountCombineFn)/KeyWithVoid+Combine/CombinePerKey/GroupByKey+Combine/CombinePerKey/Combine/Partial+Combine/CombinePerKey/GroupByKey/Write+Count/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey+Count/CombineGlobally(CountCombineFn)/CombinePerKey/Combine/Partial+Count/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Write INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:49:32.449Z: JOB_MESSAGE_BASIC: Executing operation Combine/CombinePerKey/GroupByKey/Close INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:49:32.472Z: JOB_MESSAGE_BASIC: Executing operation Count/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Close INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:49:32.503Z: JOB_MESSAGE_BASIC: Finished operation Combine/CombinePerKey/GroupByKey/Close INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:49:32.521Z: JOB_MESSAGE_BASIC: Finished operation Count/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Close INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:49:32.556Z: JOB_MESSAGE_BASIC: Executing operation Combine/CombinePerKey/GroupByKey/Read+Combine/CombinePerKey/Combine+Combine/CombinePerKey/Combine/Extract+Combine/UnKey INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:49:32.578Z: JOB_MESSAGE_BASIC: Executing operation Count/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Read+Count/CombineGlobally(CountCombineFn)/CombinePerKey/Combine+Count/CombineGlobally(CountCombineFn)/CombinePerKey/Combine/Extract+Count/CombineGlobally(CountCombineFn)/UnKey INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:49:34.703Z: JOB_MESSAGE_BASIC: Finished operation Combine/CombinePerKey/GroupByKey/Read+Combine/CombinePerKey/Combine+Combine/CombinePerKey/Combine/Extract+Combine/UnKey INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:49:34.774Z: JOB_MESSAGE_DEBUG: Value "Combine/UnKey.None" materialized. INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:49:34.832Z: JOB_MESSAGE_BASIC: Executing operation Combine/InjectDefault/View-python_side_input0-Combine/InjectDefault INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:49:34.894Z: JOB_MESSAGE_BASIC: Finished operation Combine/InjectDefault/View-python_side_input0-Combine/InjectDefault INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:49:34.961Z: JOB_MESSAGE_DEBUG: Value "Combine/InjectDefault/View-python_side_input0-Combine/InjectDefault.out" materialized. INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:49:36.128Z: JOB_MESSAGE_BASIC: Finished operation Count/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Read+Count/CombineGlobally(CountCombineFn)/CombinePerKey/Combine+Count/CombineGlobally(CountCombineFn)/CombinePerKey/Combine/Extract+Count/CombineGlobally(CountCombineFn)/UnKey INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:49:36.198Z: JOB_MESSAGE_DEBUG: Value "Count/CombineGlobally(CountCombineFn)/UnKey.None" materialized. INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:49:36.252Z: JOB_MESSAGE_BASIC: Executing operation Count/CombineGlobally(CountCombineFn)/InjectDefault/View-python_side_input0-Count/CombineGlobally(CountCombineFn)/InjectDefault INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:49:36.318Z: JOB_MESSAGE_BASIC: Finished operation Count/CombineGlobally(CountCombineFn)/InjectDefault/View-python_side_input0-Count/CombineGlobally(CountCombineFn)/InjectDefault INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:49:36.373Z: JOB_MESSAGE_DEBUG: Value "Count/CombineGlobally(CountCombineFn)/InjectDefault/View-python_side_input0-Count/CombineGlobally(CountCombineFn)/InjectDefault.out" materialized. INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:49:36.434Z: JOB_MESSAGE_BASIC: Executing operation assert_that/Group/CoGroupByKeyImpl/GroupByKey/Create INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:49:36.584Z: JOB_MESSAGE_BASIC: Finished operation assert_that/Group/CoGroupByKeyImpl/GroupByKey/Create INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:49:36.636Z: JOB_MESSAGE_DEBUG: Value "assert_that/Group/CoGroupByKeyImpl/GroupByKey/Session" materialized. INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:49:36.682Z: JOB_MESSAGE_BASIC: Executing operation Combine/DoOnce/Impulse+Combine/DoOnce/FlatMap(<lambda at core.py:3507>)+Combine/DoOnce/Map(decode)+Combine/InjectDefault+assert_that/WindowInto(WindowIntoFn)+assert_that/ToVoidKey+assert_that/Group/CoGroupByKeyImpl/Tag[1]+assert_that/Group/CoGroupByKeyImpl/Flatten/OutputIdentity+assert_that/Group/CoGroupByKeyImpl/GroupByKey/Write INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:49:36.714Z: JOB_MESSAGE_BASIC: Executing operation assert_that/Create/Impulse+assert_that/Create/FlatMap(<lambda at core.py:3507>)+assert_that/Create/Map(decode)+assert_that/Group/CoGroupByKeyImpl/Tag[0]+assert_that/Group/CoGroupByKeyImpl/Flatten/OutputIdentity+assert_that/Group/CoGroupByKeyImpl/GroupByKey/Write INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:49:36.736Z: JOB_MESSAGE_BASIC: Executing operation Count/CombineGlobally(CountCombineFn)/DoOnce/Impulse+Count/CombineGlobally(CountCombineFn)/DoOnce/FlatMap(<lambda at core.py:3507>)+Count/CombineGlobally(CountCombineFn)/DoOnce/Map(decode)+Count/CombineGlobally(CountCombineFn)/InjectDefault+Flatten/InputIdentity+assert_that/WindowInto(WindowIntoFn)+assert_that/ToVoidKey+assert_that/Group/CoGroupByKeyImpl/Tag[1]+assert_that/Group/CoGroupByKeyImpl/Flatten/OutputIdentity+assert_that/Group/CoGroupByKeyImpl/GroupByKey/Write INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:49:37.855Z: JOB_MESSAGE_BASIC: Finished operation assert_that/Create/Impulse+assert_that/Create/FlatMap(<lambda at core.py:3507>)+assert_that/Create/Map(decode)+assert_that/Group/CoGroupByKeyImpl/Tag[0]+assert_that/Group/CoGroupByKeyImpl/Flatten/OutputIdentity+assert_that/Group/CoGroupByKeyImpl/GroupByKey/Write INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:49:38.963Z: JOB_MESSAGE_BASIC: Finished operation Count/CombineGlobally(CountCombineFn)/DoOnce/Impulse+Count/CombineGlobally(CountCombineFn)/DoOnce/FlatMap(<lambda at core.py:3507>)+Count/CombineGlobally(CountCombineFn)/DoOnce/Map(decode)+Count/CombineGlobally(CountCombineFn)/InjectDefault+Flatten/InputIdentity+assert_that/WindowInto(WindowIntoFn)+assert_that/ToVoidKey+assert_that/Group/CoGroupByKeyImpl/Tag[1]+assert_that/Group/CoGroupByKeyImpl/Flatten/OutputIdentity+assert_that/Group/CoGroupByKeyImpl/GroupByKey/Write INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:49:41.101Z: JOB_MESSAGE_BASIC: Finished operation Combine/DoOnce/Impulse+Combine/DoOnce/FlatMap(<lambda at core.py:3507>)+Combine/DoOnce/Map(decode)+Combine/InjectDefault+assert_that/WindowInto(WindowIntoFn)+assert_that/ToVoidKey+assert_that/Group/CoGroupByKeyImpl/Tag[1]+assert_that/Group/CoGroupByKeyImpl/Flatten/OutputIdentity+assert_that/Group/CoGroupByKeyImpl/GroupByKey/Write INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:49:41.153Z: JOB_MESSAGE_BASIC: Executing operation assert_that/Group/CoGroupByKeyImpl/GroupByKey/Close INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:49:41.202Z: JOB_MESSAGE_BASIC: Finished operation assert_that/Group/CoGroupByKeyImpl/GroupByKey/Close INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:49:41.268Z: JOB_MESSAGE_BASIC: Executing operation assert_that/Group/CoGroupByKeyImpl/GroupByKey/Read+assert_that/Group/CoGroupByKeyImpl/MapTuple(collect_values)+assert_that/Group/RestoreTags+assert_that/Unkey+assert_that/Match INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:49:42.459Z: JOB_MESSAGE_BASIC: Finished operation assert_that/Group/CoGroupByKeyImpl/GroupByKey/Read+assert_that/Group/CoGroupByKeyImpl/MapTuple(collect_values)+assert_that/Group/RestoreTags+assert_that/Unkey+assert_that/Match INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:49:42.524Z: JOB_MESSAGE_DEBUG: Executing success step success71 INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:49:42.579Z: JOB_MESSAGE_DETAILED: Cleaning up. INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:49:42.634Z: JOB_MESSAGE_DEBUG: Starting worker pool teardown. INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:49:42.662Z: JOB_MESSAGE_BASIC: Stopping worker pool... INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:51:53.415Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized worker pool from 1 to 0. INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:51:53.444Z: JOB_MESSAGE_BASIC: Worker pool stopped. INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:51:53.468Z: JOB_MESSAGE_DEBUG: Tearing down pending resources... INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2023-02-22_07_42_53-6683275953297707361 is in state JOB_STATE_DONE INFO:__main__:Reading documents from mongodb finished in 553.171 seconds INFO:__main__:================================================================================ INFO:__main__:Reading from mongodb beam_mongodbio_it_db:integration_test_1677079235 INFO:__main__:reader params : {'projection': ['number'], 'bucket_auto': True} INFO:__main__:expected results: {'number_sum': 499999500000, 'docs_count': 1000000} INFO:apache_beam.runners.portability.stager:Copying Beam SDK "/home/jenkins/jenkins-slave/workspace/beam_python_mongoio_load_test/src/sdks/python/build/apache-beam.tar.gz" to staging location. INFO:root:Default Python SDK image for environment is apache/beam_python3.7_sdk:2.46.0.dev INFO:root:Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python37:beam-master-20230126 INFO:root:Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python37:beam-master-20230126" for Docker environment INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function pack_combiners at 0x7f4c447757a0> ==================== INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function sort_stages at 0x7f4c44775f80> ==================== INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/beamapp-jenkins-0222155205-900988-9cn6fyd0.1677081125.901310/dataflow_python_sdk.tar... INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/beamapp-jenkins-0222155205-900988-9cn6fyd0.1677081125.901310/dataflow_python_sdk.tar in 0 seconds. INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/beamapp-jenkins-0222155205-900988-9cn6fyd0.1677081125.901310/pipeline.pb... INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/beamapp-jenkins-0222155205-900988-9cn6fyd0.1677081125.901310/pipeline.pb in 0 seconds. INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job clientRequestId: '20230222155205902125-9121' createTime: '2023-02-22T15:52:10.135061Z' currentStateTime: '1970-01-01T00:00:00Z' id: '2023-02-22_07_52_06-2712740237206067' location: 'us-central1' name: 'beamapp-jenkins-0222155205-900988-9cn6fyd0' projectId: 'apache-beam-testing' stageStates: [] startTime: '2023-02-22T15:52:10.135061Z' steps: [] tempFiles: [] type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)> INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2023-02-22_07_52_06-2712740237206067] INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2023-02-22_07_52_06-2712740237206067 INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2023-02-22_07_52_06-2712740237206067?project=apache-beam-testing INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2023-02-22_07_52_06-2712740237206067 is in state JOB_STATE_PENDING INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:52:07.030Z: JOB_MESSAGE_BASIC: Dataflow Runner V2 auto-enabled. Use --experiments=disable_runner_v2 to opt out. INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:52:11.629Z: JOB_MESSAGE_DETAILED: Autoscaling is enabled for job 2023-02-22_07_52_06-2712740237206067. The number of workers will be between 1 and 1000. INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:52:11.708Z: JOB_MESSAGE_DETAILED: Autoscaling was automatically enabled for job 2023-02-22_07_52_06-2712740237206067. INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:52:13.715Z: JOB_MESSAGE_BASIC: Worker configuration: e2-standard-2 in us-central1-b. INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:52:14.612Z: JOB_MESSAGE_DETAILED: Expanding SplittableParDo operations into optimizable parts. INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:52:14.646Z: JOB_MESSAGE_DETAILED: Expanding CollectionToSingleton operations into optimizable parts. INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:52:14.704Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts. INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:52:14.735Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step assert_that/Group/CoGroupByKeyImpl/GroupByKey: GroupByKey not followed by a combiner. INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:52:14.831Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts. INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:52:14.862Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information. INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:52:14.897Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:52:14.919Z: JOB_MESSAGE_DEBUG: Inserted coder converter before flatten ref_AppliedPTransform_Flatten_36 INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:52:14.953Z: JOB_MESSAGE_DEBUG: Inserted coder converter after flatten ref_AppliedPTransform_assert_that-Group-CoGroupByKeyImpl-Flatten_49 INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:52:14.978Z: JOB_MESSAGE_DETAILED: Unzipping flatten ref_AppliedPTransform_assert_that-Group-CoGroupByKeyImpl-Flatten_49 for input ref_AppliedPTransform_assert_that-Group-CoGroupByKeyImpl-Tag-0-_47.None INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:52:15.008Z: JOB_MESSAGE_DETAILED: Fusing unzipped copy of assert_that/Group/CoGroupByKeyImpl/Flatten/OutputIdentity, through flatten assert_that/Group/CoGroupByKeyImpl/Flatten, into producer assert_that/Group/CoGroupByKeyImpl/Tag[0] INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:52:15.041Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Group/CoGroupByKeyImpl/MapTuple(collect_values) into assert_that/Group/CoGroupByKeyImpl/GroupByKey/Read INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:52:15.068Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Group/RestoreTags into assert_that/Group/CoGroupByKeyImpl/MapTuple(collect_values) INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:52:15.096Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Unkey into assert_that/Group/RestoreTags INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:52:15.121Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Match into assert_that/Unkey INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:52:15.154Z: JOB_MESSAGE_DETAILED: Unzipping flatten ref_AppliedPTransform_assert_that-Group-CoGroupByKeyImpl-Flatten_49-u47 for input ref_AppliedPTransform_assert_that-Group-CoGroupByKeyImpl-Flatten_49.None-c45 INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:52:15.183Z: JOB_MESSAGE_DETAILED: Fusing unzipped copy of assert_that/Group/CoGroupByKeyImpl/GroupByKey/Write, through flatten assert_that/Group/CoGroupByKeyImpl/Flatten/Unzipped-1, into producer assert_that/Group/CoGroupByKeyImpl/Flatten/OutputIdentity INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:52:15.216Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Group/CoGroupByKeyImpl/Flatten/OutputIdentity into assert_that/Group/CoGroupByKeyImpl/Tag[1] INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:52:15.251Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Group/CoGroupByKeyImpl/GroupByKey/Write into assert_that/Group/CoGroupByKeyImpl/Flatten/OutputIdentity INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:52:15.283Z: JOB_MESSAGE_DETAILED: Unzipping flatten ref_AppliedPTransform_Flatten_36 for input ref_AppliedPTransform_Combine-InjectDefault_21.None INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:52:15.310Z: JOB_MESSAGE_DETAILED: Fusing unzipped copy of assert_that/WindowInto(WindowIntoFn), through flatten Flatten, into producer Combine/InjectDefault INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:52:15.339Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Group/CoGroupByKeyImpl/Tag[1] into assert_that/ToVoidKey INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:52:15.363Z: JOB_MESSAGE_DETAILED: Unzipping flatten ref_AppliedPTransform_Flatten_36-u54 for input ref_AppliedPTransform_assert_that-WindowInto-WindowIntoFn-_43.None-c52 INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:52:15.383Z: JOB_MESSAGE_DETAILED: Fusing unzipped copy of assert_that/ToVoidKey, through flatten Flatten/Unzipped-1, into producer assert_that/WindowInto(WindowIntoFn) INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:52:15.416Z: JOB_MESSAGE_DETAILED: Fusing consumer Combine/KeyWithVoid into Map INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:52:15.445Z: JOB_MESSAGE_DETAILED: Fusing consumer Count/CombineGlobally(CountCombineFn)/KeyWithVoid into Map INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:52:15.472Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/WindowInto(WindowIntoFn) into Flatten/InputIdentity INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:52:15.503Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/ToVoidKey into assert_that/WindowInto(WindowIntoFn) INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:52:15.526Z: JOB_MESSAGE_DETAILED: Fusing consumer ReadFromMongoDB/Read/Map(<lambda at iobase.py:908>) into ReadFromMongoDB/Read/Impulse INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:52:15.553Z: JOB_MESSAGE_DETAILED: Fusing consumer ref_AppliedPTransform_ReadFromMongoDB-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_7/PairWithRestriction into ReadFromMongoDB/Read/Map(<lambda at iobase.py:908>) INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:52:15.587Z: JOB_MESSAGE_DETAILED: Fusing consumer ref_AppliedPTransform_ReadFromMongoDB-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_7/SplitWithSizing into ref_AppliedPTransform_ReadFromMongoDB-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_7/PairWithRestriction INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:52:15.611Z: JOB_MESSAGE_DETAILED: Fusing consumer Map into ref_AppliedPTransform_ReadFromMongoDB-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_7/ProcessElementAndRestrictionWithSizing INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:52:15.634Z: JOB_MESSAGE_DETAILED: Fusing consumer Combine/CombinePerKey/GroupByKey+Combine/CombinePerKey/Combine/Partial into Combine/KeyWithVoid INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:52:15.657Z: JOB_MESSAGE_DETAILED: Fusing consumer Combine/CombinePerKey/GroupByKey/Write into Combine/CombinePerKey/GroupByKey+Combine/CombinePerKey/Combine/Partial INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:52:15.683Z: JOB_MESSAGE_DETAILED: Fusing consumer Combine/CombinePerKey/Combine into Combine/CombinePerKey/GroupByKey/Read INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:52:15.712Z: JOB_MESSAGE_DETAILED: Fusing consumer Combine/CombinePerKey/Combine/Extract into Combine/CombinePerKey/Combine INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:52:15.743Z: JOB_MESSAGE_DETAILED: Fusing consumer Combine/UnKey into Combine/CombinePerKey/Combine/Extract INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:52:15.774Z: JOB_MESSAGE_DETAILED: Fusing consumer Count/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey+Count/CombineGlobally(CountCombineFn)/CombinePerKey/Combine/Partial into Count/CombineGlobally(CountCombineFn)/KeyWithVoid INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:52:15.800Z: JOB_MESSAGE_DETAILED: Fusing consumer Count/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Write into Count/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey+Count/CombineGlobally(CountCombineFn)/CombinePerKey/Combine/Partial INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:52:15.873Z: JOB_MESSAGE_DETAILED: Fusing consumer Count/CombineGlobally(CountCombineFn)/CombinePerKey/Combine into Count/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Read INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:52:15.905Z: JOB_MESSAGE_DETAILED: Fusing consumer Count/CombineGlobally(CountCombineFn)/CombinePerKey/Combine/Extract into Count/CombineGlobally(CountCombineFn)/CombinePerKey/Combine INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:52:15.941Z: JOB_MESSAGE_DETAILED: Fusing consumer Count/CombineGlobally(CountCombineFn)/UnKey into Count/CombineGlobally(CountCombineFn)/CombinePerKey/Combine/Extract INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:52:15.975Z: JOB_MESSAGE_DETAILED: Fusing consumer Combine/DoOnce/FlatMap(<lambda at core.py:3507>) into Combine/DoOnce/Impulse INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:52:15.999Z: JOB_MESSAGE_DETAILED: Fusing consumer Combine/DoOnce/Map(decode) into Combine/DoOnce/FlatMap(<lambda at core.py:3507>) INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:52:16.028Z: JOB_MESSAGE_DETAILED: Fusing consumer Combine/InjectDefault into Combine/DoOnce/Map(decode) INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:52:16.047Z: JOB_MESSAGE_DETAILED: Fusing consumer Count/CombineGlobally(CountCombineFn)/DoOnce/FlatMap(<lambda at core.py:3507>) into Count/CombineGlobally(CountCombineFn)/DoOnce/Impulse INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:52:16.077Z: JOB_MESSAGE_DETAILED: Fusing consumer Count/CombineGlobally(CountCombineFn)/DoOnce/Map(decode) into Count/CombineGlobally(CountCombineFn)/DoOnce/FlatMap(<lambda at core.py:3507>) INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:52:16.118Z: JOB_MESSAGE_DETAILED: Fusing consumer Count/CombineGlobally(CountCombineFn)/InjectDefault into Count/CombineGlobally(CountCombineFn)/DoOnce/Map(decode) INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:52:16.146Z: JOB_MESSAGE_DETAILED: Fusing consumer Flatten/InputIdentity into Count/CombineGlobally(CountCombineFn)/InjectDefault INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:52:16.172Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Create/FlatMap(<lambda at core.py:3507>) into assert_that/Create/Impulse INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:52:16.190Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Create/Map(decode) into assert_that/Create/FlatMap(<lambda at core.py:3507>) INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:52:16.222Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Group/CoGroupByKeyImpl/Tag[0] into assert_that/Create/Map(decode) INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:52:16.269Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec. INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:52:16.300Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph. INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:52:16.332Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps. INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:52:16.368Z: JOB_MESSAGE_DEBUG: Assigning stage ids. INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:52:16.552Z: JOB_MESSAGE_DEBUG: Executing wait step start73 INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:52:16.610Z: JOB_MESSAGE_BASIC: Executing operation ReadFromMongoDB/Read/Impulse+ReadFromMongoDB/Read/Map(<lambda at iobase.py:908>)+ref_AppliedPTransform_ReadFromMongoDB-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_7/PairWithRestriction+ref_AppliedPTransform_ReadFromMongoDB-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_7/SplitWithSizing INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:52:16.648Z: JOB_MESSAGE_DEBUG: Starting worker pool setup. INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:52:16.668Z: JOB_MESSAGE_BASIC: Starting 5 workers in us-central1-b... INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2023-02-22_07_52_06-2712740237206067 is in state JOB_STATE_RUNNING INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:52:37.268Z: JOB_MESSAGE_BASIC: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:52:45.654Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of workers to 1 based on the rate of progress in the currently running stage(s). INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:52:45.681Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized worker pool to 1, though goal was 5. This could be a quota issue. INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:52:55.330Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of workers to 5 based on the rate of progress in the currently running stage(s). INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:52:55.366Z: JOB_MESSAGE_DETAILED: Autoscaling: Would further reduce the number of workers but reached the minimum number allowed for the job. INFO:apache_beam.runners.dataflow.dataflow_runner:2023-02-22T15:53:22.494Z: JOB_MESSAGE_DETAILED: Workers have started successfully. FATAL: command execution failed java.io.IOException: Backing channel 'apache-beam-jenkins-2' is disconnected. at hudson.remoting.RemoteInvocationHandler.channelOrFail(RemoteInvocationHandler.java:215) at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:285) at com.sun.proxy.$Proxy138.isAlive(Unknown Source) at hudson.Launcher$RemoteLauncher$ProcImpl.isAlive(Launcher.java:1215) at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:1207) at hudson.Launcher$ProcStarter.join(Launcher.java:524) at hudson.plugins.gradle.Gradle.perform(Gradle.java:317) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:814) at hudson.model.Build$BuildExecution.build(Build.java:199) at hudson.model.Build$BuildExecution.doRun(Build.java:164) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:522) at hudson.model.Run.execute(Run.java:1896) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44) at hudson.model.ResourceController.execute(ResourceController.java:101) at hudson.model.Executor.run(Executor.java:442) Caused by: java.io.IOException: Pipe closed after 0 cycles at org.apache.sshd.common.channel.ChannelPipedInputStream.read(ChannelPipedInputStream.java:126) at org.apache.sshd.common.channel.ChannelPipedInputStream.read(ChannelPipedInputStream.java:105) at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:94) at hudson.remoting.ChunkedInputStream.readHeader(ChunkedInputStream.java:75) at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:105) at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:39) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:61) Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure ERROR: apache-beam-jenkins-2 is offline; cannot locate jdk_1.8_latest --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
