See <https://ci-beam.apache.org/job/beam_PostCommit_Python37/4315/display/redirect>
Changes: ------------------------------------------ [...truncated 213.00 KB...] + docker-compose -p hdfs_IT-jenkins-beam_PostCommit_Python37-4315 --no-ansi down Removing hdfs_it-jenkins-beam_postcommit_python37-4315_test_1 ... Removing hdfs_it-jenkins-beam_postcommit_python37-4315_datanode_1 ... Removing hdfs_it-jenkins-beam_postcommit_python37-4315_namenode_1 ... Removing hdfs_it-jenkins-beam_postcommit_python37-4315_datanode_1 ... done Removing hdfs_it-jenkins-beam_postcommit_python37-4315_test_1 ... done Removing hdfs_it-jenkins-beam_postcommit_python37-4315_namenode_1 ... done Removing network hdfs_it-jenkins-beam_postcommit_python37-4315_test_net real 0m1.319s user 0m0.605s sys 0m0.132s > Task :sdks:python:test-suites:direct:py37:installGcpTest > Task :sdks:python:test-suites:direct:py37:directRunnerIT > Task :sdks:python:test-suites:direct:py37:mongodbioIT INFO:root:Missing pipeline option (runner). Executing pipeline using the default runner: DirectRunner. INFO:__main__:Writing 100000 documents to mongodb <https://ci-beam.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/io/mongodbio_it_test.py>:80: FutureWarning: WriteToMongoDB is experimental. known_args.batch_size)) WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter. INFO:root:Default Python SDK image for environment is apache/beam_python3.7_sdk:2.34.0.dev INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function annotate_downstream_side_inputs at 0x7fb072a0b268> ==================== INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function fix_side_input_pcoll_coders at 0x7fb072a0b378> ==================== INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function pack_combiners at 0x7fb072a0b7b8> ==================== INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function lift_combiners at 0x7fb072a0b840> ==================== INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function expand_sdf at 0x7fb072a0b9d8> ==================== INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function expand_gbk at 0x7fb072a0ba60> ==================== INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function sink_flattens at 0x7fb072a0bb70> ==================== INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function greedily_fuse at 0x7fb072a0bbf8> ==================== INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function read_to_impulse at 0x7fb072a0bc80> ==================== INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function impulse_to_input at 0x7fb072a0bd08> ==================== INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function sort_stages at 0x7fb072a0bf28> ==================== INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function setup_timer_mapping at 0x7fb072a0bea0> ==================== INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function populate_data_channel_coders at 0x7fb072a0c048> ==================== INFO:apache_beam.runners.worker.statecache:Creating state cache with size 100 INFO:apache_beam.runners.portability.fn_api_runner.worker_handlers:Created Worker handler <apache_beam.runners.portability.fn_api_runner.worker_handlers.EmbeddedWorkerHandler object at 0x7fb07269dfd0> for environment ref_Environment_default_environment_1 (beam:env:embedded_python:v1, b'') INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running (((((((ref_AppliedPTransform_Create-Impulse_3)+(ref_AppliedPTransform_Create-FlatMap-lambda-at-core-py-2965-_4))+(ref_AppliedPTransform_Create-Map-decode-_6))+(ref_AppliedPTransform_Create-documents_7))+(ref_AppliedPTransform_WriteToMongoDB-ParDo-_GenerateObjectIdFn-_9))+(ref_AppliedPTransform_WriteToMongoDB-Reshuffle-AddRandomKeys_11))+(ref_AppliedPTransform_WriteToMongoDB-Reshuffle-ReshufflePerKey-Map-reify_timestamps-_13))+(WriteToMongoDB/Reshuffle/ReshufflePerKey/GroupByKey/Write) INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running (((WriteToMongoDB/Reshuffle/ReshufflePerKey/GroupByKey/Read)+(ref_AppliedPTransform_WriteToMongoDB-Reshuffle-ReshufflePerKey-FlatMap-restore_timestamps-_15))+(ref_AppliedPTransform_WriteToMongoDB-Reshuffle-RemoveRandomKeys_16))+(ref_AppliedPTransform_WriteToMongoDB-ParDo-_WriteMongoFn-_17) INFO:__main__:Writing 100000 documents to mongodb finished in 28.050 seconds INFO:root:Missing pipeline option (runner). Executing pipeline using the default runner: DirectRunner. INFO:__main__:================================================================================ INFO:__main__:Reading from mongodb beam_mongodbio_it_db:integration_test_1632571979 INFO:__main__:reader params : {'projection': ['number']} INFO:__main__:expected results: {'number_sum': 4999950000, 'docs_count': 100000} <https://ci-beam.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/io/mongodbio_it_test.py>:153: FutureWarning: ReadFromMongoDB is experimental. | 'Map' >> beam.Map(lambda doc: doc['number'])) WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter. INFO:root:Default Python SDK image for environment is apache/beam_python3.7_sdk:2.34.0.dev INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function annotate_downstream_side_inputs at 0x7fb072a0b268> ==================== INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function fix_side_input_pcoll_coders at 0x7fb072a0b378> ==================== INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function pack_combiners at 0x7fb072a0b7b8> ==================== INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function lift_combiners at 0x7fb072a0b840> ==================== INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function expand_sdf at 0x7fb072a0b9d8> ==================== INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function expand_gbk at 0x7fb072a0ba60> ==================== INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function sink_flattens at 0x7fb072a0bb70> ==================== INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function greedily_fuse at 0x7fb072a0bbf8> ==================== INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function read_to_impulse at 0x7fb072a0bc80> ==================== INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function impulse_to_input at 0x7fb072a0bd08> ==================== INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function sort_stages at 0x7fb072a0bf28> ==================== INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function setup_timer_mapping at 0x7fb072a0bea0> ==================== INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function populate_data_channel_coders at 0x7fb072a0c048> ==================== INFO:apache_beam.runners.worker.statecache:Creating state cache with size 100 INFO:apache_beam.runners.portability.fn_api_runner.worker_handlers:Created Worker handler <apache_beam.runners.portability.fn_api_runner.worker_handlers.EmbeddedWorkerHandler object at 0x7fb070a344a8> for environment ref_Environment_default_environment_1 (beam:env:embedded_python:v1, b'') INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running (((((ref_AppliedPTransform_assert_that-Create-Impulse_39)+(ref_AppliedPTransform_assert_that-Create-FlatMap-lambda-at-core-py-2965-_40))+(ref_AppliedPTransform_assert_that-Create-Map-decode-_42))+(ref_AppliedPTransform_assert_that-Group-CoGroupByKeyImpl-Tag-0-_47))+(assert_that/Group/CoGroupByKeyImpl/Flatten/Transcode/0))+(assert_that/Group/CoGroupByKeyImpl/Flatten/Write/0) INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running ((((ref_AppliedPTransform_ReadFromMongoDB-Read-Impulse_4)+(ref_AppliedPTransform_ReadFromMongoDB-Read-Map-lambda-at-iobase-py-898-_5))+(ReadFromMongoDB/Read/SDFBoundedSourceReader/ParDo(SDFBoundedSourceDoFn)/PairWithRestriction))+(ReadFromMongoDB/Read/SDFBoundedSourceReader/ParDo(SDFBoundedSourceDoFn)/SplitAndSizeRestriction))+(ref_PCollection_PCollection_2_split/Write) INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running ((((((((ref_PCollection_PCollection_2_split/Read)+(ReadFromMongoDB/Read/SDFBoundedSourceReader/ParDo(SDFBoundedSourceDoFn)/Process))+(ref_AppliedPTransform_Map_8))+(ref_AppliedPTransform_Combine-KeyWithVoid_10))+(ref_AppliedPTransform_Count-CombineGlobally-CountCombineFn-KeyWithVoid_24))+(Combine/CombinePerKey/Precombine))+(Combine/CombinePerKey/Group/Write))+(Count/CombineGlobally(CountCombineFn)/CombinePerKey/Precombine))+(Count/CombineGlobally(CountCombineFn)/CombinePerKey/Group/Write) INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running ((((Combine/CombinePerKey/Group/Read)+(Combine/CombinePerKey/Merge))+(Combine/CombinePerKey/ExtractOutputs))+(ref_AppliedPTransform_Combine-UnKey_15))+(ref_PCollection_PCollection_8/Write) INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running ((((ref_AppliedPTransform_Combine-DoOnce-Impulse_17)+(ref_AppliedPTransform_Combine-DoOnce-FlatMap-lambda-at-core-py-2965-_18))+(ref_AppliedPTransform_Combine-DoOnce-Map-decode-_20))+(ref_AppliedPTransform_Combine-InjectDefault_21))+(Flatten/Write/0) INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running ((((Count/CombineGlobally(CountCombineFn)/CombinePerKey/Group/Read)+(Count/CombineGlobally(CountCombineFn)/CombinePerKey/Merge))+(Count/CombineGlobally(CountCombineFn)/CombinePerKey/ExtractOutputs))+(ref_AppliedPTransform_Count-CombineGlobally-CountCombineFn-UnKey_29))+(ref_PCollection_PCollection_16/Write) INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running (((((ref_AppliedPTransform_Count-CombineGlobally-CountCombineFn-DoOnce-Impulse_31)+(ref_AppliedPTransform_Count-CombineGlobally-CountCombineFn-DoOnce-FlatMap-lambda-at-core-py-2965-_32))+(ref_AppliedPTransform_Count-CombineGlobally-CountCombineFn-DoOnce-Map-decode-_34))+(ref_AppliedPTransform_Count-CombineGlobally-CountCombineFn-InjectDefault_35))+(Flatten/Transcode/1))+(Flatten/Write/1) INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running (((((Flatten/Read)+(ref_AppliedPTransform_assert_that-WindowInto-WindowIntoFn-_43))+(ref_AppliedPTransform_assert_that-ToVoidKey_44))+(ref_AppliedPTransform_assert_that-Group-CoGroupByKeyImpl-Tag-1-_48))+(assert_that/Group/CoGroupByKeyImpl/Flatten/Transcode/1))+(assert_that/Group/CoGroupByKeyImpl/Flatten/Write/1) INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running (assert_that/Group/CoGroupByKeyImpl/Flatten/Read)+(assert_that/Group/CoGroupByKeyImpl/GroupByKey/Write) INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running ((((assert_that/Group/CoGroupByKeyImpl/GroupByKey/Read)+(ref_AppliedPTransform_assert_that-Group-CoGroupByKeyImpl-MapTuple-collect_values-_51))+(ref_AppliedPTransform_assert_that-Group-RestoreTags_52))+(ref_AppliedPTransform_assert_that-Unkey_53))+(ref_AppliedPTransform_assert_that-Match_54) INFO:__main__:Reading documents from mongodb finished in 5.442 seconds INFO:root:Missing pipeline option (runner). Executing pipeline using the default runner: DirectRunner. INFO:__main__:================================================================================ INFO:__main__:Reading from mongodb beam_mongodbio_it_db:integration_test_1632571979 INFO:__main__:reader params : {'filter': {'number_mod_3': 0}, 'projection': ['number']} INFO:__main__:expected results: {'number_sum': 1666683333, 'docs_count': 33334} WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter. INFO:root:Default Python SDK image for environment is apache/beam_python3.7_sdk:2.34.0.dev INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function annotate_downstream_side_inputs at 0x7fb072a0b268> ==================== INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function fix_side_input_pcoll_coders at 0x7fb072a0b378> ==================== INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function pack_combiners at 0x7fb072a0b7b8> ==================== INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function lift_combiners at 0x7fb072a0b840> ==================== INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function expand_sdf at 0x7fb072a0b9d8> ==================== INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function expand_gbk at 0x7fb072a0ba60> ==================== INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function sink_flattens at 0x7fb072a0bb70> ==================== INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function greedily_fuse at 0x7fb072a0bbf8> ==================== INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function read_to_impulse at 0x7fb072a0bc80> ==================== INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function impulse_to_input at 0x7fb072a0bd08> ==================== INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function sort_stages at 0x7fb072a0bf28> ==================== INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function setup_timer_mapping at 0x7fb072a0bea0> ==================== INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function populate_data_channel_coders at 0x7fb072a0c048> ==================== INFO:apache_beam.runners.worker.statecache:Creating state cache with size 100 INFO:apache_beam.runners.portability.fn_api_runner.worker_handlers:Created Worker handler <apache_beam.runners.portability.fn_api_runner.worker_handlers.EmbeddedWorkerHandler object at 0x7fb071ec2908> for environment ref_Environment_default_environment_1 (beam:env:embedded_python:v1, b'') INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running ((((ref_AppliedPTransform_ReadFromMongoDB-Read-Impulse_4)+(ref_AppliedPTransform_ReadFromMongoDB-Read-Map-lambda-at-iobase-py-898-_5))+(ReadFromMongoDB/Read/SDFBoundedSourceReader/ParDo(SDFBoundedSourceDoFn)/PairWithRestriction))+(ReadFromMongoDB/Read/SDFBoundedSourceReader/ParDo(SDFBoundedSourceDoFn)/SplitAndSizeRestriction))+(ref_PCollection_PCollection_2_split/Write) INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running ((((((((ref_PCollection_PCollection_2_split/Read)+(ReadFromMongoDB/Read/SDFBoundedSourceReader/ParDo(SDFBoundedSourceDoFn)/Process))+(ref_AppliedPTransform_Map_8))+(ref_AppliedPTransform_Combine-KeyWithVoid_10))+(ref_AppliedPTransform_Count-CombineGlobally-CountCombineFn-KeyWithVoid_24))+(Combine/CombinePerKey/Precombine))+(Combine/CombinePerKey/Group/Write))+(Count/CombineGlobally(CountCombineFn)/CombinePerKey/Precombine))+(Count/CombineGlobally(CountCombineFn)/CombinePerKey/Group/Write) INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running ((((Combine/CombinePerKey/Group/Read)+(Combine/CombinePerKey/Merge))+(Combine/CombinePerKey/ExtractOutputs))+(ref_AppliedPTransform_Combine-UnKey_15))+(ref_PCollection_PCollection_8/Write) INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running ((((ref_AppliedPTransform_Combine-DoOnce-Impulse_17)+(ref_AppliedPTransform_Combine-DoOnce-FlatMap-lambda-at-core-py-2965-_18))+(ref_AppliedPTransform_Combine-DoOnce-Map-decode-_20))+(ref_AppliedPTransform_Combine-InjectDefault_21))+(Flatten/Write/0) INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running (((((ref_AppliedPTransform_assert_that-Create-Impulse_39)+(ref_AppliedPTransform_assert_that-Create-FlatMap-lambda-at-core-py-2965-_40))+(ref_AppliedPTransform_assert_that-Create-Map-decode-_42))+(ref_AppliedPTransform_assert_that-Group-CoGroupByKeyImpl-Tag-0-_47))+(assert_that/Group/CoGroupByKeyImpl/Flatten/Transcode/0))+(assert_that/Group/CoGroupByKeyImpl/Flatten/Write/0) INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running ((((Count/CombineGlobally(CountCombineFn)/CombinePerKey/Group/Read)+(Count/CombineGlobally(CountCombineFn)/CombinePerKey/Merge))+(Count/CombineGlobally(CountCombineFn)/CombinePerKey/ExtractOutputs))+(ref_AppliedPTransform_Count-CombineGlobally-CountCombineFn-UnKey_29))+(ref_PCollection_PCollection_16/Write) INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running (((((ref_AppliedPTransform_Count-CombineGlobally-CountCombineFn-DoOnce-Impulse_31)+(ref_AppliedPTransform_Count-CombineGlobally-CountCombineFn-DoOnce-FlatMap-lambda-at-core-py-2965-_32))+(ref_AppliedPTransform_Count-CombineGlobally-CountCombineFn-DoOnce-Map-decode-_34))+(ref_AppliedPTransform_Count-CombineGlobally-CountCombineFn-InjectDefault_35))+(Flatten/Transcode/1))+(Flatten/Write/1) INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running (((((Flatten/Read)+(ref_AppliedPTransform_assert_that-WindowInto-WindowIntoFn-_43))+(ref_AppliedPTransform_assert_that-ToVoidKey_44))+(ref_AppliedPTransform_assert_that-Group-CoGroupByKeyImpl-Tag-1-_48))+(assert_that/Group/CoGroupByKeyImpl/Flatten/Transcode/1))+(assert_that/Group/CoGroupByKeyImpl/Flatten/Write/1) INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running (assert_that/Group/CoGroupByKeyImpl/Flatten/Read)+(assert_that/Group/CoGroupByKeyImpl/GroupByKey/Write) INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running ((((assert_that/Group/CoGroupByKeyImpl/GroupByKey/Read)+(ref_AppliedPTransform_assert_that-Group-CoGroupByKeyImpl-MapTuple-collect_values-_51))+(ref_AppliedPTransform_assert_that-Group-RestoreTags_52))+(ref_AppliedPTransform_assert_that-Unkey_53))+(ref_AppliedPTransform_assert_that-Match_54) INFO:__main__:Reading documents from mongodb finished in 3.087 seconds INFO:root:Missing pipeline option (runner). Executing pipeline using the default runner: DirectRunner. INFO:__main__:================================================================================ INFO:__main__:Reading from mongodb beam_mongodbio_it_db:integration_test_1632571979 INFO:__main__:reader params : {'projection': ['number'], 'bucket_auto': True} INFO:__main__:expected results: {'number_sum': 4999950000, 'docs_count': 100000} WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter. INFO:root:Default Python SDK image for environment is apache/beam_python3.7_sdk:2.34.0.dev INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function annotate_downstream_side_inputs at 0x7fb072a0b268> ==================== INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function fix_side_input_pcoll_coders at 0x7fb072a0b378> ==================== INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function pack_combiners at 0x7fb072a0b7b8> ==================== INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function lift_combiners at 0x7fb072a0b840> ==================== INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function expand_sdf at 0x7fb072a0b9d8> ==================== INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function expand_gbk at 0x7fb072a0ba60> ==================== INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function sink_flattens at 0x7fb072a0bb70> ==================== INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function greedily_fuse at 0x7fb072a0bbf8> ==================== INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function read_to_impulse at 0x7fb072a0bc80> ==================== INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function impulse_to_input at 0x7fb072a0bd08> ==================== INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function sort_stages at 0x7fb072a0bf28> ==================== INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function setup_timer_mapping at 0x7fb072a0bea0> ==================== INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function populate_data_channel_coders at 0x7fb072a0c048> ==================== INFO:apache_beam.runners.worker.statecache:Creating state cache with size 100 INFO:apache_beam.runners.portability.fn_api_runner.worker_handlers:Created Worker handler <apache_beam.runners.portability.fn_api_runner.worker_handlers.EmbeddedWorkerHandler object at 0x7fb07150b6d8> for environment ref_Environment_default_environment_1 (beam:env:embedded_python:v1, b'') INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running ((((ref_AppliedPTransform_ReadFromMongoDB-Read-Impulse_4)+(ref_AppliedPTransform_ReadFromMongoDB-Read-Map-lambda-at-iobase-py-898-_5))+(ReadFromMongoDB/Read/SDFBoundedSourceReader/ParDo(SDFBoundedSourceDoFn)/PairWithRestriction))+(ReadFromMongoDB/Read/SDFBoundedSourceReader/ParDo(SDFBoundedSourceDoFn)/SplitAndSizeRestriction))+(ref_PCollection_PCollection_2_split/Write) INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running ((((((((ref_PCollection_PCollection_2_split/Read)+(ReadFromMongoDB/Read/SDFBoundedSourceReader/ParDo(SDFBoundedSourceDoFn)/Process))+(ref_AppliedPTransform_Map_8))+(ref_AppliedPTransform_Combine-KeyWithVoid_10))+(ref_AppliedPTransform_Count-CombineGlobally-CountCombineFn-KeyWithVoid_24))+(Combine/CombinePerKey/Precombine))+(Combine/CombinePerKey/Group/Write))+(Count/CombineGlobally(CountCombineFn)/CombinePerKey/Precombine))+(Count/CombineGlobally(CountCombineFn)/CombinePerKey/Group/Write) INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running ((((Combine/CombinePerKey/Group/Read)+(Combine/CombinePerKey/Merge))+(Combine/CombinePerKey/ExtractOutputs))+(ref_AppliedPTransform_Combine-UnKey_15))+(ref_PCollection_PCollection_8/Write) INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running (((((ref_AppliedPTransform_assert_that-Create-Impulse_39)+(ref_AppliedPTransform_assert_that-Create-FlatMap-lambda-at-core-py-2965-_40))+(ref_AppliedPTransform_assert_that-Create-Map-decode-_42))+(ref_AppliedPTransform_assert_that-Group-CoGroupByKeyImpl-Tag-0-_47))+(assert_that/Group/CoGroupByKeyImpl/Flatten/Transcode/0))+(assert_that/Group/CoGroupByKeyImpl/Flatten/Write/0) INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running ((((ref_AppliedPTransform_Combine-DoOnce-Impulse_17)+(ref_AppliedPTransform_Combine-DoOnce-FlatMap-lambda-at-core-py-2965-_18))+(ref_AppliedPTransform_Combine-DoOnce-Map-decode-_20))+(ref_AppliedPTransform_Combine-InjectDefault_21))+(Flatten/Write/0) INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running ((((Count/CombineGlobally(CountCombineFn)/CombinePerKey/Group/Read)+(Count/CombineGlobally(CountCombineFn)/CombinePerKey/Merge))+(Count/CombineGlobally(CountCombineFn)/CombinePerKey/ExtractOutputs))+(ref_AppliedPTransform_Count-CombineGlobally-CountCombineFn-UnKey_29))+(ref_PCollection_PCollection_16/Write) INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running (((((ref_AppliedPTransform_Count-CombineGlobally-CountCombineFn-DoOnce-Impulse_31)+(ref_AppliedPTransform_Count-CombineGlobally-CountCombineFn-DoOnce-FlatMap-lambda-at-core-py-2965-_32))+(ref_AppliedPTransform_Count-CombineGlobally-CountCombineFn-DoOnce-Map-decode-_34))+(ref_AppliedPTransform_Count-CombineGlobally-CountCombineFn-InjectDefault_35))+(Flatten/Transcode/1))+(Flatten/Write/1) INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running (((((Flatten/Read)+(ref_AppliedPTransform_assert_that-WindowInto-WindowIntoFn-_43))+(ref_AppliedPTransform_assert_that-ToVoidKey_44))+(ref_AppliedPTransform_assert_that-Group-CoGroupByKeyImpl-Tag-1-_48))+(assert_that/Group/CoGroupByKeyImpl/Flatten/Transcode/1))+(assert_that/Group/CoGroupByKeyImpl/Flatten/Write/1) INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running (assert_that/Group/CoGroupByKeyImpl/Flatten/Read)+(assert_that/Group/CoGroupByKeyImpl/GroupByKey/Write) INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running ((((assert_that/Group/CoGroupByKeyImpl/GroupByKey/Read)+(ref_AppliedPTransform_assert_that-Group-CoGroupByKeyImpl-MapTuple-collect_values-_51))+(ref_AppliedPTransform_assert_that-Group-RestoreTags_52))+(ref_AppliedPTransform_assert_that-Unkey_53))+(ref_AppliedPTransform_assert_that-Match_54) INFO:__main__:Reading documents from mongodb finished in 6.828 seconds INFO:root:Missing pipeline option (runner). Executing pipeline using the default runner: DirectRunner. INFO:__main__:================================================================================ INFO:__main__:Reading from mongodb beam_mongodbio_it_db:integration_test_1632571979 INFO:__main__:reader params : {'filter': {'number_mod_3': 0}, 'projection': ['number'], 'bucket_auto': True} INFO:__main__:expected results: {'number_sum': 1666683333, 'docs_count': 33334} WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter. INFO:root:Default Python SDK image for environment is apache/beam_python3.7_sdk:2.34.0.dev INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function annotate_downstream_side_inputs at 0x7fb072a0b268> ==================== INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function fix_side_input_pcoll_coders at 0x7fb072a0b378> ==================== INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function pack_combiners at 0x7fb072a0b7b8> ==================== INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function lift_combiners at 0x7fb072a0b840> ==================== INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function expand_sdf at 0x7fb072a0b9d8> ==================== INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function expand_gbk at 0x7fb072a0ba60> ==================== INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function sink_flattens at 0x7fb072a0bb70> ==================== INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function greedily_fuse at 0x7fb072a0bbf8> ==================== INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function read_to_impulse at 0x7fb072a0bc80> ==================== INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function impulse_to_input at 0x7fb072a0bd08> ==================== INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function sort_stages at 0x7fb072a0bf28> ==================== INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function setup_timer_mapping at 0x7fb072a0bea0> ==================== INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function populate_data_channel_coders at 0x7fb072a0c048> ==================== INFO:apache_beam.runners.worker.statecache:Creating state cache with size 100 INFO:apache_beam.runners.portability.fn_api_runner.worker_handlers:Created Worker handler <apache_beam.runners.portability.fn_api_runner.worker_handlers.EmbeddedWorkerHandler object at 0x7fb071b556a0> for environment ref_Environment_default_environment_1 (beam:env:embedded_python:v1, b'') INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running ((((ref_AppliedPTransform_ReadFromMongoDB-Read-Impulse_4)+(ref_AppliedPTransform_ReadFromMongoDB-Read-Map-lambda-at-iobase-py-898-_5))+(ReadFromMongoDB/Read/SDFBoundedSourceReader/ParDo(SDFBoundedSourceDoFn)/PairWithRestriction))+(ReadFromMongoDB/Read/SDFBoundedSourceReader/ParDo(SDFBoundedSourceDoFn)/SplitAndSizeRestriction))+(ref_PCollection_PCollection_2_split/Write) INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running ((((((((ref_PCollection_PCollection_2_split/Read)+(ReadFromMongoDB/Read/SDFBoundedSourceReader/ParDo(SDFBoundedSourceDoFn)/Process))+(ref_AppliedPTransform_Map_8))+(ref_AppliedPTransform_Combine-KeyWithVoid_10))+(ref_AppliedPTransform_Count-CombineGlobally-CountCombineFn-KeyWithVoid_24))+(Combine/CombinePerKey/Precombine))+(Combine/CombinePerKey/Group/Write))+(Count/CombineGlobally(CountCombineFn)/CombinePerKey/Precombine))+(Count/CombineGlobally(CountCombineFn)/CombinePerKey/Group/Write) INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running ((((Count/CombineGlobally(CountCombineFn)/CombinePerKey/Group/Read)+(Count/CombineGlobally(CountCombineFn)/CombinePerKey/Merge))+(Count/CombineGlobally(CountCombineFn)/CombinePerKey/ExtractOutputs))+(ref_AppliedPTransform_Count-CombineGlobally-CountCombineFn-UnKey_29))+(ref_PCollection_PCollection_16/Write) INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running (((((ref_AppliedPTransform_Count-CombineGlobally-CountCombineFn-DoOnce-Impulse_31)+(ref_AppliedPTransform_Count-CombineGlobally-CountCombineFn-DoOnce-FlatMap-lambda-at-core-py-2965-_32))+(ref_AppliedPTransform_Count-CombineGlobally-CountCombineFn-DoOnce-Map-decode-_34))+(ref_AppliedPTransform_Count-CombineGlobally-CountCombineFn-InjectDefault_35))+(Flatten/Transcode/1))+(Flatten/Write/1) INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running ((((Combine/CombinePerKey/Group/Read)+(Combine/CombinePerKey/Merge))+(Combine/CombinePerKey/ExtractOutputs))+(ref_AppliedPTransform_Combine-UnKey_15))+(ref_PCollection_PCollection_8/Write) INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running (((((ref_AppliedPTransform_assert_that-Create-Impulse_39)+(ref_AppliedPTransform_assert_that-Create-FlatMap-lambda-at-core-py-2965-_40))+(ref_AppliedPTransform_assert_that-Create-Map-decode-_42))+(ref_AppliedPTransform_assert_that-Group-CoGroupByKeyImpl-Tag-0-_47))+(assert_that/Group/CoGroupByKeyImpl/Flatten/Transcode/0))+(assert_that/Group/CoGroupByKeyImpl/Flatten/Write/0) INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running ((((ref_AppliedPTransform_Combine-DoOnce-Impulse_17)+(ref_AppliedPTransform_Combine-DoOnce-FlatMap-lambda-at-core-py-2965-_18))+(ref_AppliedPTransform_Combine-DoOnce-Map-decode-_20))+(ref_AppliedPTransform_Combine-InjectDefault_21))+(Flatten/Write/0) INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running (((((Flatten/Read)+(ref_AppliedPTransform_assert_that-WindowInto-WindowIntoFn-_43))+(ref_AppliedPTransform_assert_that-ToVoidKey_44))+(ref_AppliedPTransform_assert_that-Group-CoGroupByKeyImpl-Tag-1-_48))+(assert_that/Group/CoGroupByKeyImpl/Flatten/Transcode/1))+(assert_that/Group/CoGroupByKeyImpl/Flatten/Write/1) INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running (assert_that/Group/CoGroupByKeyImpl/Flatten/Read)+(assert_that/Group/CoGroupByKeyImpl/GroupByKey/Write) INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running ((((assert_that/Group/CoGroupByKeyImpl/GroupByKey/Read)+(ref_AppliedPTransform_assert_that-Group-CoGroupByKeyImpl-MapTuple-collect_values-_51))+(ref_AppliedPTransform_assert_that-Group-RestoreTags_52))+(ref_AppliedPTransform_assert_that-Unkey_53))+(ref_AppliedPTransform_assert_that-Match_54) INFO:__main__:Reading documents from mongodb finished in 3.818 seconds > Task :sdks:python:test-suites:direct:py37:postCommitIT FAILURE: Build completed with 5 failures. 1: Task failed with an exception. ----------- * What went wrong: Execution failed for task ':sdks:java:extensions:sql:expansion-service:compileJava'. > Could not create task ':sdks:java:testing:test-utils:clean'. > java.util.ConcurrentModificationException (no error message) * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. ============================================================================== 2: Task failed with an exception. ----------- * What went wrong: Execution failed for task ':sdks:java:container:pullLicenses'. > Could not create task ':sdks:java:testing:test-utils:clean'. > java.util.ConcurrentModificationException (no error message) * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. ============================================================================== 3: Task failed with an exception. ----------- * What went wrong: Execution failed for task ':sdks:python:container:py37:copyDockerfileDependencies'. > Could not create task ':sdks:java:testing:test-utils:clean'. > java.util.ConcurrentModificationException (no error message) * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. ============================================================================== 4: Task failed with an exception. ----------- * What went wrong: Execution failed for task ':sdks:python:container:py37:copyGolangLicenses'. > Could not create task ':sdks:java:testing:test-utils:clean'. > java.util.ConcurrentModificationException (no error message) * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. ============================================================================== 5: Task failed with an exception. ----------- * What went wrong: Execution failed for task ':sdks:java:container:java8:copyGolangLicenses'. > Could not create task ':sdks:java:testing:test-utils:clean'. > java.util.ConcurrentModificationException (no error message) * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. ============================================================================== * Get more help at https://help.gradle.org Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/6.9.1/userguide/command_line_interface.html#sec:command_line_warnings BUILD FAILED in 1h 40m 54s 204 actionable tasks: 143 executed, 57 from cache, 4 up-to-date Publishing build scan... https://gradle.com/s/ggsguq56j4kqy Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
