See
<https://ci-beam.apache.org/job/beam_PostCommit_Python37/4525/display/redirect?page=changes>
Changes:
[aydar.zaynutdinov] [BEAM-13178][Playground]
[Pablo Estrada] [BEAM-8033] Throwing an error if a dataSourceProviderFn is
defined twice
[minherz] fix: update dlp dependency version range
[noreply] Merge pull request #15740 from [BEAM-12936][Playground] Code editor -
[noreply] Merge pull request #15976 from [Playground][BEAM-12941][Bugfix] Fix
[noreply] Merge pull request #15991 [BEAM-13251][Playground] [Bugfix] Lint Fails
[noreply] Merge pull request #15990 from [BEAM-13177][Playground] Change using
of
[noreply] [BEAM-13262] Forward for metrics.SingleResult (#15993)
[noreply] Generate Python container dependencies in an automated way. (#15927)
[noreply] [BEAM-13264] Allow pyarrow up to 6.x (#15995)
[noreply] Update the go get command with v2 (#15945)
[noreply] Go Quickstart: switch to Spark 3 JobServer for better out-of-box
[noreply] Golint fixes for recent Go SDK import (#15999)
------------------------------------------
[...truncated 18.03 MB...]
return self._next()
File
"<https://ci-beam.apache.org/job/beam_PostCommit_Python37/ws/src/build/gradleenv/2022703441/lib/python3.7/site-packages/grpc/_channel.py";,>
line 826, in _next
raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that
terminated with:
status = StatusCode.UNAVAILABLE
details = "Socket closed"
debug_error_string =
"{"created":"@1637108223.567763205","description":"Error received from peer
ipv4:127.0.0.1:33267","file":"src/core/lib/surface/call.cc","file_line":1069,"grpc_message":"Socket
closed","grpc_status":14}"
>
Exception in thread run_worker_1-1:
Traceback (most recent call last):
File "/usr/lib/python3.7/threading.py", line 926, in _bootstrap_inner
self.run()
File "/usr/lib/python3.7/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File
"<https://ci-beam.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker.py";,>
line 234, in run
for work_request in self._control_stub.Control(get_responses()):
File
"<https://ci-beam.apache.org/job/beam_PostCommit_Python37/ws/src/build/gradleenv/2022703441/lib/python3.7/site-packages/grpc/_channel.py";,>
line 426, in __next__
return self._next()
File
"<https://ci-beam.apache.org/job/beam_PostCommit_Python37/ws/src/build/gradleenv/2022703441/lib/python3.7/site-packages/grpc/_channel.py";,>
line 826, in _next
raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that
terminated with:
status = StatusCode.UNAVAILABLE
details = "Socket closed"
debug_error_string =
"{"created":"@1637108223.567793209","description":"Error received from peer
ipv4:127.0.0.1:41329","file":"src/core/lib/surface/call.cc","file_line":1069,"grpc_message":"Socket
closed","grpc_status":14}"
>
> Task :sdks:python:test-suites:portable:py37:postCommitPy37
> Task :sdks:python:test-suites:direct:py37:directRunnerIT FAILED
> Task :sdks:python:test-suites:direct:py37:mongodbioIT
INFO:root:Missing pipeline option (runner). Executing pipeline using the
default runner: DirectRunner.
INFO:__main__:Writing 100000 documents to mongodb
<https://ci-beam.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/io/mongodbio_it_test.py>:80:
FutureWarning: WriteToMongoDB is experimental.
known_args.batch_size))
WARNING:root:Make sure that locally built Python SDK docker image has Python
3.7 interpreter.
INFO:root:Default Python SDK image for environment is
apache/beam_python3.7_sdk:2.35.0.dev
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function annotate_downstream_side_inputs at 0x7f28de906560>
====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function fix_side_input_pcoll_coders at 0x7f28de906680> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function pack_combiners at 0x7f28de906b90> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function lift_combiners at 0x7f28de906c20> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function expand_sdf at 0x7f28de906dd0> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function expand_gbk at 0x7f28de906e60> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function sink_flattens at 0x7f28de906f80> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function greedily_fuse at 0x7f28de907050> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function read_to_impulse at 0x7f28de9070e0> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function impulse_to_input at 0x7f28de907170> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function sort_stages at 0x7f28de9073b0> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function setup_timer_mapping at 0x7f28de907320> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function populate_data_channel_coders at 0x7f28de907440> ====================
INFO:apache_beam.runners.worker.statecache:Creating state cache with size 100
INFO:apache_beam.runners.portability.fn_api_runner.worker_handlers:Created
Worker handler
<apache_beam.runners.portability.fn_api_runner.worker_handlers.EmbeddedWorkerHandler
object at 0x7f28de586a90> for environment
ref_Environment_default_environment_1 (beam:env:embedded_python:v1, b'')
INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running
(((((((ref_AppliedPTransform_Create-Impulse_3)+(ref_AppliedPTransform_Create-FlatMap-lambda-at-core-py-3224-_4))+(ref_AppliedPTransform_Create-Map-decode-_6))+(ref_AppliedPTransform_Create-documents_7))+(ref_AppliedPTransform_WriteToMongoDB-ParDo-_GenerateObjectIdFn-_9))+(ref_AppliedPTransform_WriteToMongoDB-Reshuffle-AddRandomKeys_11))+(ref_AppliedPTransform_WriteToMongoDB-Reshuffle-ReshufflePerKey-Map-reify_timestamps-_13))+(WriteToMongoDB/Reshuffle/ReshufflePerKey/GroupByKey/Write)
INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running
(((WriteToMongoDB/Reshuffle/ReshufflePerKey/GroupByKey/Read)+(ref_AppliedPTransform_WriteToMongoDB-Reshuffle-ReshufflePerKey-FlatMap-restore_timestamps-_15))+(ref_AppliedPTransform_WriteToMongoDB-Reshuffle-RemoveRandomKeys_16))+(ref_AppliedPTransform_WriteToMongoDB-ParDo-_WriteMongoFn-_17)
INFO:__main__:Writing 100000 documents to mongodb finished in 27.596 seconds
INFO:root:Missing pipeline option (runner). Executing pipeline using the
default runner: DirectRunner.
INFO:__main__:================================================================================
INFO:__main__:Reading from mongodb
beam_mongodbio_it_db:integration_test_1637108246
INFO:__main__:reader params : {'projection': ['number']}
INFO:__main__:expected results: {'number_sum': 4999950000, 'docs_count': 100000}
<https://ci-beam.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/io/mongodbio_it_test.py>:153:
FutureWarning: ReadFromMongoDB is experimental.
| 'Map' >> beam.Map(lambda doc: doc['number']))
WARNING:root:Make sure that locally built Python SDK docker image has Python
3.7 interpreter.
INFO:root:Default Python SDK image for environment is
apache/beam_python3.7_sdk:2.35.0.dev
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function annotate_downstream_side_inputs at 0x7f28de906560>
====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function fix_side_input_pcoll_coders at 0x7f28de906680> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function pack_combiners at 0x7f28de906b90> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function lift_combiners at 0x7f28de906c20> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function expand_sdf at 0x7f28de906dd0> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function expand_gbk at 0x7f28de906e60> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function sink_flattens at 0x7f28de906f80> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function greedily_fuse at 0x7f28de907050> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function read_to_impulse at 0x7f28de9070e0> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function impulse_to_input at 0x7f28de907170> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function sort_stages at 0x7f28de9073b0> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function setup_timer_mapping at 0x7f28de907320> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function populate_data_channel_coders at 0x7f28de907440> ====================
INFO:apache_beam.runners.worker.statecache:Creating state cache with size 100
INFO:apache_beam.runners.portability.fn_api_runner.worker_handlers:Created
Worker handler
<apache_beam.runners.portability.fn_api_runner.worker_handlers.EmbeddedWorkerHandler
object at 0x7f28dd72de50> for environment
ref_Environment_default_environment_1 (beam:env:embedded_python:v1, b'')
INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running
((((ref_AppliedPTransform_ReadFromMongoDB-Read-Impulse_4)+(ref_AppliedPTransform_ReadFromMongoDB-Read-Map-lambda-at-iobase-py-898-_5))+(ReadFromMongoDB/Read/SDFBoundedSourceReader/ParDo(SDFBoundedSourceDoFn)/PairWithRestriction))+(ReadFromMongoDB/Read/SDFBoundedSourceReader/ParDo(SDFBoundedSourceDoFn)/SplitAndSizeRestriction))+(ref_PCollection_PCollection_2_split/Write)
INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running
((((((((ref_PCollection_PCollection_2_split/Read)+(ReadFromMongoDB/Read/SDFBoundedSourceReader/ParDo(SDFBoundedSourceDoFn)/Process))+(ref_AppliedPTransform_Map_8))+(ref_AppliedPTransform_Combine-KeyWithVoid_10))+(ref_AppliedPTransform_Count-CombineGlobally-CountCombineFn-KeyWithVoid_24))+(Combine/CombinePerKey/Precombine))+(Combine/CombinePerKey/Group/Write))+(Count/CombineGlobally(CountCombineFn)/CombinePerKey/Precombine))+(Count/CombineGlobally(CountCombineFn)/CombinePerKey/Group/Write)
INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running
(((((ref_AppliedPTransform_assert_that-Create-Impulse_39)+(ref_AppliedPTransform_assert_that-Create-FlatMap-lambda-at-core-py-3224-_40))+(ref_AppliedPTransform_assert_that-Create-Map-decode-_42))+(ref_AppliedPTransform_assert_that-Group-CoGroupByKeyImpl-Tag-0-_47))+(assert_that/Group/CoGroupByKeyImpl/Flatten/Transcode/0))+(assert_that/Group/CoGroupByKeyImpl/Flatten/Write/0)
INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running
((((Count/CombineGlobally(CountCombineFn)/CombinePerKey/Group/Read)+(Count/CombineGlobally(CountCombineFn)/CombinePerKey/Merge))+(Count/CombineGlobally(CountCombineFn)/CombinePerKey/ExtractOutputs))+(ref_AppliedPTransform_Count-CombineGlobally-CountCombineFn-UnKey_29))+(ref_PCollection_PCollection_16/Write)
INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running
(((((ref_AppliedPTransform_Count-CombineGlobally-CountCombineFn-DoOnce-Impulse_31)+(ref_AppliedPTransform_Count-CombineGlobally-CountCombineFn-DoOnce-FlatMap-lambda-at-core-py-3224-_32))+(ref_AppliedPTransform_Count-CombineGlobally-CountCombineFn-DoOnce-Map-decode-_34))+(ref_AppliedPTransform_Count-CombineGlobally-CountCombineFn-InjectDefault_35))+(Flatten/Transcode/1))+(Flatten/Write/1)
INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running
((((Combine/CombinePerKey/Group/Read)+(Combine/CombinePerKey/Merge))+(Combine/CombinePerKey/ExtractOutputs))+(ref_AppliedPTransform_Combine-UnKey_15))+(ref_PCollection_PCollection_8/Write)
INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running
((((ref_AppliedPTransform_Combine-DoOnce-Impulse_17)+(ref_AppliedPTransform_Combine-DoOnce-FlatMap-lambda-at-core-py-3224-_18))+(ref_AppliedPTransform_Combine-DoOnce-Map-decode-_20))+(ref_AppliedPTransform_Combine-InjectDefault_21))+(Flatten/Write/0)
INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running
(((((Flatten/Read)+(ref_AppliedPTransform_assert_that-WindowInto-WindowIntoFn-_43))+(ref_AppliedPTransform_assert_that-ToVoidKey_44))+(ref_AppliedPTransform_assert_that-Group-CoGroupByKeyImpl-Tag-1-_48))+(assert_that/Group/CoGroupByKeyImpl/Flatten/Transcode/1))+(assert_that/Group/CoGroupByKeyImpl/Flatten/Write/1)
INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running
(assert_that/Group/CoGroupByKeyImpl/Flatten/Read)+(assert_that/Group/CoGroupByKeyImpl/GroupByKey/Write)
INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running
((((assert_that/Group/CoGroupByKeyImpl/GroupByKey/Read)+(ref_AppliedPTransform_assert_that-Group-CoGroupByKeyImpl-MapTuple-collect_values-_51))+(ref_AppliedPTransform_assert_that-Group-RestoreTags_52))+(ref_AppliedPTransform_assert_that-Unkey_53))+(ref_AppliedPTransform_assert_that-Match_54)
INFO:__main__:Reading documents from mongodb finished in 5.197 seconds
INFO:root:Missing pipeline option (runner). Executing pipeline using the
default runner: DirectRunner.
INFO:__main__:================================================================================
INFO:__main__:Reading from mongodb
beam_mongodbio_it_db:integration_test_1637108246
INFO:__main__:reader params : {'filter': {'number_mod_3': 0}, 'projection':
['number']}
INFO:__main__:expected results: {'number_sum': 1666683333, 'docs_count': 33334}
WARNING:root:Make sure that locally built Python SDK docker image has Python
3.7 interpreter.
INFO:root:Default Python SDK image for environment is
apache/beam_python3.7_sdk:2.35.0.dev
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function annotate_downstream_side_inputs at 0x7f28de906560>
====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function fix_side_input_pcoll_coders at 0x7f28de906680> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function pack_combiners at 0x7f28de906b90> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function lift_combiners at 0x7f28de906c20> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function expand_sdf at 0x7f28de906dd0> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function expand_gbk at 0x7f28de906e60> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function sink_flattens at 0x7f28de906f80> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function greedily_fuse at 0x7f28de907050> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function read_to_impulse at 0x7f28de9070e0> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function impulse_to_input at 0x7f28de907170> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function sort_stages at 0x7f28de9073b0> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function setup_timer_mapping at 0x7f28de907320> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function populate_data_channel_coders at 0x7f28de907440> ====================
INFO:apache_beam.runners.worker.statecache:Creating state cache with size 100
INFO:apache_beam.runners.portability.fn_api_runner.worker_handlers:Created
Worker handler
<apache_beam.runners.portability.fn_api_runner.worker_handlers.EmbeddedWorkerHandler
object at 0x7f28dc3f8490> for environment
ref_Environment_default_environment_1 (beam:env:embedded_python:v1, b'')
INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running
((((ref_AppliedPTransform_ReadFromMongoDB-Read-Impulse_4)+(ref_AppliedPTransform_ReadFromMongoDB-Read-Map-lambda-at-iobase-py-898-_5))+(ReadFromMongoDB/Read/SDFBoundedSourceReader/ParDo(SDFBoundedSourceDoFn)/PairWithRestriction))+(ReadFromMongoDB/Read/SDFBoundedSourceReader/ParDo(SDFBoundedSourceDoFn)/SplitAndSizeRestriction))+(ref_PCollection_PCollection_2_split/Write)
INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running
((((((((ref_PCollection_PCollection_2_split/Read)+(ReadFromMongoDB/Read/SDFBoundedSourceReader/ParDo(SDFBoundedSourceDoFn)/Process))+(ref_AppliedPTransform_Map_8))+(ref_AppliedPTransform_Combine-KeyWithVoid_10))+(ref_AppliedPTransform_Count-CombineGlobally-CountCombineFn-KeyWithVoid_24))+(Combine/CombinePerKey/Precombine))+(Combine/CombinePerKey/Group/Write))+(Count/CombineGlobally(CountCombineFn)/CombinePerKey/Precombine))+(Count/CombineGlobally(CountCombineFn)/CombinePerKey/Group/Write)
INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running
((((Count/CombineGlobally(CountCombineFn)/CombinePerKey/Group/Read)+(Count/CombineGlobally(CountCombineFn)/CombinePerKey/Merge))+(Count/CombineGlobally(CountCombineFn)/CombinePerKey/ExtractOutputs))+(ref_AppliedPTransform_Count-CombineGlobally-CountCombineFn-UnKey_29))+(ref_PCollection_PCollection_16/Write)
INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running
(((((ref_AppliedPTransform_assert_that-Create-Impulse_39)+(ref_AppliedPTransform_assert_that-Create-FlatMap-lambda-at-core-py-3224-_40))+(ref_AppliedPTransform_assert_that-Create-Map-decode-_42))+(ref_AppliedPTransform_assert_that-Group-CoGroupByKeyImpl-Tag-0-_47))+(assert_that/Group/CoGroupByKeyImpl/Flatten/Transcode/0))+(assert_that/Group/CoGroupByKeyImpl/Flatten/Write/0)
INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running
((((Combine/CombinePerKey/Group/Read)+(Combine/CombinePerKey/Merge))+(Combine/CombinePerKey/ExtractOutputs))+(ref_AppliedPTransform_Combine-UnKey_15))+(ref_PCollection_PCollection_8/Write)
INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running
(((((ref_AppliedPTransform_Count-CombineGlobally-CountCombineFn-DoOnce-Impulse_31)+(ref_AppliedPTransform_Count-CombineGlobally-CountCombineFn-DoOnce-FlatMap-lambda-at-core-py-3224-_32))+(ref_AppliedPTransform_Count-CombineGlobally-CountCombineFn-DoOnce-Map-decode-_34))+(ref_AppliedPTransform_Count-CombineGlobally-CountCombineFn-InjectDefault_35))+(Flatten/Transcode/1))+(Flatten/Write/1)
INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running
((((ref_AppliedPTransform_Combine-DoOnce-Impulse_17)+(ref_AppliedPTransform_Combine-DoOnce-FlatMap-lambda-at-core-py-3224-_18))+(ref_AppliedPTransform_Combine-DoOnce-Map-decode-_20))+(ref_AppliedPTransform_Combine-InjectDefault_21))+(Flatten/Write/0)
INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running
(((((Flatten/Read)+(ref_AppliedPTransform_assert_that-WindowInto-WindowIntoFn-_43))+(ref_AppliedPTransform_assert_that-ToVoidKey_44))+(ref_AppliedPTransform_assert_that-Group-CoGroupByKeyImpl-Tag-1-_48))+(assert_that/Group/CoGroupByKeyImpl/Flatten/Transcode/1))+(assert_that/Group/CoGroupByKeyImpl/Flatten/Write/1)
INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running
(assert_that/Group/CoGroupByKeyImpl/Flatten/Read)+(assert_that/Group/CoGroupByKeyImpl/GroupByKey/Write)
INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running
((((assert_that/Group/CoGroupByKeyImpl/GroupByKey/Read)+(ref_AppliedPTransform_assert_that-Group-CoGroupByKeyImpl-MapTuple-collect_values-_51))+(ref_AppliedPTransform_assert_that-Group-RestoreTags_52))+(ref_AppliedPTransform_assert_that-Unkey_53))+(ref_AppliedPTransform_assert_that-Match_54)
INFO:__main__:Reading documents from mongodb finished in 2.644 seconds
INFO:root:Missing pipeline option (runner). Executing pipeline using the
default runner: DirectRunner.
INFO:__main__:================================================================================
INFO:__main__:Reading from mongodb
beam_mongodbio_it_db:integration_test_1637108246
INFO:__main__:reader params : {'projection': ['number'], 'bucket_auto': True}
INFO:__main__:expected results: {'number_sum': 4999950000, 'docs_count': 100000}
WARNING:root:Make sure that locally built Python SDK docker image has Python
3.7 interpreter.
INFO:root:Default Python SDK image for environment is
apache/beam_python3.7_sdk:2.35.0.dev
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function annotate_downstream_side_inputs at 0x7f28de906560>
====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function fix_side_input_pcoll_coders at 0x7f28de906680> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function pack_combiners at 0x7f28de906b90> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function lift_combiners at 0x7f28de906c20> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function expand_sdf at 0x7f28de906dd0> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function expand_gbk at 0x7f28de906e60> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function sink_flattens at 0x7f28de906f80> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function greedily_fuse at 0x7f28de907050> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function read_to_impulse at 0x7f28de9070e0> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function impulse_to_input at 0x7f28de907170> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function sort_stages at 0x7f28de9073b0> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function setup_timer_mapping at 0x7f28de907320> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function populate_data_channel_coders at 0x7f28de907440> ====================
INFO:apache_beam.runners.worker.statecache:Creating state cache with size 100
INFO:apache_beam.runners.portability.fn_api_runner.worker_handlers:Created
Worker handler
<apache_beam.runners.portability.fn_api_runner.worker_handlers.EmbeddedWorkerHandler
object at 0x7f28dd2adf90> for environment
ref_Environment_default_environment_1 (beam:env:embedded_python:v1, b'')
INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running
((((ref_AppliedPTransform_ReadFromMongoDB-Read-Impulse_4)+(ref_AppliedPTransform_ReadFromMongoDB-Read-Map-lambda-at-iobase-py-898-_5))+(ReadFromMongoDB/Read/SDFBoundedSourceReader/ParDo(SDFBoundedSourceDoFn)/PairWithRestriction))+(ReadFromMongoDB/Read/SDFBoundedSourceReader/ParDo(SDFBoundedSourceDoFn)/SplitAndSizeRestriction))+(ref_PCollection_PCollection_2_split/Write)
INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running
((((((((ref_PCollection_PCollection_2_split/Read)+(ReadFromMongoDB/Read/SDFBoundedSourceReader/ParDo(SDFBoundedSourceDoFn)/Process))+(ref_AppliedPTransform_Map_8))+(ref_AppliedPTransform_Combine-KeyWithVoid_10))+(ref_AppliedPTransform_Count-CombineGlobally-CountCombineFn-KeyWithVoid_24))+(Combine/CombinePerKey/Precombine))+(Combine/CombinePerKey/Group/Write))+(Count/CombineGlobally(CountCombineFn)/CombinePerKey/Precombine))+(Count/CombineGlobally(CountCombineFn)/CombinePerKey/Group/Write)
INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running
((((Count/CombineGlobally(CountCombineFn)/CombinePerKey/Group/Read)+(Count/CombineGlobally(CountCombineFn)/CombinePerKey/Merge))+(Count/CombineGlobally(CountCombineFn)/CombinePerKey/ExtractOutputs))+(ref_AppliedPTransform_Count-CombineGlobally-CountCombineFn-UnKey_29))+(ref_PCollection_PCollection_16/Write)
INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running
(((((ref_AppliedPTransform_Count-CombineGlobally-CountCombineFn-DoOnce-Impulse_31)+(ref_AppliedPTransform_Count-CombineGlobally-CountCombineFn-DoOnce-FlatMap-lambda-at-core-py-3224-_32))+(ref_AppliedPTransform_Count-CombineGlobally-CountCombineFn-DoOnce-Map-decode-_34))+(ref_AppliedPTransform_Count-CombineGlobally-CountCombineFn-InjectDefault_35))+(Flatten/Transcode/1))+(Flatten/Write/1)
INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running
((((Combine/CombinePerKey/Group/Read)+(Combine/CombinePerKey/Merge))+(Combine/CombinePerKey/ExtractOutputs))+(ref_AppliedPTransform_Combine-UnKey_15))+(ref_PCollection_PCollection_8/Write)
INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running
((((ref_AppliedPTransform_Combine-DoOnce-Impulse_17)+(ref_AppliedPTransform_Combine-DoOnce-FlatMap-lambda-at-core-py-3224-_18))+(ref_AppliedPTransform_Combine-DoOnce-Map-decode-_20))+(ref_AppliedPTransform_Combine-InjectDefault_21))+(Flatten/Write/0)
INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running
(((((Flatten/Read)+(ref_AppliedPTransform_assert_that-WindowInto-WindowIntoFn-_43))+(ref_AppliedPTransform_assert_that-ToVoidKey_44))+(ref_AppliedPTransform_assert_that-Group-CoGroupByKeyImpl-Tag-1-_48))+(assert_that/Group/CoGroupByKeyImpl/Flatten/Transcode/1))+(assert_that/Group/CoGroupByKeyImpl/Flatten/Write/1)
INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running
(((((ref_AppliedPTransform_assert_that-Create-Impulse_39)+(ref_AppliedPTransform_assert_that-Create-FlatMap-lambda-at-core-py-3224-_40))+(ref_AppliedPTransform_assert_that-Create-Map-decode-_42))+(ref_AppliedPTransform_assert_that-Group-CoGroupByKeyImpl-Tag-0-_47))+(assert_that/Group/CoGroupByKeyImpl/Flatten/Transcode/0))+(assert_that/Group/CoGroupByKeyImpl/Flatten/Write/0)
INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running
(assert_that/Group/CoGroupByKeyImpl/Flatten/Read)+(assert_that/Group/CoGroupByKeyImpl/GroupByKey/Write)
INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running
((((assert_that/Group/CoGroupByKeyImpl/GroupByKey/Read)+(ref_AppliedPTransform_assert_that-Group-CoGroupByKeyImpl-MapTuple-collect_values-_51))+(ref_AppliedPTransform_assert_that-Group-RestoreTags_52))+(ref_AppliedPTransform_assert_that-Unkey_53))+(ref_AppliedPTransform_assert_that-Match_54)
INFO:__main__:Reading documents from mongodb finished in 5.206 seconds
INFO:root:Missing pipeline option (runner). Executing pipeline using the
default runner: DirectRunner.
INFO:__main__:================================================================================
INFO:__main__:Reading from mongodb
beam_mongodbio_it_db:integration_test_1637108246
INFO:__main__:reader params : {'filter': {'number_mod_3': 0}, 'projection':
['number'], 'bucket_auto': True}
INFO:__main__:expected results: {'number_sum': 1666683333, 'docs_count': 33334}
WARNING:root:Make sure that locally built Python SDK docker image has Python
3.7 interpreter.
INFO:root:Default Python SDK image for environment is
apache/beam_python3.7_sdk:2.35.0.dev
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function annotate_downstream_side_inputs at 0x7f28de906560>
====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function fix_side_input_pcoll_coders at 0x7f28de906680> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function pack_combiners at 0x7f28de906b90> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function lift_combiners at 0x7f28de906c20> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function expand_sdf at 0x7f28de906dd0> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function expand_gbk at 0x7f28de906e60> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function sink_flattens at 0x7f28de906f80> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function greedily_fuse at 0x7f28de907050> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function read_to_impulse at 0x7f28de9070e0> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function impulse_to_input at 0x7f28de907170> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function sort_stages at 0x7f28de9073b0> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function setup_timer_mapping at 0x7f28de907320> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function populate_data_channel_coders at 0x7f28de907440> ====================
INFO:apache_beam.runners.worker.statecache:Creating state cache with size 100
INFO:apache_beam.runners.portability.fn_api_runner.worker_handlers:Created
Worker handler
<apache_beam.runners.portability.fn_api_runner.worker_handlers.EmbeddedWorkerHandler
object at 0x7f28de515a50> for environment
ref_Environment_default_environment_1 (beam:env:embedded_python:v1, b'')
INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running
(((((ref_AppliedPTransform_assert_that-Create-Impulse_39)+(ref_AppliedPTransform_assert_that-Create-FlatMap-lambda-at-core-py-3224-_40))+(ref_AppliedPTransform_assert_that-Create-Map-decode-_42))+(ref_AppliedPTransform_assert_that-Group-CoGroupByKeyImpl-Tag-0-_47))+(assert_that/Group/CoGroupByKeyImpl/Flatten/Transcode/0))+(assert_that/Group/CoGroupByKeyImpl/Flatten/Write/0)
INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running
((((ref_AppliedPTransform_ReadFromMongoDB-Read-Impulse_4)+(ref_AppliedPTransform_ReadFromMongoDB-Read-Map-lambda-at-iobase-py-898-_5))+(ReadFromMongoDB/Read/SDFBoundedSourceReader/ParDo(SDFBoundedSourceDoFn)/PairWithRestriction))+(ReadFromMongoDB/Read/SDFBoundedSourceReader/ParDo(SDFBoundedSourceDoFn)/SplitAndSizeRestriction))+(ref_PCollection_PCollection_2_split/Write)
INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running
((((((((ref_PCollection_PCollection_2_split/Read)+(ReadFromMongoDB/Read/SDFBoundedSourceReader/ParDo(SDFBoundedSourceDoFn)/Process))+(ref_AppliedPTransform_Map_8))+(ref_AppliedPTransform_Combine-KeyWithVoid_10))+(ref_AppliedPTransform_Count-CombineGlobally-CountCombineFn-KeyWithVoid_24))+(Combine/CombinePerKey/Precombine))+(Combine/CombinePerKey/Group/Write))+(Count/CombineGlobally(CountCombineFn)/CombinePerKey/Precombine))+(Count/CombineGlobally(CountCombineFn)/CombinePerKey/Group/Write)
INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running
((((Combine/CombinePerKey/Group/Read)+(Combine/CombinePerKey/Merge))+(Combine/CombinePerKey/ExtractOutputs))+(ref_AppliedPTransform_Combine-UnKey_15))+(ref_PCollection_PCollection_8/Write)
INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running
((((ref_AppliedPTransform_Combine-DoOnce-Impulse_17)+(ref_AppliedPTransform_Combine-DoOnce-FlatMap-lambda-at-core-py-3224-_18))+(ref_AppliedPTransform_Combine-DoOnce-Map-decode-_20))+(ref_AppliedPTransform_Combine-InjectDefault_21))+(Flatten/Write/0)
INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running
((((Count/CombineGlobally(CountCombineFn)/CombinePerKey/Group/Read)+(Count/CombineGlobally(CountCombineFn)/CombinePerKey/Merge))+(Count/CombineGlobally(CountCombineFn)/CombinePerKey/ExtractOutputs))+(ref_AppliedPTransform_Count-CombineGlobally-CountCombineFn-UnKey_29))+(ref_PCollection_PCollection_16/Write)
INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running
(((((ref_AppliedPTransform_Count-CombineGlobally-CountCombineFn-DoOnce-Impulse_31)+(ref_AppliedPTransform_Count-CombineGlobally-CountCombineFn-DoOnce-FlatMap-lambda-at-core-py-3224-_32))+(ref_AppliedPTransform_Count-CombineGlobally-CountCombineFn-DoOnce-Map-decode-_34))+(ref_AppliedPTransform_Count-CombineGlobally-CountCombineFn-InjectDefault_35))+(Flatten/Transcode/1))+(Flatten/Write/1)
INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running
(((((Flatten/Read)+(ref_AppliedPTransform_assert_that-WindowInto-WindowIntoFn-_43))+(ref_AppliedPTransform_assert_that-ToVoidKey_44))+(ref_AppliedPTransform_assert_that-Group-CoGroupByKeyImpl-Tag-1-_48))+(assert_that/Group/CoGroupByKeyImpl/Flatten/Transcode/1))+(assert_that/Group/CoGroupByKeyImpl/Flatten/Write/1)
INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running
(assert_that/Group/CoGroupByKeyImpl/Flatten/Read)+(assert_that/Group/CoGroupByKeyImpl/GroupByKey/Write)
INFO:apache_beam.runners.portability.fn_api_runner.fn_runner:Running
((((assert_that/Group/CoGroupByKeyImpl/GroupByKey/Read)+(ref_AppliedPTransform_assert_that-Group-CoGroupByKeyImpl-MapTuple-collect_values-_51))+(ref_AppliedPTransform_assert_that-Group-RestoreTags_52))+(ref_AppliedPTransform_assert_that-Unkey_53))+(ref_AppliedPTransform_assert_that-Match_54)
INFO:__main__:Reading documents from mongodb finished in 2.716 seconds
> Task :sdks:python:test-suites:direct:py37:postCommitIT
> Task :sdks:python:test-suites:dataflow:py37:postCommitIT FAILED
FAILURE: Build completed with 2 failures.
1: Task failed with an exception.
-----------
* Where:
Script
'<https://ci-beam.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/test-suites/direct/common.gradle'>
line: 112
* What went wrong:
Execution failed for task ':sdks:python:test-suites:direct:py37:directRunnerIT'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug
option to get more log output. Run with --scan to get full insights.
==============================================================================
2: Task failed with an exception.
-----------
* Where:
Script
'<https://ci-beam.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/test-suites/dataflow/common.gradle'>
line: 120
* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py37:postCommitIT'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug
option to get more log output. Run with --scan to get full insights.
==============================================================================
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with
Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See
https://docs.gradle.org/6.9.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 1h 36m 34s
214 actionable tasks: 153 executed, 57 from cache, 4 up-to-date
Publishing build scan...
Publishing build scan failed due to network error
'java.net.SocketTimeoutException: Read timed out' (2 retries remaining)...
Publishing build scan failed due to network error
'java.net.SocketTimeoutException: Read timed out' (1 retry remaining)...
A network error occurred.
If you require assistance with this problem, please report it via
https://gradle.com/help/plugin and include the following information via
copy/paste.
----------
Gradle version: 6.9.1
Plugin version: 3.4.1
Request URL: https://scans-in.gradle.com/scans/publish/gradle/3.4.1/token
Request ID: 32cbf699-5ef2-453b-955e-47e9c9e5af21
Exception: java.net.SocketTimeoutException: Read timed out
----------
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
