[
https://issues.apache.org/jira/browse/BEAM-10334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Keshav updated BEAM-10334:
--------------------------
Description:
The default limit of a grpc message is set low to 4 MB, and is not overridden
in the sdk_worker.py for the _control_channel. This was first discovered when
submitting a beam job via a [flink operator orchestrated by
kubernetes|[https://github.com/GoogleCloudPlatform/flink-on-k8s-operator]]. The
below error (for apache-beam 2.18.0) was seen.
{code:java}
2020/06/24 18:37:28 Executing: python -m
apache_beam.runners.worker.sdk_worker_main Traceback (most recent call last):
File "/usr/local/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec) File "/usr/local/lib/python3.7/runpy.py", line 85, in
_run_code exec(code, run_globals) File
"/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker_main.py",
line 221, in <module> main(sys.argv) File
"/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker_main.py",
line 155, in main sdk_pipeline_options.view_as(ProfilingOptions)) File
"/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
line 117, in run for work_request in control_stub.Control(get_responses()):
File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 416, in
__next__ return self._next() File
"/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 706, in _next
raise self grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of
RPC that terminated with: status = StatusCode.RESOURCE_EXHAUSTED details =
"Received message larger than max (4847139 vs. 4194304)" debug_error_string =
"{"created":"@1593023849.325990420","description":"Received message larger than
max (4847139 vs.
4194304)","file":"src/core/ext/filters/message_size/message_size_filter.cc","file_line":188,"grpc_status":8}"
{code}
This can be fixed by modifying the options parameter to allow any size grpc
message to be sent or received.
was:
The default limit of a grpc message is set low to 4 MB, and is not overridden
in the sdk_worker.py for the _control_channel. This was first discovered when
submitting a beam job via a [flink operator orchestrated by
kubernetes|[https://github.com/GoogleCloudPlatform/flink-on-k8s-operator]]. The
below error (for apache-beam 2.18.0) was seen.
{code:java}
2020/06/24 18:37:28 Executing: python -m
apache_beam.runners.worker.sdk_worker_main Traceback (most recent call last):
File "/usr/local/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec) File "/usr/local/lib/python3.7/runpy.py", line 85, in
_run_code exec(code, run_globals) File
"/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker_main.py",
line 221, in <module> main(sys.argv) File
"/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker_main.py",
line 155, in main sdk_pipeline_options.view_as(ProfilingOptions)) File
"/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
line 117, in run for work_request in control_stub.Control(get_responses()):
File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 416, in
__next__ return self._next() File
"/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 706, in _next
raise self grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of
RPC that terminated with: status = StatusCode.RESOURCE_EXHAUSTED details =
"Received message larger than max (4847139 vs. 4194304)" debug_error_string =
"{"created":"@1593023849.325990420","description":"Received message larger than
max (4847139 vs.
4194304)","file":"src/core/ext/filters/message_size/message_size_filter.cc","file_line":188,"grpc_status":8}"
{code}
This can be easily fixed by modifying the options parameter to allow any size
grpc message to be sent or received.
> Message limit for grpc message is set to default 4 MB
> -----------------------------------------------------
>
> Key: BEAM-10334
> URL: https://issues.apache.org/jira/browse/BEAM-10334
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-harness
> Affects Versions: 2.18.0, 2.22.0
> Reporter: Keshav
> Priority: P2
>
> The default limit of a grpc message is set low to 4 MB, and is not overridden
> in the sdk_worker.py for the _control_channel. This was first discovered when
> submitting a beam job via a [flink operator orchestrated by
> kubernetes|[https://github.com/GoogleCloudPlatform/flink-on-k8s-operator]].
> The below error (for apache-beam 2.18.0) was seen.
>
> {code:java}
> 2020/06/24 18:37:28 Executing: python -m
> apache_beam.runners.worker.sdk_worker_main Traceback (most recent call last):
> File "/usr/local/lib/python3.7/runpy.py", line 193, in _run_module_as_main
> "__main__", mod_spec) File "/usr/local/lib/python3.7/runpy.py", line 85, in
> _run_code exec(code, run_globals) File
> "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker_main.py",
> line 221, in <module> main(sys.argv) File
> "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker_main.py",
> line 155, in main sdk_pipeline_options.view_as(ProfilingOptions)) File
> "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
> line 117, in run for work_request in control_stub.Control(get_responses()):
> File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 416, in
> __next__ return self._next() File
> "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 706, in _next
> raise self grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous
> of RPC that terminated with: status = StatusCode.RESOURCE_EXHAUSTED details =
> "Received message larger than max (4847139 vs. 4194304)" debug_error_string =
> "{"created":"@1593023849.325990420","description":"Received message larger
> than max (4847139 vs.
> 4194304)","file":"src/core/ext/filters/message_size/message_size_filter.cc","file_line":188,"grpc_status":8}"
> {code}
>
> This can be fixed by modifying the options parameter to allow any size grpc
> message to be sent or received.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
