Hyukjin Kwon created SPARK-59037:
------------------------------------
Summary: Spark Connect Rust client ignores gRPC max message length
(128 MiB parsed but never applied)
Key: SPARK-59037
URL: https://issues.apache.org/jira/browse/SPARK-59037
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: 5.0.0
Reporter: Hyukjin Kwon
The Rust Spark Connect client declares GRPC_MAX_MESSAGE_LENGTH_DEFAULT (128
MiB) in channel.rs but never applies it: no max_decoding_message_size /
max_encoding_message_size call exists on the stub or the raw tonic Grpc calls.
tonic therefore uses its 4 MiB default receive cap (codec/mod.rs), so any
collect() returning a >4 MiB Arrow batch fails with a decode error that the
reference (grpcio) client handles fine.
Same bug-class as SPARK-59032 (a documented channel option parsed but never
applied). Fix: apply max_decoding_message_size / max_encoding_message_size
(from the parsed grpc_max_message_size, default 128 MiB) to the
SparkConnectServiceClient stub and to the raw Grpc instances in
execute_plan_raw / reattach_execute_raw / analyze_plan_raw.
Found during review of PR #58
(https://github.com/apache/spark-connect-rust/pull/58) by viirya.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
