[
https://issues.apache.org/jira/browse/SPARK-59037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-59037.
----------------------------------
Fix Version/s: connect-rust-4.2.0
Resolution: Fixed
Issue resolved by pull request 64
[https://github.com/apache/spark-connect-rust/pull/64]
> Spark Connect Rust client ignores gRPC max message length (128 MiB parsed but
> never applied)
> --------------------------------------------------------------------------------------------
>
> Key: SPARK-59037
> URL: https://issues.apache.org/jira/browse/SPARK-59037
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: 5.0.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.2.0
>
>
> The Rust Spark Connect client declares GRPC_MAX_MESSAGE_LENGTH_DEFAULT (128
> MiB) in channel.rs but never applies it: no max_decoding_message_size /
> max_encoding_message_size call exists on the stub or the raw tonic Grpc
> calls. tonic therefore uses its 4 MiB default receive cap (codec/mod.rs), so
> any collect() returning a >4 MiB Arrow batch fails with a decode error that
> the reference (grpcio) client handles fine.
> Same bug-class as SPARK-59032 (a documented channel option parsed but never
> applied). Fix: apply max_decoding_message_size / max_encoding_message_size
> (from the parsed grpc_max_message_size, default 128 MiB) to the
> SparkConnectServiceClient stub and to the raw Grpc instances in
> execute_plan_raw / reattach_execute_raw / analyze_plan_raw.
> Found during review of PR #58
> (https://github.com/apache/spark-connect-rust/pull/58) by viirya.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
