Tim Meehan created SPARK-61262:
----------------------------------
Summary: Allow Arrow batch chunking in ExecutePlan, so that a
value over 128 MB can be collected, as PySpark does
Key: SPARK-61262
URL: https://issues.apache.org/jira/browse/SPARK-61262
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Tim Meehan
PySpark's Connect client asks the server to split large Arrow batches, so it
can collect a value bigger than the 128 MB message limit. The Rust client does
not, and fails.
{code:python}
from pyspark.sql import SparkSession, functions as F
spark = SparkSession.builder.remote("sc://localhost:15002").getOrCreate()
spark.range(1).select(F.repeat(F.lit("x"), 140 * 1024 * 1024)).collect()
{code}
Against an OSS Spark 4.3.0 Connect server on loopback:
* pyspark-client 4.3.0 returns the row. Its ExecutePlan requests carry
request_options { result_chunking_options { allow_arrow_batch_chunking: true }
}, so the server splits the batch.
* spark-connect-rust fails with "decoded message length too large: found
146801134 bytes, the limit is: 134217728 bytes" (measured at 4ad36945). At
8768cc4 no code under crates/ sets result_chunking_options.
Both clients cap a message at 128 MB, and up to 100 MB both work.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
