Hyukjin Kwon created SPARK-61181:
------------------------------------
Summary: Read the local relation settings of the server for local
relations of 64 KiB or more, and raise LOCAL_RELATION_SIZE_LIMIT_EXCEEDED, in
the Rust and Python clients as PySpark does
Key: SPARK-61181
URL: https://issues.apache.org/jira/browse/SPARK-61181
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
PySpark Connect's createDataFrame reads
spark.sql.session.localRelationCacheThreshold and
spark.sql.session.localRelationSizeLimit for every local relation: at the
threshold it uploads the relation as cache artifacts, and past the size limit
it raises LOCAL_RELATION_SIZE_LIMIT_EXCEEDED.
The Rust client reads them only for a relation of at least 32 MiB of Arrow
data, and sends a smaller one inline. So, with both settings lowered, local
rows and some pandas frames (those of the float64 and string fast paths) under
32 MiB are sent without the error, while the same frame with a DDL schema goes
through the Arrow importer and raises it. For example, with a threshold of 1
byte and a size limit of 100 bytes, createDataFrame of 200 rows of (long,
string) succeeds, where PySpark raises LOCAL_RELATION_SIZE_LIMIT_EXCEEDED.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
