[
https://issues.apache.org/jira/browse/SPARK-61181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61181:
---------------------------------
Description:
PySpark Connect's createDataFrame reads
spark.sql.session.localRelationCacheThreshold and
spark.sql.session.localRelationSizeLimit for every local relation: at the
threshold it uploads the relation as cache artifacts, and past the size limit
it raises LOCAL_RELATION_SIZE_LIMIT_EXCEEDED.
The Rust client reads them only for a relation of at least 32 MiB of Arrow
data, and sends a smaller one inline. So, with both settings lowered, local
rows and some pandas frames (those of the float64 and string fast paths) under
32 MiB are sent without the error, while the same frame with a DDL schema goes
through the Arrow importer and raises it. For example, with a threshold of 1
byte and a size limit of 100 bytes, createDataFrame of 2,000 rows of a long and
a 100-character string (about 225 KB) succeeds, where PySpark raises
LOCAL_RELATION_SIZE_LIMIT_EXCEEDED. With the default settings too, a relation
of 1 MiB (the default threshold) to 32 MiB is sent inline, where PySpark
uploads it as cache artifacts.
was:
PySpark Connect's createDataFrame reads
spark.sql.session.localRelationCacheThreshold and
spark.sql.session.localRelationSizeLimit for every local relation: at the
threshold it uploads the relation as cache artifacts, and past the size limit
it raises LOCAL_RELATION_SIZE_LIMIT_EXCEEDED.
The Rust client reads them only for a relation of at least 32 MiB of Arrow
data, and sends a smaller one inline. So, with both settings lowered, local
rows and some pandas frames (those of the float64 and string fast paths) under
32 MiB are sent without the error, while the same frame with a DDL schema goes
through the Arrow importer and raises it. For example, with a threshold of 1
byte and a size limit of 100 bytes, createDataFrame of 200 rows of (long,
string) succeeds, where PySpark raises LOCAL_RELATION_SIZE_LIMIT_EXCEEDED.
> Read the local relation settings of the server for local relations of 64 KiB
> or more, and raise LOCAL_RELATION_SIZE_LIMIT_EXCEEDED, in the Rust and Python
> clients as PySpark does
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61181
> URL: https://issues.apache.org/jira/browse/SPARK-61181
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> PySpark Connect's createDataFrame reads
> spark.sql.session.localRelationCacheThreshold and
> spark.sql.session.localRelationSizeLimit for every local relation: at the
> threshold it uploads the relation as cache artifacts, and past the size limit
> it raises LOCAL_RELATION_SIZE_LIMIT_EXCEEDED.
> The Rust client reads them only for a relation of at least 32 MiB of Arrow
> data, and sends a smaller one inline. So, with both settings lowered, local
> rows and some pandas frames (those of the float64 and string fast paths)
> under 32 MiB are sent without the error, while the same frame with a DDL
> schema goes through the Arrow importer and raises it. For example, with a
> threshold of 1 byte and a size limit of 100 bytes, createDataFrame of 2,000
> rows of a long and a 100-character string (about 225 KB) succeeds, where
> PySpark raises LOCAL_RELATION_SIZE_LIMIT_EXCEEDED. With the default settings
> too, a relation of 1 MiB (the default threshold) to 32 MiB is sent inline,
> where PySpark uploads it as cache artifacts.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
