[ 
https://issues.apache.org/jira/browse/SPARK-62012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon resolved SPARK-62012.
----------------------------------
    Fix Version/s: connect-rust-4.4.0
       Resolution: Fixed

Issue resolved by pull request 855
[https://github.com/apache/spark-connect-rust/pull/855]

> Send local rows and pandas string columns of PySpark's large Arrow types 
> under spark.sql.execution.arrow.useLargeVarTypes, and compare the cache 
> threshold with pyarrow's nbytes
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-62012
>                 URL: https://issues.apache.org/jira/browse/SPARK-62012
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: 5.0.0
>            Reporter: Hyukjin Kwon
>            Assignee: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.4.0
>
>
> Problem
> PySpark 4.3.0's createDataFrame reads 
> spark.sql.execution.arrow.useLargeVarTypes and, when it is
> "true", builds its Arrow table with large types:
> - the strings and binaries of the schema, also in arrays, maps, structs and 
> the SQL type of a
>   user-defined type, are large_string and large_binary
>   (LocalDataToArrowConversion.convert, conversion.py:994-1005; to_arrow_type,
>   pandas/types.py:137-139);
> - the binaries of a variant, a geometry and a geography stay binary.
> It then compares `_table.nbytes` with 
> spark.sql.session.localRelationCacheThreshold to choose
> between an inline and a cached relation (connect/session.py:782). pyarrow's 
> nbytes counts `len`
> offsets of a string, binary, list or map array, not the `len + 1` of its 
> buffer.
> The drop-in and the Rust API read the setting, but two paths ignored it:
> - local rows: the core's Arrow conversion always made Utf8 and Binary
>   (crates/spark-connect/src/session.rs, datatype_to_arrow);
> - the binding's IPC of pandas object-string columns 
> (crates/pyspark-rs/src/pandas_input.rs,
>   _pandas_string_ipc), which writes Utf8.
> The size that every path compared with the threshold was arrow-rs's 
> get_slice_memory_size, which
> counts `len + 1` offsets: 4 bytes more per string column than PySpark's 
> measure
> (crates/spark-connect/src/arrow_input.rs, batch_data_size).
> Repro
>     spark.conf.set("spark.sql.session.localRelationSizeLimit", "1000")
>     spark.conf.set("spark.sql.session.localRelationCacheThreshold", "100000")
>     spark.conf.set("spark.sql.execution.arrow.useLargeVarTypes", "true")
>     schema = StructType([StructField("s", StringType())])
>     spark.createDataFrame([("abcd",)] * 10000, schema).count()
>     spark.createDataFrame(pd.DataFrame({"s": ["abcd"] * 10000}), 
> schema).count()
>     spark.conf.unset("spark.sql.execution.arrow.useLargeVarTypes")
>     spark.conf.set("spark.sql.session.localRelationCacheThreshold", "8001")
>     spark.createDataFrame([("abcd",)] * 1000, schema).count()
> Actual (drop-in):
> - the first two: 10000, sent inline, as strings of 80000 bytes;
> - the third: AnalysisException [LOCAL_RELATION_SIZE_LIMIT_EXCEEDED], as it 
> measured 8004
>   bytes.
> Expected (pyspark-client 4.3.0):
> - the first two: AnalysisException [LOCAL_RELATION_SIZE_LIMIT_EXCEEDED], as 
> large strings of
>   120000 bytes are cached;
> - the third: 1000, inline, as its table is 8000 bytes.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to