Hyukjin Kwon created SPARK-62012:
------------------------------------
Summary: Send local rows and pandas string columns of PySpark's
large Arrow types under spark.sql.execution.arrow.useLargeVarTypes, and compare
the cache threshold with pyarrow's nbytes
Key: SPARK-62012
URL: https://issues.apache.org/jira/browse/SPARK-62012
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: 5.0.0
Reporter: Hyukjin Kwon
Problem
PySpark 4.3.0's createDataFrame reads
spark.sql.execution.arrow.useLargeVarTypes and, when it is
"true", builds its Arrow table with large types:
- the strings and binaries of the schema, also in arrays, maps, structs and the
SQL type of a
user-defined type, are large_string and large_binary
(LocalDataToArrowConversion.convert, conversion.py:994-1005; to_arrow_type,
pandas/types.py:137-139);
- the binaries of a variant, a geometry and a geography stay binary.
It then compares `_table.nbytes` with
spark.sql.session.localRelationCacheThreshold to choose
between an inline and a cached relation (connect/session.py:782). pyarrow's
nbytes counts `len`
offsets of a string, binary, list or map array, not the `len + 1` of its buffer.
The drop-in and the Rust API read the setting, but two paths ignored it:
- local rows: the core's Arrow conversion always made Utf8 and Binary
(crates/spark-connect/src/session.rs, datatype_to_arrow);
- the binding's IPC of pandas object-string columns
(crates/pyspark-rs/src/pandas_input.rs,
_pandas_string_ipc), which writes Utf8.
The size that every path compared with the threshold was arrow-rs's
get_slice_memory_size, which
counts `len + 1` offsets: 4 bytes more per string column than PySpark's measure
(crates/spark-connect/src/arrow_input.rs, batch_data_size).
Repro
spark.conf.set("spark.sql.session.localRelationSizeLimit", "1000")
spark.conf.set("spark.sql.session.localRelationCacheThreshold", "100000")
spark.conf.set("spark.sql.execution.arrow.useLargeVarTypes", "true")
schema = StructType([StructField("s", StringType())])
spark.createDataFrame([("abcd",)] * 10000, schema).count()
spark.createDataFrame(pd.DataFrame({"s": ["abcd"] * 10000}), schema).count()
spark.conf.unset("spark.sql.execution.arrow.useLargeVarTypes")
spark.conf.set("spark.sql.session.localRelationCacheThreshold", "8001")
spark.createDataFrame([("abcd",)] * 1000, schema).count()
Actual (drop-in):
- the first two: 10000, sent inline, as strings of 80000 bytes;
- the third: AnalysisException [LOCAL_RELATION_SIZE_LIMIT_EXCEEDED], as it
measured 8004
bytes.
Expected (pyspark-client 4.3.0):
- the first two: AnalysisException [LOCAL_RELATION_SIZE_LIMIT_EXCEEDED], as
large strings of
120000 bytes are cached;
- the third: 1000, inline, as its table is 8000 bytes.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]