L. C. Hsieh created SPARK-58019:
-----------------------------------
Summary: Convert Arrow list columns to Python rows in bulk
Key: SPARK-58019
URL: https://issues.apache.org/jira/browse/SPARK-58019
Project: Spark
Issue Type: Improvement
Components: PySpark
Affects Versions: 5.0.0
Reporter: L. C. Hsieh
PyArrow's {{Array.to_pylist()}} materializes one Scalar per element; for
list-typed columns each row additionally allocates a C++ scalar, a Python
Scalar wrapper and a Python Array wrapper before converting elements one by
one. This makes Arrow-optimized Python UDF inputs and Spark Connect
{{collect()}} several times slower on array columns than necessary. This is the
root cause of the performance regression observed when Arrow-serialized Python
UDFs read array columns.
Upstream Arrow issue: https://github.com/apache/arrow/issues/50326 (fix
proposed in https://github.com/apache/arrow/pull/50327), but that will only be
available in a future PyArrow release.
Until the minimum supported PyArrow version includes the fix, PySpark can
convert list columns in bulk itself using only public PyArrow APIs: convert the
flattened child values of a list column in a single pass, then slice the
resulting Python list per row using the offsets and validity bitmap. Leaf
values are still converted by Arrow's own {{to_pylist}}, so results are exactly
identical (unlike a pandas round trip, which coerces {{[1, None, 3]}} in an int
list to floats/NaN).
ASV microbenchmark (1M rows): {{list<string>}} 769ms -> 507ms (1.5x);
{{list<list<int32>>}} with nulls 1.86s -> 537ms (3.5x); peak memory unchanged.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
