L. C. Hsieh created SPARK-58023:
-----------------------------------
Summary: Fast paths for converting Arrow string, binary, numeric
and boolean columns to Python rows
Key: SPARK-58023
URL: https://issues.apache.org/jira/browse/SPARK-58023
Project: Spark
Issue Type: Improvement
Components: PySpark
Affects Versions: 5.0.0
Reporter: L. C. Hsieh
Follow-up of SPARK-58019. {{ArrowTableToRowsConversion._to_pylist}} still
converts leaf (non-nested) values one Scalar at a time via {{to_pylist()}}. Add
fast paths that keep exact {{as_py}} semantics:
* string/large_string/binary/large_binary: Arrow's object-dtype NumPy
conversion produces exactly {{str}}/{{bytes}}/{{None}}.
* integer/float32/float64/boolean: zero-copy NumPy view + {{ndarray.tolist}};
with nulls, fill with a placeholder ({{pc.fill_null}}) and restore {{None}}
from the validity bitmap, so ints are materialized from the int buffer and
never coerced through floats.
Types whose {{as_py}} returns non-primitive objects (dates, timestamps,
decimals, ...) keep using {{to_pylist}}.
ASV microbenchmark (1M rows): string with nulls 196ms -> 20ms (9.7x); int64
with nulls 99ms -> 28ms (3.6x); float64 without nulls 100ms -> 9ms (11x).
End-to-end {{list<string>}} conversion improves from 507ms to 118ms on top of
SPARK-58019.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
