viirya opened a new pull request, #57104: URL: https://github.com/apache/spark/pull/57104
### What changes were proposed in this pull request? Follow-up of SPARK-58019 (#57099); **only the last commit is new — this PR is stacked on #57099 and will be rebased once it merges.** Extend `ArrowTableToRowsConversion._to_pylist` with fast paths for leaf (non-nested) columns: - **string/large_string/binary/large_binary** columns use Arrow's object-dtype NumPy conversion, which produces exactly `str`/`bytes`/`None` — no other type can come out of it. - **integer/float32/float64/boolean** columns without nulls convert via a zero-copy NumPy view (booleans are bit-packed, hence copied) and `ndarray.tolist()`, which materializes exact Python `int`/`float`/`bool`. With nulls, values are filled with a placeholder first (`pc.fill_null`) and nulls are restored to `None` from the validity bitmap, so ints are always materialized from the original int buffer, never through a float representation. Types whose `as_py` returns non-primitive objects (dates, timestamps, decimals, ...) keep using `to_pylist`. Since list columns convert their flattened child values through `_to_pylist`, list-typed columns get the leaf speedup on top of the SPARK-58019 bulk offsets slicing. ### Why are the changes needed? After SPARK-58019, leaf values are still converted one Scalar at a time by PyArrow's `to_pylist()` (apache/arrow#50326). ASV microbenchmark (`bench_arrow.ArrowLeafColumnToRowsBenchmark`, 1M rows, PyArrow 24.0.0): | case | `to_pylist()` | this PR | speedup | |---|---|---|---| | string, 10% nulls | 196 ms | 20 ms | 9.7x | | int64, 10% nulls | 99 ms | 28 ms | 3.6x | | float64, no nulls | 100 ms | 9 ms | 11x | End-to-end `list<string>` conversion (`ArrowListColumnToRowsBenchmark`, 1M rows) improves from 507 ms (SPARK-58019) to 118 ms. ### Does this PR introduce _any_ user-facing change? No. Only performance; conversion results are identical (covered by exact-type tests). ### How was this patch tested? Extended `ArrowColumnToPylistTests` with flat string/large_string/binary/large_binary, int8/int64/uint64 (including values beyond 2^62), float32/float64 (NaN/inf), bool, no-null and all-null variants, sliced and chunked views — all compared against `column.to_pylist()` with exact element-type assertions — plus non-primitive leaves (date/timestamp/decimal) exercising the fallback, and lists of fast-path leaves. New ASV benchmark class `ArrowLeafColumnToRowsBenchmark`. ### Was this patch authored or co-authored using generative AI tooling? Yes. This pull request and its description were written by Isaac (Claude Code). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
