viirya opened a new pull request, #57099: URL: https://github.com/apache/spark/pull/57099
### What changes were proposed in this pull request? Add `ArrowTableToRowsConversion._to_pylist`, which converts Arrow list-typed columns to Python values in bulk: the flattened child values are converted with a single recursive `to_pylist` call, and the resulting Python list is sliced per row using the offsets and the validity bitmap. It is used in the Arrow-to-rows conversion paths (Spark Connect `collect()`, Arrow batch UDF inputs, Arrow UDTF inputs). Non-list columns, map columns and environments without NumPy fall back to plain `column.to_pylist()`. Leaf values are still converted by Arrow's own `to_pylist`, so results are exactly identical to `column.to_pylist()`: `None` stays `None` and values inside numeric lists stay Python ints. NumPy is used only for the offsets (non-null integers) and the validity bitmap (booleans), never for the values, so the type coercion problems of a pandas round trip (`list<int32>` of `[1, None, 3]` becoming `[1.0, nan, 3.0]`) cannot occur. This is an interim measure for a PyArrow-side inefficiency: `Array.to_pylist()` materializes one Scalar per element, and for list types each row additionally allocates a C++ scalar, a Python Scalar wrapper and a Python Array wrapper before converting elements one by one (root-caused in apache/arrow#50326, fix proposed in apache/arrow#50327). The helper documents this and can be removed once the minimum supported PyArrow version includes the upstream fix. ### Why are the changes needed? Converting Arrow list columns to Python rows is the hot path of Arrow-optimized Python UDF inputs and Spark Connect `collect()`. With plain `to_pylist()` it is several times slower than necessary, which caused a performance regression on array columns when Arrow serialization became the default for regular Python UDFs. ASV microbenchmark (`python/benchmarks/bench_arrow.py::ArrowListColumnToRowsBenchmark`, added in this PR; 1M rows, macOS arm64, PyArrow 24.0.0): | case | `to_pylist()` | this PR | speedup | |---|---|---|---| | `list<string>` | 769 ms | 507 ms | 1.5x | | `list<list<int32>>` with nulls | 1.86 s | 537 ms | 3.5x | Peak memory is unchanged. ### Does this PR introduce _any_ user-facing change? No. Only performance; conversion results are byte-identical (covered by exact-type tests). ### How was this patch tested? New `ArrowColumnToPylistTests` in `python/pyspark/sql/tests/test_conversion.py`, comparing `_to_pylist` against `column.to_pylist()` with exact element-type assertions across list/large_list/nested/struct/map/fixed-size-list columns, sliced and chunked variants, plus a dedicated test that `list<int32>` of `[1, None, 3]` stays ints, and an end-to-end `ArrowTableToRowsConversion.convert` test with array columns. Full `test_conversion.py` passes. The new ASV benchmark class parametrizes `baseline` vs `bulk`, so the comparison above is reproducible with `./python/asv run --python=same --quick -b 'bench_arrow.ArrowListColumnToRowsBenchmark'`. ### Was this patch authored or co-authored using generative AI tooling? Yes. This pull request and its description were written by Isaac (Claude Code). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
