Yicong-Huang opened a new pull request, #57316: URL: https://github.com/apache/spark/pull/57316
### What changes were proposed in this pull request? Follow-up to #253 (SPARK-58019). That PR added `ArrowTableToRowsConversion._to_pylist`, a drop-in, byte-identical replacement for `Array.to_pylist()` that converts list/struct/map columns in bulk (flatten the child values once, slice per row via the offset buffer and validity bitmap) instead of materializing one Arrow Scalar per element -- measured 1.5x-3.5x faster on array columns (apache/arrow#50326) -- and transparently falls back to `to_pylist()` when the installed PyArrow is >= 25.0.1, NumPy is absent, or the column is not list/map/struct. Two analogous Arrow-to-rows call sites were left out of that change and still used plain `to_pylist()`. This follow-up routes both through `_to_pylist`: the Spark Connect `DataFrame.foreachPartition` path, and the row-based Python data source write path (`batch_to_rows` in `write_into_data_source.py`). `plan_data_source_read.py` is intentionally left unchanged: it converts a single `BinaryType` cell (`num_columns == 1, num_rows == 1`, a pickled `InputPartition`), where the bulk path has zero benefit and binary falls through to `to_pylist()` regardless. ### Why are the changes needed? Both changed sites carry arbitrary user-defined schemas that realistically include array/struct/map columns, so they now get the same speedup as the main PR when eligible, and are otherwise identical to before (the conversion falls back to `to_pylist()` for scalar-only columns or newer PyArrow, so it can never regress correctness or performance). ### Does this PR introduce any user-facing change? No. `_to_pylist` is byte-identical to `to_pylist()`. ### How was this patch tested? No behavior change; the byte-identical contract of `_to_pylist` is covered by the existing `test_matches_to_pylist` in `python/pyspark/sql/tests/test_conversion.py`. Both call sites already have integration coverage (`foreachPartition` and `DataSourceWriter` tests). ### Was this patch authored or co-authored using generative AI tooling? No -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
