viirya opened a new pull request, #56943: URL: https://github.com/apache/spark/pull/56943
### What changes were proposed in this pull request? In the Arrow-optimized (non-legacy) regular Python UDF path, input columns are converted from Arrow to Python objects. For an array column whose leaf element type needs no per-element input converter (numeric / bool / string / binary and nested arrays of those), the column was materialized with `pa.Array.to_pylist()`, which is markedly slower than `pa.Array.to_pandas()` for list types. This PR converts such columns via `to_pandas()` and then recursively turns the resulting numpy ndarrays back into Python lists, so the UDF still receives the same objects (Python `list`, not `ndarray`) that `to_pylist()` would have produced, at any nesting depth. A per-column predicate `_input_fast_listify_safe` gates this: it applies only when the column-level input converter is `None` (no per-element coercion needed) and the type is an array whose leaf needs no converter. Columns needing a per-element converter (e.g. timestamp, struct, map, UDT) keep the existing `to_pylist()`-based path unchanged. This is independent of, and complementary to, the output-side fix in SPARK-57863: that one removes redundant per-element output conversion, this one speeds up input conversion. Together they eliminate the nested-column regression introduced when the Arrow serializer became the default for regular Python UDFs. ### Why are the changes needed? `to_pylist()` on nested/list Arrow arrays is the dominant cost of the Arrow Python UDF input path. Microbenchmark (`select max(f(arr))`, 4M rows, best-of-6), new Arrow path vs the legacy pandas path (`spark.sql.legacy.execution.pythonUDF.pandas.conversion.enabled=true`): | input type | before (ratio) | after (ratio) | |-----------------------|----------------|---------------| | array<long> | 1.44x | 0.89x | | array<string> | 2.44x | 1.02x | | array<array<int>> | 1.43x | 0.80x | (ratio = Arrow path time / legacy pandas path time; > 1 means the Arrow path is slower.) ### Does this PR introduce _any_ user-facing change? No. The UDF receives the same Python objects as before; only the conversion mechanism changes, gated to cases proven equivalent. ### How was this patch tested? New tests in `test_arrow_python_udf.py` verify array inputs (including nested arrays) reach the UDF as Python lists and that array<string> values are preserved. The full `test_arrow_python_udf` module passes. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 4.8) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
