viirya commented on PR #56943: URL: https://github.com/apache/spark/pull/56943#issuecomment-4860516163
Thanks @gaogaotiantian — you're right, and I reproduced the failing case. I'm going to close this. The concrete breakage is a numeric array with null elements. For `array<int>` value `[1, None, 3]`, `to_pandas()` promotes the inner array to a numpy `float64` array `[1., nan, 3.]` (numpy int can't hold nulls), so the UDF receives `nan` instead of `None` and the downstream int cast fails with `Float value nan was truncated`. `to_pylist()` returns the correct `[1, None, 3]`. The recursive ndarray->list step can't recover this, since the null is already lost to `nan` at the numpy layer. This is exactly the coercion corner case you predicted — my `test_arrow_python_udf` additions only used non-null arrays, so they missed it. And I think your deeper point is the right framing: this is fundamentally an Arrow issue. `to_pylist()` on a nested/list array shouldn't be several times slower than `to_pandas()` + converting back, and the right fix is upstream in pyarrow rather than a pandas detour in Spark that keeps re-introducing type-coercion differences. Routing through pandas/numpy trades a perf gap for a class of correctness risks (this null case, plus the bytes/str coercion that sank the companion output-side PR #56940). Closing this. I'll look at whether the `to_pylist()` slowness can be raised/fixed on the Arrow side instead. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
