HyukjinKwon commented on PR #57899: URL: https://github.com/apache/spark/pull/57899#issuecomment-5246962351
Thanks for the thorough review, @dongjoon-hyun — all three issues were real. Fixed in a3735aad84a9: **Bugs** 1. **Per-argument element type in the iterator path** — the iter path now converts each argument with its own element type (`element_types[o]`), matching the non-iter path, instead of reusing the first argument's type. Added `test_scalar_pandas_iter_udf_multiple_arguments_differ_in_type`, which iterates an `array<int>` alongside a repeated `string` outer column and would have hit the wrong-type conversion before. 2. **`pa.concat_arrays` timestamp type mismatch** — the streamed buffer is now created lazily from the first chunk's actual type rather than the UTC-typed `arrow_element_type`, so a session-timezone-typed pandas timestamp chunk concatenates cleanly. Added `test_scalar_pandas_iter_udf_timestamp_return_type` under `America/Los_Angeles`, asserting the iterator result matches the equivalent non-iterator pandas UDF (both share the session-tz typing, so the check is offset-agnostic). Confirmed it reproduces the `ArrowInvalid` before the fix. 3. **`ruff format`** — resolved (the reformatted call was part of fix 1). **Minor items (all taken)** - Non-iter path verifies the result length *before* re-nesting, so a wrong-length result raises the friendly `RESULT_ROWS_MISMATCH` rather than an opaque pyarrow offset error. - Non-iter pandas path restores the base scalar-pandas "must return a `pandas.DataFrame` for `StructType`" check. - Non-iter path flattens each input column once per batch and shares it across fused UDFs (matching the 102 path), instead of re-flattening per UDF. - Removed the dead `chunk.combine_chunks()` branch and its inaccurate comment (`_elementwise_result_to_arrow` always returns a `pa.Array`). - Iterator path fails fast on over-production via `OUTPUT_EXCEEDS_INPUT_ROWS`, instead of only detecting it at the end after the buffer has grown. Full suite (classic + Connect parity, 50 tests each) and `ruff format`/`ruff check` are green. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
