Yicong-Huang opened a new pull request, #57728: URL: https://github.com/apache/spark/pull/57728
### What changes were proposed in this pull request? `worker.py` raises `RESULT_ROWS_MISMATCH` from four places when a UDF returns a row count that does not match the input. SPARK-58161 hardcoded a `message=` override at each of them carrying a pandas-specific tail: `... Result vector from pandas_udf was not the required length: expected N, got M.`. This tail is incorrect on the arrow UDF path: `verify_scalar_result` is also called for arrow UDFs, yet the message says `pandas_udf`. This PR removes all four hardcoded `message=` overrides so the generic `RESULT_ROWS_MISMATCH` template applies uniformly. While here, it consolidates the two row-count helpers -- `verify_result_row_count` (takes a list length) and the iterator-based generator (renamed `verify_iter_result_row_count`) -- so the generator delegates the final mismatch check to the single `verify_result_row_count` judgement function, and it drops the now-redundant `error_class` parameter and `if/else` branch on that generator (both call sites only ever passed `RESULT_ROWS_MISMATCH`). ### Why are the changes needed? The pandas-specific tail is misleading on the arrow UDF path, and the row-count mismatch judgement was duplicated across four call sites. Consolidating to one helper and letting the generic error template render the message removes the duplication and makes the message consistent and correct across pandas and arrow UDFs. ### Does this PR introduce _any_ user-facing change? Yes. The `RESULT_ROWS_MISMATCH` message changes from: `The number of output rows (M) must match the number of input rows (N). Result vector from pandas_udf was not the required length: expected N, got M.` to: `The number of output rows (M) must match the number of input rows (N).` The error class (`RESULT_ROWS_MISMATCH`) and its `messageParameters` (`output_length`, `input_length`) are unchanged. ### How was this patch tested? Existing assertions in `python/pyspark/sql/tests/pandas/test_pandas_udf_scalar.py` already match on the template text (`The number of output rows.*must match the number of input rows`), which remains produced by the generic template. The consolidated helpers were exercised in isolation (immediate list, iterator match/mismatch, callable expected, scalar delegation, non-array-like type error) to confirm identical error class, message parameters, and rendered message. ### Was this patch authored or co-authored using generative AI tooling? No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
