Yicong-Huang opened a new pull request, #59433: URL: https://github.com/apache/spark/pull/59433
### What changes were proposed in this pull request? Move `SQL_ARROW_BATCHED_UDF` (Arrow-optimized `@udf`) from `read_udfs` onto the `EvalTypeHandler` pipeline as two handlers, selected by `use_legacy_pandas_udf_conversion` through the variant registry: - `ArrowBatchedUDFArrowHandler` (`_arrow.py`, `False`): the Arrow conversion path. - `ArrowBatchedUDFLegacyPandasHandler` (`_pandas.py`, `True`): the legacy pandas conversion path. Each branch body moves into its handler unchanged. `read_udfs` drops both branches and removes `SQL_ARROW_BATCHED_UDF` from its serializer selection. ``` BatchEvalTypeHandler ├─ ArrowBatchedUDFArrowHandler SQL_ARROW_BATCHED_UDF, legacy pandas conversion = False └─ ArrowBatchedUDFLegacyPandasHandler SQL_ARROW_BATCHED_UDF, legacy pandas conversion = True ``` ### Why are the changes needed? Part of SPARK-59415: run all Arrow/pandas UDF eval types through one handler dispatch path. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? - New handler unit tests for both variants in `test_arrow_eval_type_handlers.py` / `test_pandas_eval_type_handlers.py`. - `test_arrow_python_udf.py` (default, legacy and non-legacy suites) and `test_arrow_python_udf_cached.py` pass locally. - `bench_eval_type.py` `_ArrowBatchedBenchMixin` scenarios, 7 runs each, median +- stdev: | conversion | scenario | udf | before | after | |---|---|---|---|---| | arrow | sm_batch_many_col | identity | 71.5 +- 0.3 ms | 71.2 +- 0.1 ms | | arrow | lg_batch_few_col | stringify | 73.8 +- 0.3 ms | 74.1 +- 0.7 ms | | arrow | pure_strings | identity | 183.3 +- 7.8 ms | 182.8 +- 1.2 ms | | arrow | mixed_types | stringify | 148.8 +- 1.8 ms | 146.6 +- 0.3 ms | | legacy | sm_batch_many_col | identity | 14.2 +- 0.1 ms | 14.4 +- 0.1 ms | | legacy | lg_batch_few_col | stringify | 16.8 +- 0.0 ms | 17.1 +- 0.1 ms | | legacy | pure_strings | identity | 39.6 +- 3.0 ms | 39.9 +- 0.3 ms | | legacy | mixed_types | stringify | 30.1 +- 0.0 ms | 30.5 +- 5.2 ms | ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
