Yicong-Huang opened a new pull request, #59776: URL: https://github.com/apache/spark/pull/59776
### What changes were proposed in this pull request? Two handler API changes under the EvalTypeHandler umbrella (SPARK-59415): 1. **Single-UDF handlers are enforced by the base.** `EvalTypeHandler` gets a `_single_udf` class flag, set on every handler that takes exactly one UDF (grouped/cogrouped map, map iter, scalar iter, grouped agg iter, scalar iter elementwise, the stateful handlers, and `UDTFHandlerBase`). For those, `__init__` fails fast on any other UDF count and exposes the UDF and its functions as `_udf`, `_raw_func` and `_func`, replacing the per-handler `assert len(udfs) == 1` and `udfs[0]` / `_funcs[0]` lookups. Multi-UDF handlers keep iterating the list. 2. **Chaining nested UDF calls moves from the worker into the handler.** `worker.py` only deserializes (`unpickle_and_chain` -> `unpickle_udf_funcs`, `chain` removed), and the `UDF` record carries the unchained `funcs` instead of `raw_func`. `EvalTypeHandler.__init__` chains them (`chain` now lives in `_base.py`) into `_raw_funcs`, then wraps for profiling and `fail_on_stopiteration` as before. A single function is kept as is, so incremental `Aggregator` objects, `getfullargspec` and the data-source profiler `__module__` check see the same object as before. ### Why are the changes needed? Keeps the worker to reading the wire format and gives handlers one place for their UDF invariants instead of repeating them per eval type. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? New unit tests in `test_base_eval_type_handlers.py` (single-UDF guard, `_single_udf` per handler, chaining order, single function not wrapped, `unpickle_udf_funcs`); the other handler tests are updated for the `funcs` field. Existing e2e tests pass, including nested-UDF chaining in `test_udf` / `test_pandas_udf*`, `test_udf_profiler`, `test_python_datasource`, `test_udtf`, and transform-with-state. ASV (`bench_eval_type`, 11 TimeBench classes) shows no regression; per-bench geomean of after/before is 0.96-1.01. Representative run, `GroupedMapArrowUDFTimeBench`, `identity_udf`: | scenario | before | after | |---|---|---| | few_groups_sm | 5.01±0.08ms | 5.45±0.3ms | | few_groups_lg | 40.4±2ms | 39.1±0.3ms | | many_groups_sm | 162±4ms | 157±4ms | | many_groups_lg | 104±2ms | 103±0.9ms | | wide_cols | 147±2ms | 145±0.7ms | | multi_key | 28.9±0.7ms | 28.4±0.1ms | `few_groups_sm` does not reproduce in an interleaved `timeit` run of `time_worker` (4.68ms before vs 4.65ms after, median); the per-batch path is unchanged. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
