This is an automated email from the ASF dual-hosted git repository.

Yicong-Huang pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new 3f175e6c0caa [SPARK-60503][PYTHON] Assign UDFs to eval type handlers 
and move UDF wrapping onto them
3f175e6c0caa is described below

commit 3f175e6c0caafc4fb57f37659c9f0f5b4638a619
Author: Yicong Huang <[email protected]>
AuthorDate: Sat Oct 24 22:03:48 2026 +0000

    [SPARK-60503][PYTHON] Assign UDFs to eval type handlers and move UDF 
wrapping onto them
    
    ### What changes were proposed in this pull request?
    
    The worker builds a `UDF` or `UDTF` record from what it unpickled, and 
constructs the eval type handler from the records. Wrapping each UDF 
(profiling, `fail_on_stopiteration`) and instantiating the UDTF (AnalyzeResult, 
the partition wrapper) move from `worker.py` onto the handlers.
    
    ```python
    if isinstance(init_info.udf_info, UDTFInfo):
        udtf_info = init_info.udf_info
        udtf_cls = read_command(pickleSer, udtf_info.handler)
        return_type = _parse_datatype_json_string(udtf_info.return_type)
        ...  # check return_type is a StructType; unpickle the AnalyzeResult
        udtf = UDTF(
            udtf_cls=udtf_cls,
            args_offsets=udtf_info.args,
            kwargs_offsets=udtf_info.kwargs,
            return_type=return_type,
            partition_child_indexes=udtf_info.partition_child_indexes,
            analyze_result=analyze_result,
            name=udtf_info.name,
        )
        handler = handler_cls(udfs=[udtf], runner_conf=runner_conf, 
eval_conf=eval_conf)
    else:
        udfs = []
        for udf_info in init_info.udf_info:
            chained_func, return_type = unpickle_and_chain(pickleSer, 
udf_info.udfs)
            udfs.append(
                UDF(
                    raw_func=chained_func,
                    args_offsets=udf_info.args,
                    kwargs_offsets=udf_info.kwargs,
                    return_type=return_type,
                    result_id=udf_info.result_id,
                )
            )
        handler = handler_cls(udfs=udfs, runner_conf=runner_conf, 
eval_conf=eval_conf)
    ```
    
    - `worker.py`: `read_single_udf` and `read_single_udtf` are removed. All 
unpickling stays in the worker: `unpickle_and_chain` unpickles the functions 
chained on one column, and the UDTF class and AnalyzeResult are unpickled 
before building the `UDTF` record.
    - `UDF` and `UDTF` keep their field order and hold only what the worker 
read: `UDF` drops the wrapped `func`, keeping the unwrapped chained function as 
`raw_func`, and gains `result_id`; `UDTF.udtf_cls` (replacing the instance) 
plus `partition_child_indexes`, `analyze_result` and `name`.
    - `EvalTypeHandler.__init__` wraps each UDF's function for profiling 
(`_with_profiler`, which selects `_with_perf_profiler` or 
`_with_memory_profiler` by conf) and in `fail_on_stopiteration`, as 
`self._funcs` at the indexes of `self._udfs`. The hardcoded 
`fail_on_stopiteration` exemption set becomes a `_fail_on_stopiteration = 
False` class attribute on the same handlers.
    - New `UDTFHandlerBase`, the common base of the UDTF handlers. It owns 
`partition_wrapper`, and its `__init__` instantiates the UDTF as `self._udtf` 
(the rest of the former `read_single_udtf`).
    - The handlers read the wrapped functions from `self._funcs`, the unwrapped 
ones (signature, incremental Aggregator) from the record's `raw_func`, and the 
UDTF instance from `self._udtf`, directly rather than through attributes that 
only renamed them (`self._udf_func`, `self._grouped_udf`, `self._return_type`, 
...).
    
    ### Why are the changes needed?
    
    Part of SPARK-59415. How a UDF is wrapped depends on its eval type, so the 
wrapping belongs on the handler, not in eval-type sets in the worker. Keeping 
profiling on the base class also lets the UDTF handlers reuse it.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No.
    
    ### How was this patch tested?
    
    New unit tests cover the handler's UDF wrapping (the 
`fail_on_stopiteration` setting of every registered handler, profiler 
selection), the UDTF instantiation (AnalyzeResult, the pickled and Arrow 
partition wrappers, invalid UDTFs) and `unpickle_and_chain`. The handler tests 
build their handlers from `UDF` and `UDTF` records.
    
    These existing tests passed locally: `pyspark.eval_handlers.tests`, 
`test_udf_profiler`, `test_memory_profiler`, `test_udtf`, `test_arrow_udtf`, 
`test_pandas_udf_scalar`, `test_udf`, `test_arrow_python_udf`, 
`test_python_datasource`, `test_pandas_grouped_map_with_state`, 
`test_pandas_udf_grouped_agg`, `test_pandas_udf_window`, 
`test_arrow_grouped_map`, `test_pandas_map`, `test_pandas_cogrouped_map`, 
`test_pandas_grouped_map`, `test_pandas_transform_with_state`.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #59744 from Yicong-Huang/simplify-udf-preparation.
    
    Authored-by: Yicong Huang <[email protected]>
    Signed-off-by: Yicong-Huang <[email protected]>
    (cherry picked from commit 4cd8da26725e309844e8a720e50e131cb0274ee3)
    Signed-off-by: Yicong-Huang <[email protected]>
---
 python/benchmarks/bench_eval_type.py               |   2 +-
 python/pyspark/eval_handlers/_arrow.py             |  45 ++-
 python/pyspark/eval_handlers/_base.py              | 119 +++++++-
 python/pyspark/eval_handlers/_elementwise.py       |  21 +-
 python/pyspark/eval_handlers/_pandas.py            |  74 +++--
 python/pyspark/eval_handlers/_stateful.py          |  55 ++--
 python/pyspark/eval_handlers/_typing.py            |  31 ++-
 python/pyspark/eval_handlers/_udtf.py              | 150 +++++++---
 python/pyspark/eval_handlers/_vanilla.py           |   8 +-
 .../tests/test_arrow_eval_type_handlers.py         |  40 +--
 .../tests/test_base_eval_type_handlers.py          | 115 ++++++--
 .../tests/test_elementwise_eval_type_handlers.py   |   2 +-
 .../tests/test_pandas_eval_type_handlers.py        |  10 +-
 .../tests/test_stateful_eval_type_handlers.py      |   4 +-
 .../tests/test_udtf_eval_type_handlers.py          | 307 +++++++++++++--------
 .../tests/test_vanilla_eval_type_handlers.py       |   4 +-
 python/pyspark/worker.py                           | 283 ++++---------------
 17 files changed, 723 insertions(+), 547 deletions(-)

diff --git a/python/benchmarks/bench_eval_type.py 
b/python/benchmarks/bench_eval_type.py
index f659216499d8..592654c43eb9 100644
--- a/python/benchmarks/bench_eval_type.py
+++ b/python/benchmarks/bench_eval_type.py
@@ -198,7 +198,7 @@ class MockProtocolWriter:
         arg_offsets: list[int],
         buf: io.BytesIO,
     ) -> None:
-        """Write the ``read_single_udf`` portion of the protocol."""
+        """Write the ``UDFInfo`` portion of the protocol."""
         write_int(1, buf)  # num_udfs
         write_int(len(arg_offsets), buf)  # num_arg
         for offset in arg_offsets:
diff --git a/python/pyspark/eval_handlers/_arrow.py 
b/python/pyspark/eval_handlers/_arrow.py
index ed7fd87f6e3d..dd75347a0bf6 100644
--- a/python/pyspark/eval_handlers/_arrow.py
+++ b/python/pyspark/eval_handlers/_arrow.py
@@ -84,7 +84,6 @@ class 
ArrowCoGroupedMapUDFHandler(CoGroupedEvalTypeHandler["pa.RecordBatch"]):
         super().__init__(udfs, runner_conf, eval_conf)
         assert len(udfs) == 1, "One COGROUPED_MAP_ARROW UDF expected here."
         udf = udfs[0]
-        self._cogrouped_udf = udf.func
         # signature was lost when wrapping it
         self._num_udf_args = len(inspect.getfullargspec(udf.raw_func).args)
         parsed_offsets = extract_key_value_indexes(udf.args_offsets)
@@ -110,11 +109,11 @@ class 
ArrowCoGroupedMapUDFHandler(CoGroupedEvalTypeHandler["pa.RecordBatch"]):
             right_values = table_from_batches(right_batches, 
self._right_val_cols)
 
             if self._num_udf_args == 2:
-                result = self._cogrouped_udf(left_values, right_values)
+                result = self._funcs[0](left_values, right_values)
             else:
                 key_table = left_keys if left_keys.num_rows > 0 else right_keys
                 key = tuple(c[0] for c in key_table.columns)
-                result = self._cogrouped_udf(key, left_values, right_values)
+                result = self._funcs[0](key, left_values, right_values)
 
             verify_return_type(result, pa.Table)
             # Verify types (and reorder by name when configured).
@@ -142,7 +141,6 @@ class 
ArrowGroupedMapIterUDFHandler(GroupedEvalTypeHandler["pa.RecordBatch"]):
         super().__init__(udfs, runner_conf, eval_conf)
         assert len(udfs) == 1, "One GROUPED_MAP_ARROW_ITER UDF expected here."
         udf = udfs[0]
-        self._grouped_udf = udf.func
         # signature was lost when wrapping it
         self._num_udf_args = len(inspect.getfullargspec(udf.raw_func).args)
         parsed_offsets = extract_key_value_indexes(udf.args_offsets)
@@ -181,10 +179,10 @@ class 
ArrowGroupedMapIterUDFHandler(GroupedEvalTypeHandler["pa.RecordBatch"]):
 
             # Call UDF with iterator of batches
             if self._num_udf_args == 1:
-                result = self._grouped_udf(value_batches)
+                result = self._funcs[0](value_batches)
             else:
                 key = tuple(c[0] for c in keys.columns)
-                result = self._grouped_udf(key, value_batches)
+                result = self._funcs[0](key, value_batches)
 
             # Verify (and reorder by name when configured) each output batch
             for batch in verify_return_type(result, Iterator[pa.RecordBatch]):
@@ -213,7 +211,6 @@ class 
ArrowGroupedMapUDFHandler(GroupedEvalTypeHandler["pa.RecordBatch"]):
         super().__init__(udfs, runner_conf, eval_conf)
         assert len(udfs) == 1, "One GROUPED_MAP_ARROW UDF expected here."
         udf = udfs[0]
-        self._grouped_udf = udf.func
         # signature was lost when wrapping it
         self._num_udf_args = len(inspect.getfullargspec(udf.raw_func).args)
         parsed_offsets = extract_key_value_indexes(udf.args_offsets)
@@ -251,10 +248,10 @@ class 
ArrowGroupedMapUDFHandler(GroupedEvalTypeHandler["pa.RecordBatch"]):
             # Call UDF
             value_table = pa.Table.from_batches(value_batches)
             if self._num_udf_args == 1:
-                result = self._grouped_udf(value_table)
+                result = self._funcs[0](value_table)
             else:
                 key = tuple(c[0] for c in keys.columns)
-                result = self._grouped_udf(key, value_table)
+                result = self._funcs[0](key, value_table)
 
             verify_return_type(result, pa.Table)
             # Verify types (and reorder by name when configured).
@@ -292,11 +289,11 @@ class 
ArrowGroupedAggUDFHandler(GroupedEvalTypeHandler["pa.RecordBatch"]):
         for group in data:
             concatenated = ArrowBatchTransformer.concat_batches(group)
             results = [
-                udf.func(
+                func(
                     *[concatenated.column(o) for o in udf.args_offsets],
                     **{k: concatenated.column(v) for k, v in 
udf.kwargs_offsets.items()},
                 )
-                for udf in self._udfs
+                for udf, func in zip(self._udfs, self._funcs)
             ]
             result_arrays = [pa.array([r]) for r in results]
             batch = pa.RecordBatch.from_arrays(result_arrays, self._col_names)
@@ -314,7 +311,6 @@ class 
ArrowGroupedAggIterUDFHandler(GroupedEvalTypeHandler["pa.RecordBatch"]):
         super().__init__(udfs, runner_conf, eval_conf)
         assert len(udfs) == 1, "One GROUPED_AGG_ARROW_ITER UDF expected here."
         udf = udfs[0]
-        self._udf_func, self._args_offsets = udf.func, udf.args_offsets
         self._return_schema = to_arrow_schema(
             StructType([StructField("_0", udf.return_type)]),
             timezone="UTC",
@@ -324,7 +320,7 @@ class 
ArrowGroupedAggIterUDFHandler(GroupedEvalTypeHandler["pa.RecordBatch"]):
     def run(self, split_index: int, data: Iterator[GroupedBatch]) -> 
Iterator[pa.RecordBatch]:
         import pyarrow as pa
 
-        args_offsets = self._args_offsets
+        args_offsets = self._udfs[0].args_offsets
 
         def extract_args(batch: pa.RecordBatch) -> Any:
             args = tuple(batch.column(o) for o in args_offsets)
@@ -332,7 +328,7 @@ class 
ArrowGroupedAggIterUDFHandler(GroupedEvalTypeHandler["pa.RecordBatch"]):
 
         for group in data:
             batch_iter = map(extract_args, group)
-            result = self._udf_func(batch_iter)
+            result = self._funcs[0](batch_iter)
             # Drain remaining batches to maintain stream position
             for _ in batch_iter:
                 pass
@@ -367,8 +363,7 @@ class 
ArrowWindowAggUDFHandler(GroupedEvalTypeHandler["pa.RecordBatch"]):
             num_rows = concatenated.num_rows
 
             result_arrays = []
-            for udf_index, udf in enumerate(self._udfs):
-                udf_func = udf.func
+            for udf_index, (udf, udf_func) in enumerate(zip(self._udfs, 
self._funcs)):
                 args_offsets, kwargs_offsets = udf.args_offsets, 
udf.kwargs_offsets
                 bound_type = self._window_bound_types[udf_index]
                 if bound_type == "unbounded":
@@ -688,7 +683,6 @@ class 
ArrowMapUDFHandler(BatchEvalTypeHandler["pa.RecordBatch"]):
         require_minimum_pyarrow_version()
         super().__init__(udfs, runner_conf, eval_conf)
         assert len(udfs) == 1, "One MAP_ARROW_ITER UDF expected here."
-        self._udf_func = udfs[0].func
 
     def run(self, split_index: int, data: Iterator[pa.RecordBatch]) -> 
Iterator[pa.RecordBatch]:
         import pyarrow as pa
@@ -697,7 +691,7 @@ class 
ArrowMapUDFHandler(BatchEvalTypeHandler["pa.RecordBatch"]):
         input_batches = map(ArrowBatchTransformer.flatten_struct, data)
 
         # invoke the UDF
-        output_batches = self._udf_func(input_batches)
+        output_batches = self._funcs[0](input_batches)
 
         # The declared signature is Iterator[...], so a strict iterator is 
required by
         # default. With the legacy flag, accept any object Python can iterate 
over -- via
@@ -731,7 +725,6 @@ class 
ArrowScalarIterUDFHandler(BatchEvalTypeHandler["pa.RecordBatch"]):
         super().__init__(udfs, runner_conf, eval_conf)
         assert len(udfs) == 1, "One SCALAR_ARROW_ITER UDF expected here."
         udf = udfs[0]
-        self._udf_func, self._args_offsets = udf.func, udf.args_offsets
         self._arrow_return_type = to_arrow_type(
             udf.return_type, timezone="UTC", 
prefers_large_types=runner_conf.use_large_var_types
         )
@@ -739,7 +732,7 @@ class 
ArrowScalarIterUDFHandler(BatchEvalTypeHandler["pa.RecordBatch"]):
     def run(self, split_index: int, data: Iterator[pa.RecordBatch]) -> 
Iterator[pa.RecordBatch]:
         import pyarrow as pa
 
-        args_offsets = self._args_offsets
+        args_offsets = self._udfs[0].args_offsets
         num_input_rows = 0
 
         def extract_args(batch: pa.RecordBatch) -> Any:
@@ -752,7 +745,7 @@ class 
ArrowScalarIterUDFHandler(BatchEvalTypeHandler["pa.RecordBatch"]):
         args_iter = map(extract_args, data)
 
         # Call UDF and verify result type (iterator of pa.Array)
-        verified_iter = verify_return_type(self._udf_func(args_iter), 
Iterator[pa.Array])
+        verified_iter = verify_return_type(self._funcs[0](args_iter), 
Iterator[pa.Array])
 
         # Process results: enforce schema and assemble into RecordBatch
         target_schema = pa.schema([pa.field("_0", self._arrow_return_type)])
@@ -797,11 +790,11 @@ class 
ArrowScalarUDFHandler(BatchEvalTypeHandler["pa.RecordBatch"]):
         for batch in data:
             output_batch = pa.RecordBatch.from_arrays(
                 [
-                    udf.func(
+                    func(
                         *[batch.column(o) for o in udf.args_offsets],
                         **{k: batch.column(v) for k, v in 
udf.kwargs_offsets.items()},
                     )
-                    for udf in self._udfs
+                    for udf, func in zip(self._udfs, self._funcs)
                 ],
                 self._col_names,
             )
@@ -820,6 +813,8 @@ class 
ArrowBatchedUDFArrowHandler(BatchEvalTypeHandler["pa.RecordBatch"]):
     eval_type = PythonEvalType.SQL_ARROW_BATCHED_UDF
     variant_conf = "use_legacy_pandas_udf_conversion"
     variant_value = False
+    # run wraps the row-by-row evaluation in fail_on_stopiteration itself.
+    _fail_on_stopiteration = False
 
     def __init__(self, udfs: list[UDF], runner_conf: RunnerConf, eval_conf: 
EvalConf) -> None:
         require_minimum_pyarrow_version()
@@ -827,9 +822,9 @@ class 
ArrowBatchedUDFArrowHandler(BatchEvalTypeHandler["pa.RecordBatch"]):
 
         # --- UDF preparation ---
         self._udf_infos = []
-        for udf in udfs:
+        for udf, func in zip(udfs, self._funcs):
             wrapped_func, args_kwargs_offsets = wrap_kwargs_support(
-                udf.func, udf.args_offsets, udf.kwargs_offsets
+                func, udf.args_offsets, udf.kwargs_offsets
             )
             zero_arg = len(args_kwargs_offsets) == 0
             self._udf_infos.append(
diff --git a/python/pyspark/eval_handlers/_base.py 
b/python/pyspark/eval_handlers/_base.py
index a4793c5ad07a..d9bc7354b2b2 100644
--- a/python/pyspark/eval_handlers/_base.py
+++ b/python/pyspark/eval_handlers/_base.py
@@ -17,13 +17,13 @@
 
 """Base classes and registry for the eval type handlers.
 
-This leaf module imports only ``_typing`` and the serializers, so the package
-``__init__`` and the concrete-handler submodules can both import it.
+This leaf module imports only ``_typing``, the serializers and 
``pyspark.util``, so the
+package ``__init__`` and the concrete-handler submodules can both import it.
 """
 
 from abc import ABCMeta, abstractmethod
 from collections.abc import Iterator
-from typing import TYPE_CHECKING, Any, ClassVar, Generic, Optional
+from typing import TYPE_CHECKING, Any, Callable, ClassVar, Generic, Optional
 
 from pyspark.eval_handlers._typing import (
     UDF,
@@ -39,6 +39,7 @@ from pyspark.sql.pandas.serializers import (
     ArrowStreamGroupSerializer,
     ArrowStreamSerializer,
 )
+from pyspark.util import fail_on_stopiteration
 
 if TYPE_CHECKING:
     import pyarrow as pa  # noqa: F401  # only in the batch category's 
forward-ref subscript
@@ -149,6 +150,9 @@ class EvalTypeHandler(Generic[InputBatch, OutputBatch], 
metaclass=_EvalTypeHandl
     variant_value: ClassVar[Any] = None
     # Whether the handler's UDFs return an iterator.
     _iter_based: ClassVar[bool] = False
+    # Whether __init__ wraps the UDF in fail_on_stopiteration, so a 
StopIteration raised in
+    # user code fails the task instead of silently ending the loop that calls 
the UDF.
+    _fail_on_stopiteration: ClassVar[bool] = True
 
     def __init__(
         self, udfs: "list[UDF] | list[UDTF]", runner_conf: "RunnerConf", 
eval_conf: "EvalConf"
@@ -156,6 +160,115 @@ class EvalTypeHandler(Generic[InputBatch, OutputBatch], 
metaclass=_EvalTypeHandl
         self._udfs: list[Any] = udfs
         self._runner_conf = runner_conf
         self._eval_conf = eval_conf
+        # Each UDF's function wrapped for profiling and, unless 
_fail_on_stopiteration is False,
+        # in fail_on_stopiteration, at the indexes of _udfs; empty for a UDTF.
+        self._funcs: list[Callable[..., Any]] = []
+        for udf in udfs:
+            if isinstance(udf, UDF):
+                profiling_func = self._with_profiler(udf.raw_func, 
udf.result_id)
+                self._funcs.append(
+                    fail_on_stopiteration(profiling_func)
+                    if self._fail_on_stopiteration
+                    else profiling_func
+                )
+
+    def _with_profiler(self, f: Callable[..., Any], result_id: int) -> 
Callable[..., Any]:
+        """Return ``f`` profiled by the profiler its config selects, or ``f`` 
itself when none is
+        set."""
+        # If f is from pyspark.sql.worker, it is to read/write data source.
+        # In this case, we check the data_source_profiler config.
+        module = getattr(f, "__module__", "")
+        if isinstance(module, str) and 
module.startswith("pyspark.sql.worker."):
+            profiler = self._runner_conf.data_source_profiler
+        else:
+            profiler = self._runner_conf.udf_profiler
+        if profiler == "perf":
+            return self._with_perf_profiler(f, result_id)
+        elif profiler == "memory":
+            return self._with_memory_profiler(f, result_id)
+        else:
+            return f
+
+    def _with_perf_profiler(self, f: Callable[..., Any], result_id: int) -> 
Callable[..., Any]:
+        from pyspark.accumulators import SpecialAccumulatorIds, 
_deserialize_accumulator
+        from pyspark.sql.profiler import (
+            ProfileResultsParam,
+            ProfileResultsParamV2,
+            WorkerPerfProfiler,
+        )
+
+        accumulator = _deserialize_accumulator(
+            SpecialAccumulatorIds.SQL_UDF_PROFIER, None, ProfileResultsParam
+        )
+        accumulator_v2 = _deserialize_accumulator(
+            SpecialAccumulatorIds.SQL_UDF_PROFIER_V2, {}, ProfileResultsParamV2
+        )
+
+        if self._iter_based:
+
+            def profiling_func(*args: Any, **kwargs: Any) -> Any:
+                iterator = iter(f(*args, **kwargs))
+                while True:
+                    try:
+                        with WorkerPerfProfiler(accumulator, accumulator_v2, 
result_id):
+                            item = next(iterator)
+                        yield item
+                    except StopIteration:
+                        break
+
+        else:
+
+            def profiling_func(*args: Any, **kwargs: Any) -> Any:
+                with WorkerPerfProfiler(accumulator, accumulator_v2, 
result_id):
+                    ret = f(*args, **kwargs)
+                return ret
+
+        return profiling_func
+
+    def _with_memory_profiler(self, f: Callable[..., Any], result_id: int) -> 
Callable[..., Any]:
+        import pyspark.memory_profiler_ext
+        from pyspark.accumulators import SpecialAccumulatorIds, 
_deserialize_accumulator
+        from pyspark.sql.profiler import (
+            ProfileResultsParam,
+            ProfileResultsParamV2,
+            WorkerMemoryProfiler,
+        )
+
+        if not pyspark.memory_profiler_ext.has_memory_profiler:
+            return f
+
+        accumulator = _deserialize_accumulator(
+            SpecialAccumulatorIds.SQL_UDF_PROFIER, None, ProfileResultsParam
+        )
+
+        accumulator_v2 = _deserialize_accumulator(
+            SpecialAccumulatorIds.SQL_UDF_PROFIER_V2, {}, ProfileResultsParamV2
+        )
+
+        if self._iter_based:
+
+            def profiling_func(*args: Any, **kwargs: Any) -> Any:
+                g = f(*args, **kwargs)
+                iterator = iter(g)
+
+                while True:
+                    try:
+                        with WorkerMemoryProfiler(
+                            accumulator, accumulator_v2, result_id, g.gi_code
+                        ):
+                            item = next(iterator)
+                        yield item
+                    except StopIteration:
+                        break
+
+        else:
+
+            def profiling_func(*args: Any, **kwargs: Any) -> Any:
+                with WorkerMemoryProfiler(accumulator, accumulator_v2, 
result_id, f):
+                    ret = f(*args, **kwargs)
+                return ret
+
+        return profiling_func
 
     @property
     @abstractmethod
diff --git a/python/pyspark/eval_handlers/_elementwise.py 
b/python/pyspark/eval_handlers/_elementwise.py
index 2a836cabaf6e..c5a1ce4e3dd3 100644
--- a/python/pyspark/eval_handlers/_elementwise.py
+++ b/python/pyspark/eval_handlers/_elementwise.py
@@ -200,7 +200,7 @@ class 
ArrowElementwiseUDFHandler(BatchEvalTypeHandler["pa.RecordBatch"]):
         for udf_index, udf in enumerate(udfs):
             udf_return_type = udf.return_type
             wrapped_func, args_kwargs_offsets = wrap_kwargs_support(
-                udf.func, udf.args_offsets, udf.kwargs_offsets
+                self._funcs[udf_index], udf.args_offsets, udf.kwargs_offsets
             )
             depth = nesting[udf_index] if nesting is not None else 1
             # Each argument arrives as ``array^depth<T>``; convert its leaves 
with the element type
@@ -395,7 +395,7 @@ class 
_ScalarElementwiseUDFHandler(_VectorizedElementwiseUDFHandler):
         for udf_index, udf in enumerate(udfs):
             udf_return_type = udf.return_type
             wrapped_func, args_kwargs_offsets = wrap_kwargs_support(
-                udf.func, udf.args_offsets, udf.kwargs_offsets
+                self._funcs[udf_index], udf.args_offsets, udf.kwargs_offsets
             )
             udf_output_schema = _elementwise_output_schema(udf_return_type, 
runner_conf)
             depth = nesting[udf_index] if nesting is not None else 1
@@ -518,8 +518,6 @@ class 
_ScalarIterElementwiseUDFHandler(_VectorizedElementwiseUDFHandler):
 
         assert len(udfs) == 1, "One SCALAR_*_ITER_ELEMENTWISE UDF expected 
here."
         udf = udfs[0]
-        self._udf_func, self._args_offsets = udf.func, udf.args_offsets
-        self._return_type = udf.return_type
         assert not udf.kwargs_offsets, "Iterator UDFs do not take keyword 
arguments."
 
         # Each argument arrives as ``array^depth<T>``; the vectorized function 
must see the leaf
@@ -528,9 +526,9 @@ class 
_ScalarIterElementwiseUDFHandler(_VectorizedElementwiseUDFHandler):
         nesting = eval_conf.elementwise_nesting
         self._depth = nesting[0] if nesting is not None else 1
         self._udf_input_schema = _elementwise_input_schema(
-            eval_conf.input_type, self._args_offsets, self._depth
+            eval_conf.input_type, udf.args_offsets, self._depth
         )
-        self._udf_output_schema = 
_elementwise_output_schema(self._return_type, runner_conf)
+        self._udf_output_schema = _elementwise_output_schema(udf.return_type, 
runner_conf)
 
     @abstractmethod
     def _verify_return_type(self, results: Any) -> Iterator[Any]:
@@ -552,7 +550,7 @@ class 
_ScalarIterElementwiseUDFHandler(_VectorizedElementwiseUDFHandler):
             # shapes re-nest this batch's rows. The user function sees the 
flat leaves as a
             # pandas Series / DataFrame (pandas) or a pa.Array (Arrow), each 
with its leaf type.
             flat_batch, list_lengths_by_level, large_list_by_level = 
_elementwise_flatten_inputs(
-                batch, self._args_offsets, self._depth
+                batch, self._udfs[0].args_offsets, self._depth
             )
             pending_shapes.append((list_lengths_by_level, large_list_by_level, 
flat_batch.num_rows))
             num_input_elements += flat_batch.num_rows
@@ -561,7 +559,7 @@ class 
_ScalarIterElementwiseUDFHandler(_VectorizedElementwiseUDFHandler):
 
         flat_args_iter = map(extract_flat, data)
 
-        verified_iter = 
self._verify_return_type(self._udf_func(flat_args_iter))
+        verified_iter = 
self._verify_return_type(self._funcs[0](flat_args_iter))
 
         # Buffer the streamed flat element results and emit an ``array<R>`` 
row as soon as the
         # shape at the head of the FIFO is fully covered. A row whose length 
is 0 (an empty
@@ -603,7 +601,9 @@ class 
_ScalarIterElementwiseUDFHandler(_VectorizedElementwiseUDFHandler):
         def process_results() -> Iterator[pa.RecordBatch]:
             nonlocal pending_chunks, pending_len, empty_schema, 
num_output_elements
             for result in verified_iter:
-                chunk = self._to_flat_batch(result, self._return_type, 
self._udf_output_schema)
+                chunk = self._to_flat_batch(
+                    result, self._udfs[0].return_type, self._udf_output_schema
+                )
                 num_output_elements += chunk.num_rows
                 # Fail fast if the UDF over-produces, before the buffer grows 
unbounded (the
                 # base iterator paths do the same via verify_output_row_limit).
@@ -653,13 +653,14 @@ class PandasScalarIterElementwiseUDFHandler(
     an iterator of pandas Series (DataFrames for struct types)."""
 
     eval_type = PythonEvalType.SQL_SCALAR_PANDAS_ITER_ELEMENTWISE_UDF
+    _fail_on_stopiteration = False
 
     def _verify_return_type(self, results: Any) -> Iterator[Any]:
         import pandas as pd
 
         pandas_iter_type = (
             Iterator[pd.DataFrame]
-            if isinstance(self._return_type, StructType)
+            if isinstance(self._udfs[0].return_type, StructType)
             else Iterator[pd.Series]
         )
         return verify_return_type(results, pandas_iter_type)
diff --git a/python/pyspark/eval_handlers/_pandas.py 
b/python/pyspark/eval_handlers/_pandas.py
index 744c785520e4..9a503ee425ab 100644
--- a/python/pyspark/eval_handlers/_pandas.py
+++ b/python/pyspark/eval_handlers/_pandas.py
@@ -102,9 +102,9 @@ class 
PandasScalarUDFHandler(BatchEvalTypeHandler["pa.RecordBatch"]):
 
             # Process: evaluate each UDF column-wise on pandas Series.
             results = []
-            for udf in self._udfs:
+            for udf, func in zip(self._udfs, self._funcs):
                 return_type = udf.return_type
-                result = udf.func(
+                result = func(
                     *[pandas_columns[o] for o in udf.args_offsets],
                     **{k: pandas_columns[v] for k, v in 
udf.kwargs_offsets.items()},
                 )
@@ -150,6 +150,7 @@ class 
PandasScalarIterUDFHandler(BatchEvalTypeHandler["pa.RecordBatch"]):
 
     eval_type = PythonEvalType.SQL_SCALAR_PANDAS_ITER_UDF
     _iter_based = True
+    _fail_on_stopiteration = False
 
     def __init__(self, udfs: list[UDF], runner_conf: "RunnerConf", eval_conf: 
"EvalConf") -> None:
         import pandas as pd
@@ -159,12 +160,10 @@ class 
PandasScalarIterUDFHandler(BatchEvalTypeHandler["pa.RecordBatch"]):
         super().__init__(udfs, runner_conf, eval_conf)
         assert len(udfs) == 1, "One SCALAR_PANDAS_ITER UDF expected here."
         udf = udfs[0]
-        self._udf_func, self._args_offsets = udf.func, udf.args_offsets
-        self._return_type = udf.return_type
-        self._return_schema = StructType([StructField("_0", 
self._return_type)])
+        self._return_schema = StructType([StructField("_0", udf.return_type)])
         self._expected_iter_type = (
             Iterator[pd.DataFrame]
-            if isinstance(self._return_type, StructType)
+            if isinstance(udf.return_type, StructType)
             else Iterator[pd.Series]
         )
 
@@ -182,18 +181,21 @@ class 
PandasScalarIterUDFHandler(BatchEvalTypeHandler["pa.RecordBatch"]):
                 prefer_int_ext_dtype=self._runner_conf.prefer_int_ext_dtype,
                 df_for_struct=True,
             )
-            args = tuple(pandas_columns[o] for o in self._args_offsets)
+            args = tuple(pandas_columns[o] for o in self._udfs[0].args_offsets)
             num_input_rows += batch.num_rows
             return args[0] if len(args) == 1 else args
 
         # Extract args from input batches (streaming), then call the UDF and 
verify the
         # result type (iterator of pd.Series / pd.DataFrame).
         args_iter = map(extract_args, data)
-        verified_iter = verify_return_type(self._udf_func(args_iter), 
self._expected_iter_type)
+        verified_iter = verify_return_type(self._funcs[0](args_iter), 
self._expected_iter_type)
 
         def convert(result: Any) -> "pa.RecordBatch":
             verify_pandas_result(
-                result, self._return_type, assign_cols_by_name=True, 
truncate_return_schema=True
+                result,
+                self._udfs[0].return_type,
+                assign_cols_by_name=True,
+                truncate_return_schema=True,
             )
             return PandasToArrowConversion.from_pandas(
                 [result],
@@ -230,11 +232,10 @@ class 
PandasMapUDFHandler(BatchEvalTypeHandler["pa.RecordBatch"]):
         require_minimum_pyarrow_version()
         super().__init__(udfs, runner_conf, eval_conf)
         assert len(udfs) == 1, "One MAP_PANDAS_ITER UDF expected here."
-        self._udf_func, self._return_type = udfs[0].func, udfs[0].return_type
-        self._return_schema = StructType([StructField("_0", 
self._return_type)])
+        self._return_schema = StructType([StructField("_0", 
udfs[0].return_type)])
         self._expected_iter_type = (
             Iterator[pd.DataFrame]
-            if isinstance(self._return_type, StructType)
+            if isinstance(udfs[0].return_type, StructType)
             else Iterator[pd.Series]
         )
 
@@ -250,7 +251,7 @@ class 
PandasMapUDFHandler(BatchEvalTypeHandler["pa.RecordBatch"]):
                     df_for_struct=True,
                 )[0]
 
-        result = self._udf_func(dataframe_iter())
+        result = self._funcs[0](dataframe_iter())
         # The declared signature is Iterator[...], so a strict iterator is 
required by
         # default. With the legacy flag, accept any object Python can iterate 
over -- via
         # iter(...), which honors both __iter__ and the sequence protocol 
(__getitem__) --
@@ -267,7 +268,7 @@ class 
PandasMapUDFHandler(BatchEvalTypeHandler["pa.RecordBatch"]):
 
         for df in verify_return_type(result, self._expected_iter_type):
             verify_pandas_result(
-                df, self._return_type, assign_cols_by_name=True, 
truncate_return_schema=True
+                df, self._udfs[0].return_type, assign_cols_by_name=True, 
truncate_return_schema=True
             )
             yield PandasToArrowConversion.from_pandas(
                 [df],
@@ -295,13 +296,12 @@ class 
PandasCoGroupedMapUDFHandler(CoGroupedEvalTypeHandler["pa.RecordBatch"]):
         super().__init__(udfs, runner_conf, eval_conf)
         assert len(udfs) == 1, "One COGROUPED_MAP_PANDAS UDF expected here."
         udf = udfs[0]
-        self._cogrouped_udf, self._return_type = udf.func, udf.return_type
         # signature was lost when wrapping it
         self._num_udf_args = len(inspect.getfullargspec(udf.raw_func).args)
         parsed_offsets = extract_key_value_indexes(udf.args_offsets)
         self._left_key_offsets, self._left_value_offsets = parsed_offsets[0]
         self._right_key_offsets, self._right_value_offsets = parsed_offsets[1]
-        self._return_schema = StructType([StructField("_0", 
self._return_type)])
+        self._return_schema = StructType([StructField("_0", udf.return_type)])
 
     def run(self, split_index: int, data: "Iterator[CoGroupedBatch]") -> 
"Iterator[pa.RecordBatch]":
         """Apply cogroupBy Pandas UDF.
@@ -330,7 +330,7 @@ class 
PandasCoGroupedMapUDFHandler(CoGroupedEvalTypeHandler["pa.RecordBatch"]):
             right_df = pd.concat([right_series[o] for o in 
self._right_value_offsets], axis=1)
 
             if self._num_udf_args == 2:
-                result = self._cogrouped_udf(left_df, right_df)
+                result = self._funcs[0](left_df, right_df)
             else:
                 key_series = (
                     [left_series[o] for o in self._left_key_offsets]
@@ -338,14 +338,14 @@ class 
PandasCoGroupedMapUDFHandler(CoGroupedEvalTypeHandler["pa.RecordBatch"]):
                     else [right_series[o] for o in self._right_key_offsets]
                 )
                 key = tuple(s.iloc[0] for s in key_series)
-                result = self._cogrouped_udf(key, left_df, right_df)
+                result = self._funcs[0](key, left_df, right_df)
 
             del left_batches, right_batches, left_table, right_table
             del left_series, right_series, left_df, right_df
 
             verify_pandas_result(
                 result,
-                self._return_type,
+                self._udfs[0].return_type,
                 self._runner_conf.assign_cols_by_name,
                 truncate_return_schema=False,
             )
@@ -378,7 +378,6 @@ class 
PandasGroupedMapIterUDFHandler(GroupedEvalTypeHandler["pa.RecordBatch"]):
         super().__init__(udfs, runner_conf, eval_conf)
         assert len(udfs) == 1, "One GROUPED_MAP_PANDAS_ITER UDF expected here."
         udf = udfs[0]
-        self._grouped_udf, self._return_type = udf.func, udf.return_type
         # signature was lost when wrapping it
         self._num_udf_args = len(inspect.getfullargspec(udf.raw_func).args)
         parsed_offsets = extract_key_value_indexes(udf.args_offsets)
@@ -386,7 +385,7 @@ class 
PandasGroupedMapIterUDFHandler(GroupedEvalTypeHandler["pa.RecordBatch"]):
             "Expected one pair of offsets for GROUPED_MAP_PANDAS_ITER UDF."
         )
         self._key_offsets, self._value_offsets = parsed_offsets[0]
-        self._return_schema = StructType([StructField("_0", 
self._return_type)])
+        self._return_schema = StructType([StructField("_0", udf.return_type)])
 
     def run(self, split_index: int, data: "Iterator[GroupedBatch]") -> 
"Iterator[pa.RecordBatch]":
         """Apply groupBy Pandas UDF (iterator variant)."""
@@ -414,15 +413,15 @@ class 
PandasGroupedMapIterUDFHandler(GroupedEvalTypeHandler["pa.RecordBatch"]):
                     yield pd.concat([series[o] for o in value_offsets], axis=1)
 
             if self._num_udf_args == 1:
-                result = self._grouped_udf(dataframe_iter())
+                result = self._funcs[0](dataframe_iter())
             else:
                 key = tuple(first_series[o].iloc[0] for o in self._key_offsets)
-                result = self._grouped_udf(key, dataframe_iter())
+                result = self._funcs[0](key, dataframe_iter())
 
             for df in result:
                 verify_pandas_result(
                     df,
-                    self._return_type,
+                    self._udfs[0].return_type,
                     self._runner_conf.assign_cols_by_name,
                     truncate_return_schema=False,
                 )
@@ -459,13 +458,12 @@ class 
PandasGroupedMapUDFHandler(GroupedEvalTypeHandler["pa.RecordBatch"]):
         super().__init__(udfs, runner_conf, eval_conf)
         assert len(udfs) == 1, "One GROUPED_MAP_PANDAS UDF expected here."
         udf = udfs[0]
-        self._grouped_udf, self._return_type = udf.func, udf.return_type
         # signature was lost when wrapping it
         self._num_udf_args = len(inspect.getfullargspec(udf.raw_func).args)
         parsed_offsets = extract_key_value_indexes(udf.args_offsets)
         assert len(parsed_offsets) == 1, "Expected one pair of offsets for 
GROUPED_MAP_PANDAS UDF."
         self._key_offsets, self._value_offsets = parsed_offsets[0]
-        self._return_schema = StructType([StructField("_0", 
self._return_type)])
+        self._return_schema = StructType([StructField("_0", udf.return_type)])
 
     def run(self, split_index: int, data: "Iterator[GroupedBatch]") -> 
"Iterator[pa.RecordBatch]":
         # Bound each output RecordBatch toward the byte cap (no limit when 
unset).
@@ -501,16 +499,16 @@ class 
PandasGroupedMapUDFHandler(GroupedEvalTypeHandler["pa.RecordBatch"]):
             value_df = pd.concat([all_series[o] for o in self._value_offsets], 
axis=1)
 
             if self._num_udf_args == 1:
-                result = self._grouped_udf(value_df)
+                result = self._funcs[0](value_df)
             else:
                 key = tuple(all_series[o].iloc[0] for o in self._key_offsets)
-                result = self._grouped_udf(key, value_df)
+                result = self._funcs[0](key, value_df)
 
             del all_batches, table, all_series, value_df
 
             verify_pandas_result(
                 result,
-                self._return_type,
+                self._udfs[0].return_type,
                 self._runner_conf.assign_cols_by_name,
                 truncate_return_schema=False,
             )
@@ -559,11 +557,11 @@ class 
PandasGroupedAggUDFHandler(GroupedEvalTypeHandler["pa.RecordBatch"]):
                 prefer_int_ext_dtype=self._runner_conf.prefer_int_ext_dtype,
             )
             results = [
-                udf.func(
+                func(
                     *[all_series[o] for o in udf.args_offsets],
                     **{k: all_series[v] for k, v in 
udf.kwargs_offsets.items()},
                 )
-                for udf in self._udfs
+                for udf, func in zip(self._udfs, self._funcs)
             ]
             yield PandasToArrowConversion.from_pandas(
                 [pd.Series([r]) for r in results],
@@ -591,13 +589,12 @@ class 
PandasGroupedAggIterUDFHandler(GroupedEvalTypeHandler["pa.RecordBatch"]):
         super().__init__(udfs, runner_conf, eval_conf)
         assert len(udfs) == 1, "One GROUPED_AGG_PANDAS_ITER UDF expected here."
         udf = udfs[0]
-        self._udf_func, self._args_offsets = udf.func, udf.args_offsets
         self._return_schema = StructType([StructField("_0", udf.return_type)])
 
     def run(self, split_index: int, data: "Iterator[GroupedBatch]") -> 
"Iterator[pa.RecordBatch]":
         import pandas as pd
 
-        args_offsets = self._args_offsets
+        args_offsets = self._udfs[0].args_offsets
 
         def extract_series(batch: "pa.RecordBatch") -> Any:
             # Convert one RecordBatch to a pandas Series per column, then 
select args:
@@ -613,7 +610,7 @@ class 
PandasGroupedAggIterUDFHandler(GroupedEvalTypeHandler["pa.RecordBatch"]):
 
         for group in data:
             series_iter = map(extract_series, group)
-            result = self._udf_func(series_iter)
+            result = self._funcs[0](series_iter)
             # Drain remaining batches to maintain stream position
             for _ in series_iter:
                 pass
@@ -663,8 +660,7 @@ class 
PandasWindowAggUDFHandler(GroupedEvalTypeHandler["pa.RecordBatch"]):
             num_rows = table.num_rows
 
             result_series = []
-            for udf_index, udf in enumerate(self._udfs):
-                udf_func = udf.func
+            for udf_index, (udf, udf_func) in enumerate(zip(self._udfs, 
self._funcs)):
                 args_offsets, kwargs_offsets = udf.args_offsets, 
udf.kwargs_offsets
                 bound_type = self._window_bound_types[udf_index]
                 if bound_type == "unbounded":
@@ -730,6 +726,8 @@ class 
ArrowBatchedUDFLegacyPandasHandler(BatchEvalTypeHandler["pa.RecordBatch"])
     eval_type = PythonEvalType.SQL_ARROW_BATCHED_UDF
     variant_conf = "use_legacy_pandas_udf_conversion"
     variant_value = True
+    # run wraps the row-by-row evaluation in fail_on_stopiteration itself.
+    _fail_on_stopiteration = False
 
     def __init__(self, udfs: list[UDF], runner_conf: "RunnerConf", eval_conf: 
"EvalConf") -> None:
         require_minimum_pandas_version()
@@ -738,10 +736,10 @@ class 
ArrowBatchedUDFLegacyPandasHandler(BatchEvalTypeHandler["pa.RecordBatch"])
 
         # --- UDF preparation ---
         self._udf_infos = []
-        for udf in udfs:
+        for udf, func in zip(udfs, self._funcs):
             udf_return_type = udf.return_type
             wrapped_func, args_kwargs_offsets = wrap_kwargs_support(
-                udf.func, udf.args_offsets, udf.kwargs_offsets
+                func, udf.args_offsets, udf.kwargs_offsets
             )
             zero_arg = len(args_kwargs_offsets) == 0
             # Legacy coerces String/Binary for Arrow compatibility
diff --git a/python/pyspark/eval_handlers/_stateful.py 
b/python/pyspark/eval_handlers/_stateful.py
index e8a1f49660ce..87f38371cf6c 100644
--- a/python/pyspark/eval_handlers/_stateful.py
+++ b/python/pyspark/eval_handlers/_stateful.py
@@ -107,7 +107,6 @@ class 
PandasGroupedMapWithStateUDFHandler(BatchEvalTypeHandler["pa.RecordBatch"]
 
         # See FlatMapGroupsInPandasWithStateExec for how arg_offsets are used 
to
         # distinguish between grouping attributes and data attributes.
-        self._func, self._return_type = udfs[0].func, udfs[0].return_type
         parsed_offsets = extract_key_value_indexes(udfs[0].args_offsets)
         self._key_offsets = parsed_offsets[0][0]
         self._value_offsets = parsed_offsets[0][1]
@@ -228,13 +227,13 @@ class 
PandasGroupedMapWithStateUDFHandler(BatchEvalTypeHandler["pa.RecordBatch"]
             # The number of columns of the result must match the return type,
             # but an empty result with no columns at all is acceptable.
             if not (
-                len(result.columns) == len(self._return_type)
+                len(result.columns) == len(self._udfs[0].return_type)
                 or (len(result.columns) == 0 and result.empty)
             ):
                 raise PySparkRuntimeError(
                     errorClass="RESULT_COLUMN_SCHEMA_MISMATCH",
                     messageParameters={
-                        "expected": str(len(self._return_type)),
+                        "expected": str(len(self._udfs[0].return_type)),
                         "actual": str(len(result.columns)),
                     },
                 )
@@ -262,7 +261,7 @@ class 
PandasGroupedMapWithStateUDFHandler(BatchEvalTypeHandler["pa.RecordBatch"]
             else:
                 values = (pd.concat(x, axis=1) for x in value_series_gen)
 
-            result_iter = self._func(key, values, state)
+            result_iter = self._funcs[0](key, values, state)
 
             if isinstance(result_iter, pd.DataFrame):
                 raise PySparkTypeError(
@@ -393,7 +392,11 @@ class 
PandasGroupedMapWithStateUDFHandler(BatchEvalTypeHandler["pa.RecordBatch"]
                     # Flush a batch once the record threshold is exceeded.
                     if pdf_data_cnt > self._arrow_max_records_per_batch:
                         yield construct_record_batch(
-                            pdfs, pdf_data_cnt, self._return_type, state_pdfs, 
state_data_cnt
+                            pdfs,
+                            pdf_data_cnt,
+                            self._udfs[0].return_type,
+                            state_pdfs,
+                            state_data_cnt,
                         )
                         pdfs = []
                         state_pdfs = []
@@ -408,7 +411,7 @@ class 
PandasGroupedMapWithStateUDFHandler(BatchEvalTypeHandler["pa.RecordBatch"]
         # Flush the trailing batch if it has any data or state left.
         if pdf_data_cnt > 0 or state_data_cnt > 0:
             yield construct_record_batch(
-                pdfs, pdf_data_cnt, self._return_type, state_pdfs, 
state_data_cnt
+                pdfs, pdf_data_cnt, self._udfs[0].return_type, state_pdfs, 
state_data_cnt
             )
 
 
@@ -426,7 +429,6 @@ class 
PandasTransformWithStateUDFHandler(BatchEvalTypeHandler["pa.RecordBatch"])
         super().__init__(udfs, runner_conf, eval_conf)
 
         assert len(udfs) == 1, "One TRANSFORM_WITH_STATE_PANDAS UDF expected 
here."
-        self._func, self._return_type = udfs[0].func, udfs[0].return_type
 
         # See TransformWithStateInPandasExec for how arg_offsets are used to
         # distinguish between grouping attributes and data attributes
@@ -437,7 +439,7 @@ class 
PandasTransformWithStateUDFHandler(BatchEvalTypeHandler["pa.RecordBatch"])
 
         self._key_offsets = parsed_offsets[0][0]
         self._value_offsets = parsed_offsets[0][1]
-        self._output_schema = StructType([StructField("_0", 
self._return_type)])
+        self._output_schema = StructType([StructField("_0", 
udfs[0].return_type)])
 
         self._stateful_processor_api_client = StatefulProcessorApiClient(
             eval_conf.state_server_socket_port,  # type: ignore[arg-type]
@@ -515,7 +517,7 @@ class 
PandasTransformWithStateUDFHandler(BatchEvalTypeHandler["pa.RecordBatch"])
 
         def convert_results(result_iter: Iterable[pd.DataFrame]) -> 
Iterator[pa.RecordBatch]:
             for result in result_iter:
-                if isinstance(self._return_type, StructType) and not 
isinstance(
+                if isinstance(self._udfs[0].return_type, StructType) and not 
isinstance(
                     result, pd.DataFrame
                 ):
                     raise PySparkValueError(
@@ -538,7 +540,7 @@ class 
PandasTransformWithStateUDFHandler(BatchEvalTypeHandler["pa.RecordBatch"])
             # This must be a generator expression - do not materialize.
             values_gen = (df.iloc[:, self._value_offsets] for _, df in group)
             yield from convert_results(
-                self._func(
+                self._funcs[0](
                     self._stateful_processor_api_client,
                     TransformWithStateInPandasFuncMode.PROCESS_DATA,
                     key,
@@ -547,7 +549,7 @@ class 
PandasTransformWithStateUDFHandler(BatchEvalTypeHandler["pa.RecordBatch"])
             )
 
         yield from convert_results(
-            self._func(
+            self._funcs[0](
                 self._stateful_processor_api_client,
                 TransformWithStateInPandasFuncMode.PROCESS_TIMER,
                 None,
@@ -556,7 +558,7 @@ class 
PandasTransformWithStateUDFHandler(BatchEvalTypeHandler["pa.RecordBatch"])
         )
 
         yield from convert_results(
-            self._func(
+            self._funcs[0](
                 self._stateful_processor_api_client,
                 TransformWithStateInPandasFuncMode.COMPLETE,
                 None,
@@ -579,7 +581,6 @@ class 
PandasTransformWithStateInitStateUDFHandler(BatchEvalTypeHandler["pa.Recor
         super().__init__(udfs, runner_conf, eval_conf)
 
         assert len(udfs) == 1, "One TRANSFORM_WITH_STATE_PANDAS_INIT_STATE UDF 
expected here."
-        self._func, self._return_type = udfs[0].func, udfs[0].return_type
 
         # See TransformWithStateInPandasExec for how arg_offsets are used to
         # distinguish between grouping attributes and data attributes.
@@ -591,7 +592,7 @@ class 
PandasTransformWithStateInitStateUDFHandler(BatchEvalTypeHandler["pa.Recor
         parsed_offsets = extract_key_value_indexes(udfs[0].args_offsets)
         self._key_offsets = parsed_offsets[0][0]
         self._init_key_offsets = parsed_offsets[1][0]
-        self._output_schema = StructType([StructField("_0", 
self._return_type)])
+        self._output_schema = StructType([StructField("_0", 
udfs[0].return_type)])
 
         self._stateful_processor_api_client = StatefulProcessorApiClient(
             eval_conf.state_server_socket_port,  # type: ignore[arg-type]
@@ -728,7 +729,7 @@ class 
PandasTransformWithStateInitStateUDFHandler(BatchEvalTypeHandler["pa.Recor
             # TODO(SPARK-49100): add verification that elements in result_iter 
are
             # indeed of type pd.DataFrame and conform to assigned cols
             for result in result_iter:
-                if isinstance(self._return_type, StructType) and not 
isinstance(
+                if isinstance(self._udfs[0].return_type, StructType) and not 
isinstance(
                     result, pd.DataFrame
                 ):
                     raise PySparkValueError(
@@ -755,7 +756,7 @@ class 
PandasTransformWithStateInitStateUDFHandler(BatchEvalTypeHandler["pa.Recor
             state_values = (data_df for _, data_df, _ in group_data if not 
data_df.empty)
             init_states = (init_df for _, _, init_df in group_init if not 
init_df.empty)
             yield from convert_results(
-                self._func(
+                self._funcs[0](
                     self._stateful_processor_api_client,
                     TransformWithStateInPandasFuncMode.PROCESS_DATA,
                     key,
@@ -765,7 +766,7 @@ class 
PandasTransformWithStateInitStateUDFHandler(BatchEvalTypeHandler["pa.Recor
             )
 
         yield from convert_results(
-            self._func(
+            self._funcs[0](
                 self._stateful_processor_api_client,
                 TransformWithStateInPandasFuncMode.PROCESS_TIMER,
                 None,
@@ -775,7 +776,7 @@ class 
PandasTransformWithStateInitStateUDFHandler(BatchEvalTypeHandler["pa.Recor
         )
 
         yield from convert_results(
-            self._func(
+            self._funcs[0](
                 self._stateful_processor_api_client,
                 TransformWithStateInPandasFuncMode.COMPLETE,
                 None,
@@ -798,7 +799,6 @@ class 
RowTransformWithStateUDFHandler(BatchEvalTypeHandler["pa.RecordBatch"]):
         super().__init__(udfs, runner_conf, eval_conf)
 
         assert len(udfs) == 1, "One TRANSFORM_WITH_STATE_PYTHON_ROW UDF 
expected here."
-        self._func, self._return_type = udfs[0].func, udfs[0].return_type
 
         # See TransformWithStateInPySparkExec for how arg_offsets are used to
         # distinguish between grouping attributes and data attributes.
@@ -850,7 +850,7 @@ class 
RowTransformWithStateUDFHandler(BatchEvalTypeHandler["pa.RecordBatch"]):
             # Convert spark type to arrow type
             # TODO: we need to make this configurable, currently using default 
values.
             arrow_type = to_arrow_type(
-                self._return_type,
+                self._udfs[0].return_type,
                 timezone="UTC",
                 prefers_large_types=False,
             )
@@ -865,7 +865,7 @@ class 
RowTransformWithStateUDFHandler(BatchEvalTypeHandler["pa.RecordBatch"]):
             # This must be a generator expression - do not materialize.
             values_gen = map(lambda x: x[1], group)
             yield from convert_results(
-                self._func(
+                self._funcs[0](
                     self._stateful_processor_api_client,
                     TransformWithStateInPandasFuncMode.PROCESS_DATA,
                     key,
@@ -874,7 +874,7 @@ class 
RowTransformWithStateUDFHandler(BatchEvalTypeHandler["pa.RecordBatch"]):
             )
 
         yield from convert_results(
-            self._func(
+            self._funcs[0](
                 self._stateful_processor_api_client,
                 TransformWithStateInPandasFuncMode.PROCESS_TIMER,
                 None,
@@ -883,7 +883,7 @@ class 
RowTransformWithStateUDFHandler(BatchEvalTypeHandler["pa.RecordBatch"]):
         )
 
         yield from convert_results(
-            self._func(
+            self._funcs[0](
                 self._stateful_processor_api_client,
                 TransformWithStateInPandasFuncMode.COMPLETE,
                 None,
@@ -905,7 +905,6 @@ class 
RowTransformWithStateInitStateUDFHandler(BatchEvalTypeHandler["pa.RecordBa
         super().__init__(udfs, runner_conf, eval_conf)
 
         assert len(udfs) == 1, "One TRANSFORM_WITH_STATE_PYTHON_ROW_INIT_STATE 
UDF expected here."
-        self._func, self._return_type = udfs[0].func, udfs[0].return_type
 
         # See TransformWithStateInPandasExec for how arg_offsets are used to
         # distinguish between grouping attributes and data attributes.
@@ -1031,7 +1030,7 @@ class 
RowTransformWithStateInitStateUDFHandler(BatchEvalTypeHandler["pa.RecordBa
             # Convert spark type to arrow type
             # TODO: we need to make this configurable, currently using default 
values.
             arrow_type = to_arrow_type(
-                self._return_type,
+                self._udfs[0].return_type,
                 timezone="UTC",
                 prefers_large_types=False,
             )
@@ -1046,7 +1045,7 @@ class 
RowTransformWithStateInitStateUDFHandler(BatchEvalTypeHandler["pa.RecordBa
             # These must be generator expressions - do not materialize.
             for _, (values_gen, init_states_gen) in group:
                 yield from convert_results(
-                    self._func(
+                    self._funcs[0](
                         self._stateful_processor_api_client,
                         TransformWithStateInPandasFuncMode.PROCESS_DATA,
                         key,
@@ -1056,7 +1055,7 @@ class 
RowTransformWithStateInitStateUDFHandler(BatchEvalTypeHandler["pa.RecordBa
                 )
 
         yield from convert_results(
-            self._func(
+            self._funcs[0](
                 self._stateful_processor_api_client,
                 TransformWithStateInPandasFuncMode.PROCESS_TIMER,
                 None,
@@ -1066,7 +1065,7 @@ class 
RowTransformWithStateInitStateUDFHandler(BatchEvalTypeHandler["pa.RecordBa
         )
 
         yield from convert_results(
-            self._func(
+            self._funcs[0](
                 self._stateful_processor_api_client,
                 TransformWithStateInPandasFuncMode.COMPLETE,
                 None,
diff --git a/python/pyspark/eval_handlers/_typing.py 
b/python/pyspark/eval_handlers/_typing.py
index 8d533bd1278f..d01180bd5544 100644
--- a/python/pyspark/eval_handlers/_typing.py
+++ b/python/pyspark/eval_handlers/_typing.py
@@ -21,14 +21,15 @@ The Arrow element types are forward refs so pyarrow stays a 
type-checking-only
 import at runtime, as elsewhere in the worker.
 """
 
-from collections.abc import Callable, Iterator
+from collections.abc import Iterator
 from dataclasses import dataclass
-from typing import TYPE_CHECKING, Any, Protocol, TypeVar
+from typing import TYPE_CHECKING, Any, Optional, Protocol, TypeVar
 
 if TYPE_CHECKING:
     import pyarrow as pa
 
     from pyspark.sql.types import StructType
+    from pyspark.sql.udtf import AnalyzeResult
 
 # Grouped-category input element, matching what the serializer yields: a group
 # serializer yields a lazy iterator; a co-group serializer materializes each 
side.
@@ -42,19 +43,19 @@ OutputBatch = TypeVar("OutputBatch")
 
 @dataclass(frozen=True)
 class UDF:
-    """One UDF as read by ``read_single_udf``; each handler takes the fields 
it needs."""
+    """One UDF as the worker reads it; each handler takes the fields it 
needs."""
 
-    # The chained function wrapped for profiling and, for most eval types, in
-    # fail_on_stopiteration.
-    func: Callable[..., Any]
-    # The chained function before any wrapping, for handlers that need the 
pickled object
-    # itself (an incremental Aggregator) or its signature.
+    # The chained function before any wrapping: the pickled object itself (an 
incremental
+    # Aggregator) with its signature. EvalTypeHandler.__init__ keeps it 
wrapped for profiling and,
+    # for most eval types, in fail_on_stopiteration, as the handler's _funcs.
     raw_func: Any
     args_offsets: list[int]
     kwargs_offsets: dict[str, int]
     # The DataType of the last function in the chain; a StructType for the 
eval types that
     # return rows.
     return_type: Any
+    # The id the profiler reports the UDF's results under.
+    result_id: int
 
 
 class UDTFInstance(Protocol):
@@ -66,11 +67,17 @@ class UDTFInstance(Protocol):
 
 @dataclass(frozen=True)
 class UDTF:
-    """The UDTF as read by ``read_single_udtf``."""
+    """The UDTF as the worker reads it; ``UDTFHandlerBase.__init__`` 
instantiates it as the
+    handler's ``_udtf``."""
 
-    # Wrapped in the handler's partition_wrapper when its TABLE argument has 
PARTITION BY
-    # expressions.
-    udtf: UDTFInstance
+    # The user's UDTF class, as unpickled.
+    udtf_cls: type
     args_offsets: list[int]
     kwargs_offsets: dict[str, int]
     return_type: "StructType"
+    # The child indexes of the TABLE argument's PARTITION BY expressions, if 
any.
+    partition_child_indexes: list[int]
+    # Passed to the UDTF's __init__ when it takes one argument.
+    analyze_result: Optional["AnalyzeResult"]
+    # The UDTF's name in error messages.
+    name: str
diff --git a/python/pyspark/eval_handlers/_udtf.py 
b/python/pyspark/eval_handlers/_udtf.py
index c17b7e82653f..2b1b67b348e2 100644
--- a/python/pyspark/eval_handlers/_udtf.py
+++ b/python/pyspark/eval_handlers/_udtf.py
@@ -18,8 +18,8 @@
 """Handlers for the user-defined table function (UDTF) eval types, and the 
wrappers that
 implement TABLE argument PARTITION BY semantics on top of a UDTF instance.
 
-Each handler receives the single ``UDTF`` record ``read_single_udtf`` returns, 
which holds
-the instantiated UDTF.
+Each handler receives the single ``UDTF`` record the worker reads, and 
instantiates the
+UDTF from it.
 
 pyarrow is imported lazily (inside the handlers and the type-checking block) 
so the module
 stays importable and its handlers register without pyarrow installed.
@@ -27,6 +27,8 @@ stays importable and its handlers register without pyarrow 
installed.
 
 from __future__ import annotations
 
+import dataclasses
+import inspect
 import itertools
 from abc import ABCMeta, abstractmethod
 from collections.abc import Iterable, Iterator
@@ -34,10 +36,14 @@ from typing import TYPE_CHECKING, Any, Callable, ClassVar, 
Optional, Tuple
 
 from pyspark.errors import PySparkRuntimeError, PySparkTypeError, 
PySparkValueError
 from pyspark.eval_handlers._base import BatchEvalTypeHandler, EvalTypeHandler
-from pyspark.eval_handlers._typing import UDTF, InputBatch, OutputBatch
+from pyspark.eval_handlers._typing import UDTF, InputBatch, OutputBatch, 
UDTFInstance
 from pyspark.eval_handlers.utils import wrap_kwargs_support
 from pyspark.eval_handlers.verification import verify_arrow_result, 
verify_pandas_result
-from pyspark.serializers import BatchedSerializer, CPickleSerializer, 
Serializer
+from pyspark.serializers import (
+    BatchedSerializer,
+    CPickleSerializer,
+    Serializer,
+)
 from pyspark.sql.conversion import (
     ArrowBatchTransformer,
     ArrowTableToRowsConversion,
@@ -592,27 +598,101 @@ def build_null_checker(return_type: StructType) -> 
Optional[Callable[[Any], None
     return check
 
 
-class UDTFEvalTypeHandler(EvalTypeHandler[InputBatch, OutputBatch]):
-    """Base for the UDTF eval types.
+class UDTFHandlerBase(EvalTypeHandler[InputBatch, OutputBatch]):
+    """Base for every UDTF eval type handler; ``__init__`` creates the UDTF 
instance as
+    ``_udtf``."""
+
+    # Wraps the UDTF when its TABLE argument has PARTITION BY expressions.
+    partition_wrapper: ClassVar[type[PartitionedUDTF]] = UDTFWithPartitions
+
+    def __init__(self, udfs: list[UDTF], runner_conf: RunnerConf, eval_conf: 
EvalConf) -> None:
+        """Create, as ``_udtf``, the instance of the UDTF's class (wrapped in
+        ``partition_wrapper`` when its TABLE argument has PARTITION BY 
expressions, its
+        constructor given the AnalyzeResult when it takes one), and check its 
'eval' signature
+        against the call arguments."""
+        super().__init__(udfs, runner_conf, eval_conf)
+        assert len(udfs) == 1, "One UDTF expected here."
+        udtf = udfs[0]
+        if not isinstance(udtf.udtf_cls, type):
+            raise PySparkRuntimeError(
+                f"Invalid UDTF handler type. Expected a class (type 'type'), 
but "
+                f"got an instance of {type(udtf.udtf_cls).__name__}."
+            )
+
+        # Pick how to construct the UDTF from the count of named positional 
__init__ parameters
+        # after self. With an AnalyzeResult, one parameter receives a copy of 
it, zero means the
+        # class is called with no arguments, and more are an error; without 
one, any is an error.
+        handler: Callable[[], Any] = udtf.udtf_cls
+        udtf_init_args = inspect.getfullargspec(udtf.udtf_cls)
+        if udtf.analyze_result is not None:
+            if len(udtf_init_args.args) > 2:
+                raise PySparkRuntimeError(
+                    
errorClass="UDTF_CONSTRUCTOR_INVALID_IMPLEMENTS_ANALYZE_METHOD",
+                    messageParameters={"name": udtf.name},
+                )
+            elif len(udtf_init_args.args) == 2:
+                analyze_result = udtf.analyze_result
+
+                def construct_udtf() -> Any:
+                    # Here we pass the AnalyzeResult to the UDTF's __init__ 
method.
+                    return udtf.udtf_cls(dataclasses.replace(analyze_result))
+
+                handler = construct_udtf
+        elif len(udtf_init_args.args) > 1:
+            raise PySparkRuntimeError(
+                errorClass="UDTF_CONSTRUCTOR_INVALID_NO_ANALYZE_METHOD",
+                messageParameters={"name": udtf.name},
+            )
+
+        # Instantiate the UDTF class.
+        try:
+            if len(udtf.partition_child_indexes) > 0:
+                instance = self.partition_wrapper(handler, 
udtf.partition_child_indexes)
+            else:
+                instance = handler()
+        except Exception as e:
+            raise PySparkRuntimeError(
+                errorClass="UDTF_EXEC_ERROR",
+                messageParameters={"method_name": "__init__", "error": str(e)},
+            )
+
+        # Validate the UDTF
+        if not hasattr(instance, "eval"):
+            raise PySparkRuntimeError(
+                "Failed to execute the user defined table function because it 
has not "
+                "implemented the 'eval' method. Please add the 'eval' method 
and try "
+                "the query again."
+            )
+
+        # Check that the arguments provided to the UDTF call match the 
expected parameters
+        # defined in the 'eval' method signature.
+        try:
+            inspect.signature(instance.eval).bind(*udtf.args_offsets, 
**udtf.kwargs_offsets)
+        except TypeError as e:
+            raise PySparkRuntimeError(
+                errorClass="UDTF_EVAL_METHOD_ARGUMENTS_DO_NOT_MATCH_SIGNATURE",
+                messageParameters={"name": udtf.name, "reason": str(e)},
+            ) from None
+
+        self._udtf: UDTFInstance = instance
+
+
+class UDTFEvalTypeHandler(UDTFHandlerBase[InputBatch, OutputBatch]):
+    """Base for the Arrow UDTF eval types.
 
     ``run`` calls ``_eval_batch`` per input batch, then ``_eval_terminate`` 
once at the end (also
     when ``eval`` raised ``SkipRestOfInputTableException``), and always calls 
the UDTF's
     ``cleanup``. Subclasses implement the two hooks and pick the serializer.
     """
 
-    # Wraps the UDTF when its TABLE argument has PARTITION BY expressions; see 
read_single_udtf.
-    partition_wrapper: ClassVar[type[PartitionedUDTF]] = UDTFWithPartitions
-
     def __init__(self, udfs: list[UDTF], runner_conf: RunnerConf, eval_conf: 
EvalConf) -> None:
         super().__init__(udfs, runner_conf, eval_conf)
         assert len(udfs) == 1, "One UDTF expected here."
-        udtf = udfs[0].udtf
-        self._return_type = udfs[0].return_type
         self._eval_method, self._args_kwargs_offsets = wrap_kwargs_support(
-            udtf.eval, udfs[0].args_offsets, udfs[0].kwargs_offsets
+            self._udtf.eval, udfs[0].args_offsets, udfs[0].kwargs_offsets
         )
-        self._terminate_method = getattr(udtf, "terminate", None)
-        self._cleanup_method = getattr(udtf, "cleanup", None)
+        self._terminate_method = getattr(self._udtf, "terminate", None)
+        self._cleanup_method = getattr(self._udtf, "cleanup", None)
 
     @abstractmethod
     def _eval_batch(self, batch: InputBatch) -> Iterator[OutputBatch]:
@@ -638,33 +718,30 @@ class UDTFEvalTypeHandler(EvalTypeHandler[InputBatch, 
OutputBatch]):
                 cleanup()
 
 
-class TableUDTFHandler(EvalTypeHandler[Any, tuple]):
+class TableUDTFHandler(UDTFHandlerBase[Any, tuple]):
     """SQL_TABLE_UDF, the Python UDTF: input rows arrive pickled one at a time 
and the UDTF's
     ``eval`` is called once per row; the results of each call are converted to 
the internal
     representation and returned to the executor as one tuple."""
 
     eval_type = PythonEvalType.SQL_TABLE_UDF
-    # Wraps the UDTF when its TABLE argument has PARTITION BY expressions; see 
read_single_udtf.
-    partition_wrapper: ClassVar[type[PartitionedUDTF]] = UDTFWithPartitions
 
     def __init__(self, udfs: list[UDTF], runner_conf: RunnerConf, eval_conf: 
EvalConf) -> None:
         super().__init__(udfs, runner_conf, eval_conf)
         assert len(udfs) == 1, "One UDTF expected here."
-        udtf, return_type = udfs[0].udtf, udfs[0].return_type
-        self._check_output_row_against_schema = build_null_checker(return_type)
+        self._check_output_row_against_schema = 
build_null_checker(udfs[0].return_type)
 
         eval_func_kwargs_support, self._args_kwargs_offsets = 
wrap_kwargs_support(
-            udtf.eval, udfs[0].args_offsets, udfs[0].kwargs_offsets
+            self._udtf.eval, udfs[0].args_offsets, udfs[0].kwargs_offsets
         )
-        self._eval = self._wrap_udtf(eval_func_kwargs_support, return_type)
+        self._eval = self._wrap_udtf(eval_func_kwargs_support, 
udfs[0].return_type)
 
         self._terminate: Optional[Callable[..., tuple]]
-        if hasattr(udtf, "terminate"):
-            self._terminate = self._wrap_udtf(getattr(udtf, "terminate"), 
return_type)
+        if hasattr(self._udtf, "terminate"):
+            self._terminate = self._wrap_udtf(getattr(self._udtf, 
"terminate"), udfs[0].return_type)
         else:
             self._terminate = None
 
-        self._cleanup = getattr(udtf, "cleanup") if hasattr(udtf, "cleanup") 
else None
+        self._cleanup = getattr(self._udtf, "cleanup") if hasattr(self._udtf, 
"cleanup") else None
 
     @property
     def serializer(self) -> Serializer:
@@ -778,9 +855,11 @@ class ArrowUDTFHandler(
 
         super().__init__(udfs, runner_conf, eval_conf)
         arrow_return_type = to_arrow_type(
-            self._return_type, timezone="UTC", 
prefers_large_types=runner_conf.use_large_var_types
+            self._udfs[0].return_type,
+            timezone="UTC",
+            prefers_large_types=runner_conf.use_large_var_types,
         )
-        self._return_type_size = len(self._return_type)
+        self._return_type_size = len(self._udfs[0].return_type)
         self._target_schema = pa.schema(list(arrow_return_type))
 
         self._table_arg_offsets = (
@@ -885,8 +964,8 @@ class ArrowTableUDTFHandler(
 
     def __init__(self, udfs: list[UDTF], runner_conf: RunnerConf, eval_conf: 
EvalConf) -> None:
         super().__init__(udfs, runner_conf, eval_conf)
-        self._return_type_size = len(self._return_type)
-        self._check_output_row_against_schema = 
build_null_checker(self._return_type)
+        self._return_type_size = len(self._udfs[0].return_type)
+        self._check_output_row_against_schema = 
build_null_checker(self._udfs[0].return_type)
 
     @abstractmethod
     def _to_columns(self, batch: pa.RecordBatch) -> list:
@@ -941,7 +1020,7 @@ class 
ArrowTableUDTFLegacyPandasHandler(ArrowTableUDTFHandler):
         super().__init__(udfs, runner_conf, eval_conf)
         # The output pandas DataFrame is converted as a single struct column 
named
         # "_0" against this schema.
-        self._output_schema = StructType([StructField("_0", 
self._return_type)])
+        self._output_schema = StructType([StructField("_0", 
self._udfs[0].return_type)])
 
     def _verify_result(self, result: Any, method_name: str) -> Any:
         import pandas as pd
@@ -973,7 +1052,10 @@ class 
ArrowTableUDTFLegacyPandasHandler(ArrowTableUDTFHandler):
 
         # Verify the type and the schema of the result.
         verify_pandas_result(
-            result, self._return_type, assign_cols_by_name=False, 
truncate_return_schema=False
+            result,
+            self._udfs[0].return_type,
+            assign_cols_by_name=False,
+            truncate_return_schema=False,
         )
         return result
 
@@ -1050,7 +1132,9 @@ class ArrowTableUDTFArrowHandler(ArrowTableUDTFHandler):
 
         super().__init__(udfs, runner_conf, eval_conf)
         self._arrow_return_type = to_arrow_type(
-            self._return_type, timezone="UTC", 
prefers_large_types=runner_conf.use_large_var_types
+            self._udfs[0].return_type,
+            timezone="UTC",
+            prefers_large_types=runner_conf.use_large_var_types,
         )
         self._converters = [
             ArrowTableToRowsConversion._create_converter(
@@ -1133,14 +1217,14 @@ class ArrowTableUDTFArrowHandler(ArrowTableUDTFHandler):
                 errorClass="UDTF_ARROW_DATA_CONVERSION_ERROR",
                 messageParameters={
                     "data": str(data),
-                    "schema": self._return_type.simpleString(),
+                    "schema": self._udfs[0].return_type.simpleString(),
                     "arrow_schema": str(arrow_return_type),
                 },
             ) from original_exception
 
         try:
             table = LocalDataToArrowConversion.convert(
-                data, self._return_type, self._runner_conf.use_large_var_types
+                data, self._udfs[0].return_type, 
self._runner_conf.use_large_var_types
             )
         except PySparkValueError as e:
             if e.getErrorClass() == "AXIS_LENGTH_MISMATCH":
diff --git a/python/pyspark/eval_handlers/_vanilla.py 
b/python/pyspark/eval_handlers/_vanilla.py
index 2ad0e4767fe5..b20da0f2c839 100644
--- a/python/pyspark/eval_handlers/_vanilla.py
+++ b/python/pyspark/eval_handlers/_vanilla.py
@@ -45,15 +45,13 @@ class BatchedUDFHandler(EvalTypeHandler[Any, Any]):
 
     def __init__(self, udfs: "list[UDF]", runner_conf: "RunnerConf", 
eval_conf: "EvalConf") -> None:
         super().__init__(udfs, runner_conf, eval_conf)
-        self._udf_infos = [self._prepare_udf(udf) for udf in udfs]
+        self._udf_infos = [self._prepare_udf(udf, func) for udf, func in 
zip(udfs, self._funcs)]
 
     @staticmethod
-    def _prepare_udf(udf: "UDF") -> tuple[list[int], Callable[..., Any]]:
+    def _prepare_udf(udf: "UDF", func: Callable[..., Any]) -> tuple[list[int], 
Callable[..., Any]]:
         """Apply kwargs binding, and convert each result to the internal 
representation only
         when the return type requires it."""
-        func, args_kwargs_offsets = wrap_kwargs_support(
-            udf.func, udf.args_offsets, udf.kwargs_offsets
-        )
+        func, args_kwargs_offsets = wrap_kwargs_support(func, 
udf.args_offsets, udf.kwargs_offsets)
         if udf.return_type.needConversion():
             toInternal = udf.return_type.toInternal
             return args_kwargs_offsets, lambda *a: toInternal(func(*a))
diff --git 
a/python/pyspark/eval_handlers/tests/test_arrow_eval_type_handlers.py 
b/python/pyspark/eval_handlers/tests/test_arrow_eval_type_handlers.py
index 63ce1c546314..1c29e79b98d4 100644
--- a/python/pyspark/eval_handlers/tests/test_arrow_eval_type_handlers.py
+++ b/python/pyspark/eval_handlers/tests/test_arrow_eval_type_handlers.py
@@ -34,7 +34,7 @@ from pyspark.eval_handlers.utils import hashable_grouping_key
 from pyspark.sql.pandas.serializers import ArrowStreamCoGroupSerializer, 
ArrowStreamSerializer
 from pyspark.sql.types import DoubleType, LongType, StringType, StructField, 
StructType
 from pyspark.testing.utils import PySparkBaseTestCase, have_pyarrow, 
pyarrow_requirement_message
-from pyspark.util import PythonEvalType, fail_on_stopiteration
+from pyspark.util import PythonEvalType
 from pyspark.worker_metrics import WorkerMetrics
 
 with patch.dict(os.environ, {"SPARK_PYTHON_RUNTIME": "PYTHON_WORKER"}):
@@ -125,27 +125,23 @@ class ArrowEvalTypeHandlerTestCase(PySparkBaseTestCase):
         return offsets
 
     @staticmethod
-    def _udf(func, args_offsets, kwargs_offsets=None, return_type=None, 
raw_func=None):
-        """A ``UDF`` record; ``raw_func`` defaults to ``func``."""
+    def _udf(func, args_offsets, kwargs_offsets=None, return_type=None):
+        """A ``UDF`` record as the worker builds it."""
         return UDF(
-            func=func,
-            raw_func=func if raw_func is None else raw_func,
+            raw_func=func,
             args_offsets=args_offsets,
             kwargs_offsets=kwargs_offsets or {},
             return_type=return_type or LongType(),
+            result_id=0,
         )
 
     @classmethod
     def _grouped_handler(cls, handler_cls, udf, arg_offsets):
-        """Build a grouped/cogrouped-map handler with the shared return type. 
As from
-        ``read_single_udf``, ``func`` is wrapped (losing the signature) while 
``raw_func``
-        keeps it."""
+        """Build a grouped/cogrouped-map handler with the shared return type. 
The function the
+        handler wraps loses its signature, so the handler reads it from the 
record's
+        ``raw_func``."""
         return handler_cls(
-            udfs=[
-                cls._udf(
-                    lambda *a: udf(*a), arg_offsets, 
return_type=cls._RETURN_TYPE, raw_func=udf
-                )
-            ],
+            udfs=[cls._udf(udf, arg_offsets, return_type=cls._RETURN_TYPE)],
             runner_conf=RunnerConf({}),
             eval_conf=None,
         )
@@ -609,10 +605,9 @@ class 
ArrowGroupedAggIncrementalPartialUDFHandlerTests(ArrowEvalTypeHandlerTestC
     _KEY_SCHEMA = StructType([StructField("k", LongType())])
 
     def _handler(self, cap=10000):
-        # As from read_single_udf: func is wrapped, raw_func is the Aggregator 
itself.
-        agg = _SumAggregator()
+        # The handler takes the Aggregator itself from the record's raw_func.
         return ArrowGroupedAggIncrementalPartialUDFHandler(
-            udfs=[self._udf(fail_on_stopiteration(agg), [1], raw_func=agg)],
+            udfs=[self._udf(_SumAggregator(), [1])],
             
runner_conf=RunnerConf({"spark.sql.execution.arrow.maxRecordsPerBatch": 
str(cap)}),
             eval_conf=EvalConf({"grouping_key_schema": 
self._KEY_SCHEMA.json()}),
         )
@@ -669,9 +664,8 @@ class 
ArrowGroupedAggIncrementalPartialUDFHandlerTests(ArrowEvalTypeHandlerTestC
             [pa.array([float("nan"), float("nan")]), pa.array([1, 2], 
type=pa.int64())],
             ["k", "v"],
         )
-        agg = _SumAggregator()
         handler = ArrowGroupedAggIncrementalPartialUDFHandler(
-            udfs=[self._udf(fail_on_stopiteration(agg), [1], raw_func=agg)],
+            udfs=[self._udf(_SumAggregator(), [1])],
             runner_conf=RunnerConf({}),
             eval_conf=EvalConf(
                 {"grouping_key_schema": StructType([StructField("k", 
DoubleType())]).json()}
@@ -692,12 +686,9 @@ class 
ArrowGroupedAggIncrementalFinalUDFHandlerTests(ArrowEvalTypeHandlerTestCas
         return pa.RecordBatch.from_arrays([pa.array(buffers, 
type=self._BUFFER_TYPE)], ["buf"])
 
     def _handler(self):
-        # As from read_single_udf: func is wrapped, raw_func is the Aggregator 
itself.
-        agg = _SumAggregator()
+        # The handler takes the Aggregator itself from the record's raw_func.
         return ArrowGroupedAggIncrementalFinalUDFHandler(
-            udfs=[self._udf(fail_on_stopiteration(agg), [0], raw_func=agg)],
-            runner_conf=RunnerConf({}),
-            eval_conf=None,
+            udfs=[self._udf(_SumAggregator(), [0])], 
runner_conf=RunnerConf({}), eval_conf=None
         )
 
     def test_merges_partial_buffers_per_group(self):
@@ -722,8 +713,7 @@ class 
ArrowGroupedAggIncrementalFinalUDFHandlerTests(ArrowEvalTypeHandlerTestCas
 class ArrowWindowAggIncrementalUDFHandlerTests(ArrowEvalTypeHandlerTestCase):
     def _handler(self, agg, bound_type, args_offsets):
         return ArrowWindowAggIncrementalUDFHandler(
-            # As from read_single_udf: func is wrapped, raw_func is the 
Aggregator itself.
-            udfs=[self._udf(fail_on_stopiteration(agg), args_offsets, 
raw_func=agg)],
+            udfs=[self._udf(agg, args_offsets)],
             runner_conf=RunnerConf({"window_bound_types": bound_type}),
             eval_conf=None,
         )
diff --git a/python/pyspark/eval_handlers/tests/test_base_eval_type_handlers.py 
b/python/pyspark/eval_handlers/tests/test_base_eval_type_handlers.py
index fdd0d76ebaf0..ba611ba866ab 100644
--- a/python/pyspark/eval_handlers/tests/test_base_eval_type_handlers.py
+++ b/python/pyspark/eval_handlers/tests/test_base_eval_type_handlers.py
@@ -22,7 +22,6 @@ Handler-flavor tests live alongside their module, e.g. the 
Arrow handlers in
 """
 
 import os
-from types import SimpleNamespace
 from unittest.mock import patch
 
 from pyspark.errors import PySparkRuntimeError
@@ -41,13 +40,12 @@ from pyspark.sql.pandas.serializers import (
     ArrowStreamGroupSerializer,
     ArrowStreamSerializer,
 )
-from pyspark.sql.types import LongType
+from pyspark.sql.types import LongType, StringType
 from pyspark.testing.utils import PySparkBaseTestCase
 from pyspark.util import PythonEvalType
 
 with patch.dict(os.environ, {"SPARK_PYTHON_RUNTIME": "PYTHON_WORKER"}):
-    from pyspark.worker import read_single_udf
-    from pyspark.worker_message import UDFInfo
+    from pyspark.worker import unpickle_and_chain
     from pyspark.worker_util import RunnerConf, pickleSer
 
 
@@ -272,31 +270,110 @@ class EvalTypeHandlerTests(PySparkBaseTestCase):
         self.assertNotIn(unused_eval_type, _eval_type_handlers)
 
 
-class ReadSingleUDFTests(PySparkBaseTestCase):
-    def test_returns_udf_record(self):
-        def stop(x):
-            raise StopIteration(x)
-
-        udf_info = UDFInfo(
-            udfs=[CloudPickleSerializer().dumps((stop, LongType()))],
-            args=[1],
-            kwargs={"y": 2},
-            result_id=0,
+class HandlerUDFsTests(PySparkBaseTestCase):
+    class _Handler(BatchEvalTypeHandler["pa.RecordBatch"]):
+        def run(self, split_index, data):
+            return data
+
+    @staticmethod
+    def _stop(x):
+        raise StopIteration(x)
+
+    def _new(self, handler_cls, runner_conf=None):
+        return handler_cls(
+            [
+                UDF(
+                    raw_func=self._stop,
+                    args_offsets=[1],
+                    kwargs_offsets={"y": 2},
+                    return_type=LongType(),
+                    result_id=0,
+                )
+            ],
+            runner_conf or RunnerConf({}),
+            None,
         )
-        # read_single_udf only reads these two attributes of the handler class.
-        handler_cls = 
SimpleNamespace(eval_type=PythonEvalType.SQL_BATCHED_UDF, is_iter_based=False)
-        udf = read_single_udf(pickleSer, udf_info, handler_cls, RunnerConf({}))
+
+    def test_wraps_udfs(self):
+        handler = self._new(self._Handler)
+        udf = handler._udfs[0]
 
         self.assertIsInstance(udf, UDF)
         self.assertEqual(udf.args_offsets, [1])
         self.assertEqual(udf.kwargs_offsets, {"y": 2})
         self.assertEqual(udf.return_type, LongType())
-        # func is wrapped in fail_on_stopiteration; raw_func is the unwrapped 
function.
+        # The handler keeps the function wrapped in fail_on_stopiteration; the 
record keeps it
+        # unwrapped.
         with self.assertRaises(PySparkRuntimeError):
-            udf.func(1)
+            handler._funcs[0](1)
         with self.assertRaises(StopIteration):
             udf.raw_func(1)
 
+    def test_fail_on_stopiteration_per_handler(self):
+        not_wrapped = {
+            PythonEvalType.SQL_SCALAR_PANDAS_ITER_UDF,
+            PythonEvalType.SQL_SCALAR_PANDAS_ITER_ELEMENTWISE_UDF,
+            PythonEvalType.SQL_ARROW_BATCHED_UDF,
+        }
+        for _, variants in list(_eval_type_handlers.values()):
+            for handler_cls in variants.values():
+                with self.subTest(handler=handler_cls.__name__):
+                    self.assertEqual(
+                        handler_cls._fail_on_stopiteration,
+                        handler_cls.eval_type not in not_wrapped,
+                    )
+
+        class _NotWrapped(self._Handler):
+            _fail_on_stopiteration = False
+
+        with self.assertRaises(StopIteration):
+            self._new(_NotWrapped)._funcs[0](1)
+
+    def test_profiler_selected_by_conf(self):
+        def f(x):
+            return x
+
+        def data_source_f(x):
+            return x
+
+        data_source_f.__module__ = "pyspark.sql.worker.plan_data_source_read"
+
+        udf_perf = self._Handler([], 
RunnerConf({"spark.sql.pyspark.udf.profiler": "perf"}), None)
+        data_source_perf = self._Handler(
+            [], RunnerConf({"spark.sql.pyspark.dataSource.profiler": "perf"}), 
None
+        )
+        udf_memory = self._Handler(
+            [], RunnerConf({"spark.sql.pyspark.udf.profiler": "memory"}), None
+        )
+        no_profiler = self._Handler([], RunnerConf({}), None)
+        with patch.object(
+            EvalTypeHandler, "_with_perf_profiler", side_effect=lambda func, 
result_id: "perf"
+        ) as perf:
+            self.assertEqual(udf_perf._with_profiler(f, 0), "perf")
+            self.assertIs(data_source_perf._with_profiler(f, 0), f)
+            self.assertIs(udf_perf._with_profiler(data_source_f, 0), 
data_source_f)
+            self.assertEqual(data_source_perf._with_profiler(data_source_f, 
0), "perf")
+            self.assertEqual(perf.call_count, 2)
+        with patch.object(
+            EvalTypeHandler, "_with_memory_profiler", side_effect=lambda func, 
result_id: "memory"
+        ):
+            self.assertEqual(udf_memory._with_profiler(f, 0), "memory")
+        self.assertIs(no_profiler._with_profiler(f, 0), f)
+
+
+class UnpickleAndChainTests(PySparkBaseTestCase):
+    def test_chains_in_order(self):
+        ser = CloudPickleSerializer()
+        func, return_type = unpickle_and_chain(
+            pickleSer,
+            [
+                ser.dumps((lambda x: x + 1, LongType())),
+                ser.dumps((lambda x: str(x * 2), StringType())),
+            ],
+        )
+        self.assertEqual(func(3), "8")
+        self.assertEqual(return_type, StringType())
+
 
 if __name__ == "__main__":
     from pyspark.testing import main
diff --git 
a/python/pyspark/eval_handlers/tests/test_elementwise_eval_type_handlers.py 
b/python/pyspark/eval_handlers/tests/test_elementwise_eval_type_handlers.py
index a03a4726a878..9dbf346607bf 100644
--- a/python/pyspark/eval_handlers/tests/test_elementwise_eval_type_handlers.py
+++ b/python/pyspark/eval_handlers/tests/test_elementwise_eval_type_handlers.py
@@ -102,11 +102,11 @@ class 
ElementwiseEvalTypeHandlerTestCase(PySparkBaseTestCase):
         return handler_cls(
             udfs=[
                 UDF(
-                    func=func,
                     raw_func=func,
                     args_offsets=args_offsets,
                     kwargs_offsets=kwargs_offsets,
                     return_type=return_type,
+                    result_id=0,
                 )
                 for func, args_offsets, kwargs_offsets, return_type in udfs
             ],
diff --git 
a/python/pyspark/eval_handlers/tests/test_pandas_eval_type_handlers.py 
b/python/pyspark/eval_handlers/tests/test_pandas_eval_type_handlers.py
index 9b22dcae9d82..2f419d87b0f1 100644
--- a/python/pyspark/eval_handlers/tests/test_pandas_eval_type_handlers.py
+++ b/python/pyspark/eval_handlers/tests/test_pandas_eval_type_handlers.py
@@ -92,16 +92,16 @@ class PandasEvalTypeHandlerTestCase(PySparkBaseTestCase):
 
     @classmethod
     def _grouped_handler(cls, handler_cls, udf, arg_offsets, runner_conf=None):
-        """Build a grouped/cogrouped-map handler. As from ``read_single_udf``, 
``func`` is
-        wrapped (losing the signature) while ``raw_func`` keeps it."""
+        """Build a grouped/cogrouped-map handler. The function the handler 
wraps loses its
+        signature, so the handler reads it from the record's ``raw_func``."""
         return handler_cls(
             udfs=[
                 UDF(
-                    func=lambda *a: udf(*a),
                     raw_func=udf,
                     args_offsets=arg_offsets,
                     kwargs_offsets={},
                     return_type=cls._GROUPED_RETURN_TYPE,
+                    result_id=0,
                 )
             ],
             runner_conf=runner_conf or RunnerConf({}),
@@ -115,13 +115,13 @@ class PandasEvalTypeHandlerTestCase(PySparkBaseTestCase):
 
     @staticmethod
     def _udf(func, return_type=None, args=(0,), kwargs=None):
-        """A ``UDF`` record for a pandas UDF."""
+        """A ``UDF`` record for a pandas UDF, as the worker builds it."""
         return UDF(
-            func=func,
             raw_func=func,
             args_offsets=list(args),
             kwargs_offsets=dict(kwargs or {}),
             return_type=return_type or LongType(),
+            result_id=0,
         )
 
     @staticmethod
diff --git 
a/python/pyspark/eval_handlers/tests/test_stateful_eval_type_handlers.py 
b/python/pyspark/eval_handlers/tests/test_stateful_eval_type_handlers.py
index 6f8646926f01..678561e64d77 100644
--- a/python/pyspark/eval_handlers/tests/test_stateful_eval_type_handlers.py
+++ b/python/pyspark/eval_handlers/tests/test_stateful_eval_type_handlers.py
@@ -119,11 +119,11 @@ class 
TransformWithStateHandlerTestCase(PySparkBaseTestCase):
         return handler_cls(
             udfs=[
                 UDF(
-                    func=udf,
                     raw_func=udf,
                     args_offsets=arg_offsets,
                     kwargs_offsets={},
                     return_type=cls._RETURN_TYPE,
+                    result_id=0,
                 )
             ],
             runner_conf=runner_conf or RunnerConf({}),
@@ -237,11 +237,11 @@ class 
PandasGroupedMapWithStateUDFHandlerTests(PySparkBaseTestCase):
         return PandasGroupedMapWithStateUDFHandler(
             udfs=[
                 UDF(
-                    func=udf,
                     raw_func=udf,
                     args_offsets=cls._ARG_OFFSETS,
                     kwargs_offsets={},
                     return_type=cls._RETURN_TYPE,
+                    result_id=0,
                 )
             ],
             runner_conf=runner_conf or RunnerConf({}),
diff --git a/python/pyspark/eval_handlers/tests/test_udtf_eval_type_handlers.py 
b/python/pyspark/eval_handlers/tests/test_udtf_eval_type_handlers.py
index 038a5a2cba04..7d9760921418 100644
--- a/python/pyspark/eval_handlers/tests/test_udtf_eval_type_handlers.py
+++ b/python/pyspark/eval_handlers/tests/test_udtf_eval_type_handlers.py
@@ -17,9 +17,8 @@
 
 """Tests for the UDTF eval type handlers (``_udtf``).
 
-The handler gets the ``UDTF`` record ``read_single_udtf`` returns; each test 
builds that
-record around a UDTF instance, calls ``run(0, <input batches>)`` and asserts 
on the output
-batches.
+Each test builds the handler from the ``UDTF`` record of a UDTF class, calls
+``run(0, <input batches>)`` and asserts on the output batches.
 """
 
 import os
@@ -34,6 +33,7 @@ from pyspark.serializers import BatchedSerializer, 
CPickleSerializer
 from pyspark.sql.functions import SkipRestOfInputTableException
 from pyspark.sql.pandas.serializers import ArrowStreamSerializer
 from pyspark.sql.types import LongType, StructField, StructType
+from pyspark.sql.udtf import AnalyzeResult
 from pyspark.testing.utils import (
     PySparkBaseTestCase,
     have_pandas,
@@ -62,9 +62,11 @@ if have_pyarrow:
 class _RecordingUDTF:
     """Arrow UDTF that echoes its TABLE argument and records the lifecycle 
calls."""
 
-    def __init__(self, skip_after=None):
+    # Raise SkipRestOfInputTableException on the eval call after this many; 
None never skips.
+    _skip_after = None
+
+    def __init__(self):
         self.calls = []
-        self._skip_after = skip_after
 
     def eval(self, table):
         self.calls.append("eval")
@@ -83,9 +85,11 @@ class _RecordingUDTF:
 class _RowUDTF:
     """Row-wise UDTF that emits its argument and its double, and records the 
lifecycle calls."""
 
-    def __init__(self, skip_after=None):
+    # Raise SkipRestOfInputTableException on the eval call after this many; 
None never skips.
+    _skip_after = None
+
+    def __init__(self):
         self.calls = []
-        self._skip_after = skip_after
 
     def eval(self, a):
         self.calls.append("eval")
@@ -102,10 +106,30 @@ class _RowUDTF:
 
 
 class UDTFEvalTypeHandlerTestCase(PySparkBaseTestCase):
-    """Shared return type for the UDTF handler tests."""
+    """Shared return type and helpers for the UDTF handler tests."""
 
     _RETURN_TYPE = StructType([StructField("x", LongType())])
 
+    @staticmethod
+    def _udtf_record(
+        udtf_cls,
+        return_type,
+        args=(0,),
+        kwargs=None,
+        partition_child_indexes=(),
+        analyze_result=None,
+    ):
+        """The ``UDTF`` record the worker builds for ``udtf_cls``."""
+        return UDTF(
+            udtf_cls=udtf_cls,
+            args_offsets=list(args),
+            kwargs_offsets=kwargs or {},
+            return_type=return_type,
+            partition_child_indexes=list(partition_child_indexes),
+            analyze_result=analyze_result,
+            name="test_udtf",
+        )
+
 
 @unittest.skipIf(not have_pyarrow, pyarrow_requirement_message)
 class UDTFEvalTypeHandlerRegistrationTests(PySparkBaseTestCase):
@@ -136,19 +160,9 @@ class 
UDTFEvalTypeHandlerRegistrationTests(PySparkBaseTestCase):
 class TableUDTFHandlerTests(UDTFEvalTypeHandlerTestCase):
     _ROW_RETURN_TYPE = StructType([StructField("x", LongType()), 
StructField("y", LongType())])
 
-    def _handler(self, udtf, args=(0,), kwargs=None, return_type=None):
-        return TableUDTFHandler(
-            udfs=[
-                UDTF(
-                    udtf=udtf,
-                    args_offsets=list(args),
-                    kwargs_offsets=kwargs or {},
-                    return_type=return_type or self._ROW_RETURN_TYPE,
-                )
-            ],
-            runner_conf=RunnerConf({}),
-            eval_conf=EvalConf({}),
-        )
+    def _handler(self, udtf_cls, args=(0,), kwargs=None, return_type=None):
+        udtf = self._udtf_record(udtf_cls, return_type or 
self._ROW_RETURN_TYPE, args, kwargs)
+        return TableUDTFHandler(udfs=[udtf], runner_conf=RunnerConf({}), 
eval_conf=EvalConf({}))
 
     def _run(self, handler, *rows):
         # Each input row is a list of argument values; each output is the 
tuple of rows one
@@ -156,45 +170,48 @@ class TableUDTFHandlerTests(UDTFEvalTypeHandlerTestCase):
         return list(handler.run(0, iter([list(r) for r in rows])))
 
     def test_serializer_and_partition_wrapper(self):
-        ser = self._handler(_RowUDTF()).serializer
+        ser = self._handler(_RowUDTF).serializer
         self.assertIsInstance(ser, BatchedSerializer)
         self.assertIsInstance(ser.serializer, CPickleSerializer)
         self.assertEqual(ser.batchSize, 1)
         self.assertIs(TableUDTFHandler.partition_wrapper, UDTFWithPartitions)
 
     def test_eval_terminate_cleanup(self):
-        udtf = _RowUDTF()
-        out = self._run(self._handler(udtf), [1], [2])
+        handler = self._handler(_RowUDTF)
+        out = self._run(handler, [1], [2])
         self.assertEqual(out, [((1, 2),), ((2, 4),), ((-1, -2),)])
-        self.assertEqual(udtf.calls, ["eval", "eval", "terminate", "cleanup"])
+        self.assertEqual(handler._udtf.calls, ["eval", "eval", "terminate", 
"cleanup"])
 
     def test_no_terminate_or_cleanup(self):
         class EvalOnly:
             def eval(self, a):
                 yield a, a
 
-        self.assertEqual(self._run(self._handler(EvalOnly()), [1]), [((1, 
1),)])
+        self.assertEqual(self._run(self._handler(EvalOnly), [1]), [((1, 1),)])
 
     def test_kwargs(self):
         class KwargsUDTF:
             def eval(self, *, a):
                 yield a, -a
 
-        out = self._run(self._handler(KwargsUDTF(), args=(), kwargs={"a": 0}), 
[7])
+        out = self._run(self._handler(KwargsUDTF, args=(), kwargs={"a": 0}), 
[7])
         self.assertEqual(out, [((7, -7),)])
 
     def test_skip_rest_of_input_table(self):
-        udtf = _RowUDTF(skip_after=1)
-        out = self._run(self._handler(udtf), [1], [2], [3])
+        class SkipUDTF(_RowUDTF):
+            _skip_after = 1
+
+        handler = self._handler(SkipUDTF)
+        out = self._run(handler, [1], [2], [3])
         self.assertEqual(out, [((1, 2),), ((-1, -2),)])
-        self.assertEqual(udtf.calls, ["eval", "eval", "terminate", "cleanup"])
+        self.assertEqual(handler._udtf.calls, ["eval", "eval", "terminate", 
"cleanup"])
 
     def test_none_result(self):
         class NoneUDTF:
             def eval(self, a):
                 return None
 
-        self.assertEqual(self._run(self._handler(NoneUDTF()), [1]), [()])
+        self.assertEqual(self._run(self._handler(NoneUDTF), [1]), [()])
 
     def test_non_nullable_column(self):
         class NullUDTF:
@@ -205,7 +222,7 @@ class TableUDTFHandlerTests(UDTFEvalTypeHandlerTestCase):
             [StructField("x", LongType(), nullable=False), StructField("y", 
LongType())]
         )
         with self.assertRaises(PySparkRuntimeError) as ctx:
-            self._run(self._handler(NullUDTF(), return_type=return_type), [1])
+            self._run(self._handler(NullUDTF, return_type=return_type), [1])
         self.assertEqual(ctx.exception.getCondition(), "UDTF_EXEC_ERROR")
 
     def test_errors(self):
@@ -225,44 +242,39 @@ class TableUDTFHandlerTests(UDTFEvalTypeHandlerTestCase):
             def eval(self, a):
                 yield a
 
-        for udtf, error_class in [
-            (Raises(), "UDTF_EXEC_ERROR"),
-            (NotIterable(), "UDTF_RETURN_NOT_ITERABLE"),
-            (WrongColumns(), "UDTF_RETURN_SCHEMA_MISMATCH"),
-            (NotARow(), "UDTF_INVALID_OUTPUT_ROW_TYPE"),
+        for udtf_cls, error_class in [
+            (Raises, "UDTF_EXEC_ERROR"),
+            (NotIterable, "UDTF_RETURN_NOT_ITERABLE"),
+            (WrongColumns, "UDTF_RETURN_SCHEMA_MISMATCH"),
+            (NotARow, "UDTF_INVALID_OUTPUT_ROW_TYPE"),
         ]:
             with self.subTest(error_class=error_class):
                 with self.assertRaises(PySparkRuntimeError) as ctx:
-                    self._run(self._handler(udtf), [1])
+                    self._run(self._handler(udtf_cls), [1])
                 self.assertEqual(ctx.exception.getCondition(), error_class)
 
     def test_cleanup_on_error(self):
-        udtf = _RowUDTF()
-        udtf.eval = lambda a: 1  # not iterable
+        class NotIterable(_RowUDTF):
+            def eval(self, a):
+                return 1
+
+        handler = self._handler(NotIterable)
         with self.assertRaises(PySparkRuntimeError):
-            self._run(self._handler(udtf), [1])
-        self.assertEqual(udtf.calls, ["cleanup"])
+            self._run(handler, [1])
+        self.assertEqual(handler._udtf.calls, ["cleanup"])
 
 
 @unittest.skipIf(not have_pyarrow, pyarrow_requirement_message)
 class ArrowUDTFHandlerTests(UDTFEvalTypeHandlerTestCase):
-    def _handler(self, udtf, args=(0,), kwargs=None, table_arg_offsets="0"):
+    def _handler(
+        self, udtf_cls, args=(0,), kwargs=None, table_arg_offsets="0", 
partition_child_indexes=()
+    ):
         conf = {} if table_arg_offsets is None else {"table_arg_offsets": 
table_arg_offsets}
-        return ArrowUDTFHandler(
-            udfs=[
-                UDTF(
-                    udtf=udtf,
-                    args_offsets=list(args),
-                    kwargs_offsets=kwargs or {},
-                    return_type=self._RETURN_TYPE,
-                )
-            ],
-            runner_conf=RunnerConf({}),
-            eval_conf=EvalConf(conf),
-        )
+        udtf = self._udtf_record(udtf_cls, self._RETURN_TYPE, args, kwargs, 
partition_child_indexes)
+        return ArrowUDTFHandler(udfs=[udtf], runner_conf=RunnerConf({}), 
eval_conf=EvalConf(conf))
 
     def test_serializer(self):
-        self.assertIsInstance(self._handler(_RecordingUDTF()).serializer, 
ArrowStreamSerializer)
+        self.assertIsInstance(self._handler(_RecordingUDTF).serializer, 
ArrowStreamSerializer)
 
     def test_partition_wrapper(self):
         self.assertIs(UDTFEvalTypeHandler.partition_wrapper, 
UDTFWithPartitions)
@@ -272,16 +284,20 @@ class ArrowUDTFHandlerTests(UDTFEvalTypeHandlerTestCase):
         # Column 1 is the projected PARTITION BY key; the UDTF only sees 
column 0.
         created = []
 
-        def create_udtf():
-            created.append(_RecordingUDTF(skip_after=1))
-            return created[-1]
+        class Udtf(_RecordingUDTF):
+            _skip_after = 1
+
+            def __init__(self):
+                super().__init__()
+                created.append(self)
 
-        udtf = ArrowUDTFWithPartition(create_udtf, [1])
         batches = [
             ArrowBatchTransformer.wrap_struct(pa.record_batch({"x": [1, 2, 3], 
"k": [0, 0, 1]})),
             ArrowBatchTransformer.wrap_struct(pa.record_batch({"x": [4, 5], 
"k": [1, 2]})),
         ]
-        out = list(self._handler(udtf).run(0, iter(batches)))
+        handler = self._handler(Udtf, partition_child_indexes=[1])
+        self.assertIsInstance(handler._udtf, ArrowUDTFWithPartition)
+        out = list(handler.run(0, iter(batches)))
         # Partition k=1 spans both batches; its second eval call is skipped.
         self.assertEqual(
             [r["x"] for b in out for r in b.column(0).to_pylist()], [1, 2, -1, 
3, -1, 5, -1]
@@ -296,9 +312,9 @@ class ArrowUDTFHandlerTests(UDTFEvalTypeHandlerTestCase):
         )
 
     def test_eval_terminate_cleanup(self):
-        udtf = _RecordingUDTF()
+        handler = self._handler(_RecordingUDTF)
         out = list(
-            self._handler(udtf).run(
+            handler.run(
                 0,
                 iter(
                     [
@@ -309,7 +325,7 @@ class ArrowUDTFHandlerTests(UDTFEvalTypeHandlerTestCase):
             )
         )
         self.assertEqual([r["x"] for b in out for r in 
b.column(0).to_pylist()], [1, 2, 3, -1])
-        self.assertEqual(udtf.calls, ["eval", "eval", "terminate", "cleanup"])
+        self.assertEqual(handler._udtf.calls, ["eval", "eval", "terminate", 
"cleanup"])
 
     def test_kwargs(self):
         class KwargsUDTF:
@@ -317,7 +333,7 @@ class ArrowUDTFHandlerTests(UDTFEvalTypeHandlerTestCase):
                 yield pa.table({"x": table.column(0)})
 
         out = list(
-            self._handler(KwargsUDTF(), args=(), kwargs={"table": 0}).run(
+            self._handler(KwargsUDTF, args=(), kwargs={"table": 0}).run(
                 0, 
iter([ArrowBatchTransformer.wrap_struct(pa.record_batch({"x": [7]}))])
             )
         )
@@ -330,11 +346,14 @@ class ArrowUDTFHandlerTests(UDTFEvalTypeHandlerTestCase):
                 yield pa.table({"x": pa.compute.multiply(arr, 10)})
 
         batch = pa.RecordBatch.from_arrays([pa.array([1, 2], 
type=pa.int64())], ["a"])
-        out = list(self._handler(ScalarUDTF(), table_arg_offsets=None).run(0, 
iter([batch])))
+        out = list(self._handler(ScalarUDTF, table_arg_offsets=None).run(0, 
iter([batch])))
         self.assertEqual([r["x"] for b in out for r in 
b.column(0).to_pylist()], [10, 20])
 
     def test_skip_rest_of_input_table(self):
-        udtf = _RecordingUDTF(skip_after=1)
+        class SkipUDTF(_RecordingUDTF):
+            _skip_after = 1
+
+        handler = self._handler(SkipUDTF)
         batches = iter(
             [
                 ArrowBatchTransformer.wrap_struct(pa.record_batch({"x": [1]})),
@@ -342,9 +361,9 @@ class ArrowUDTFHandlerTests(UDTFEvalTypeHandlerTestCase):
                 ArrowBatchTransformer.wrap_struct(pa.record_batch({"x": [3]})),
             ]
         )
-        out = list(self._handler(udtf).run(0, batches))
+        out = list(handler.run(0, batches))
         self.assertEqual([r["x"] for b in out for r in 
b.column(0).to_pylist()], [1, -1])
-        self.assertEqual(udtf.calls, ["eval", "eval", "terminate", "cleanup"])
+        self.assertEqual(handler._udtf.calls, ["eval", "eval", "terminate", 
"cleanup"])
         # The remaining input is left unconsumed.
         self.assertEqual(len(list(batches)), 1)
 
@@ -354,7 +373,7 @@ class ArrowUDTFHandlerTests(UDTFEvalTypeHandlerTestCase):
                 return None
 
         out = list(
-            self._handler(NoneUDTF()).run(
+            self._handler(NoneUDTF).run(
                 0, 
iter([ArrowBatchTransformer.wrap_struct(pa.record_batch({"x": [1]}))])
             )
         )
@@ -378,16 +397,16 @@ class ArrowUDTFHandlerTests(UDTFEvalTypeHandlerTestCase):
             def eval(self, table):
                 yield pa.table({"x": [1], "y": [2]})
 
-        for udtf, error_class in [
-            (Raises(), "UDTF_EXEC_ERROR"),
-            (NotIterable(), "UDTF_RETURN_NOT_ITERABLE"),
-            (NotArrow(), "UDTF_ARROW_TYPE_CONVERSION_ERROR"),
-            (WrongColumns(), "UDTF_RETURN_SCHEMA_MISMATCH"),
+        for udtf_cls, error_class in [
+            (Raises, "UDTF_EXEC_ERROR"),
+            (NotIterable, "UDTF_RETURN_NOT_ITERABLE"),
+            (NotArrow, "UDTF_ARROW_TYPE_CONVERSION_ERROR"),
+            (WrongColumns, "UDTF_RETURN_SCHEMA_MISMATCH"),
         ]:
             with self.subTest(error_class=error_class):
                 with self.assertRaises(PySparkRuntimeError) as ctx:
                     list(
-                        self._handler(udtf).run(
+                        self._handler(udtf_cls).run(
                             0,
                             
iter([ArrowBatchTransformer.wrap_struct(pa.record_batch({"x": [1]}))]),
                         )
@@ -395,15 +414,18 @@ class ArrowUDTFHandlerTests(UDTFEvalTypeHandlerTestCase):
                 self.assertEqual(ctx.exception.getCondition(), error_class)
 
     def test_cleanup_on_error(self):
-        udtf = _RecordingUDTF()
-        udtf.eval = lambda table: 1  # not iterable
+        class NotIterable(_RecordingUDTF):
+            def eval(self, table):
+                return 1
+
+        handler = self._handler(NotIterable)
         with self.assertRaises(PySparkRuntimeError):
             list(
-                self._handler(udtf).run(
+                handler.run(
                     0, 
iter([ArrowBatchTransformer.wrap_struct(pa.record_batch({"x": [1]}))])
                 )
             )
-        self.assertEqual(udtf.calls, ["cleanup"])
+        self.assertEqual(handler._udtf.calls, ["cleanup"])
 
 
 class ArrowTableUDTFHandlerTestsMixin:
@@ -414,16 +436,10 @@ class ArrowTableUDTFHandlerTestsMixin:
     _ROW_RETURN_TYPE = StructType([StructField("x", LongType()), 
StructField("y", LongType())])
     _ROW_INPUT_TYPE = StructType([StructField("a", LongType())])
 
-    def _handler(self, udtf, args=(0,), kwargs=None, return_type=None):
+    def _handler(self, udtf_cls, args=(0,), kwargs=None, return_type=None):
+        udtf = self._udtf_record(udtf_cls, return_type or 
self._ROW_RETURN_TYPE, args, kwargs)
         return self.handler_cls(
-            udfs=[
-                UDTF(
-                    udtf=udtf,
-                    args_offsets=list(args),
-                    kwargs_offsets=kwargs or {},
-                    return_type=return_type or self._ROW_RETURN_TYPE,
-                )
-            ],
+            udfs=[udtf],
             runner_conf=RunnerConf({}),
             eval_conf=EvalConf({"input_type": self._ROW_INPUT_TYPE.json()}),
         )
@@ -433,35 +449,38 @@ class ArrowTableUDTFHandlerTestsMixin:
         return [r for b in handler.run(0, iter(batches)) for r in 
b.column(0).to_pylist()]
 
     def test_serializer_and_partition_wrapper(self):
-        self.assertIsInstance(self._handler(_RowUDTF()).serializer, 
ArrowStreamSerializer)
+        self.assertIsInstance(self._handler(_RowUDTF).serializer, 
ArrowStreamSerializer)
         self.assertIs(self.handler_cls.partition_wrapper, UDTFWithPartitions)
 
     def test_eval_terminate_cleanup(self):
-        udtf = _RowUDTF()
-        out = self._run(self._handler(udtf), [1, 2], [3])
+        handler = self._handler(_RowUDTF)
+        out = self._run(handler, [1, 2], [3])
         self.assertEqual([(r["x"], r["y"]) for r in out], [(1, 2), (2, 4), (3, 
6), (-1, -2)])
-        self.assertEqual(udtf.calls, ["eval", "eval", "eval", "terminate", 
"cleanup"])
+        self.assertEqual(handler._udtf.calls, ["eval", "eval", "eval", 
"terminate", "cleanup"])
 
     def test_kwargs(self):
         class KwargsUDTF:
             def eval(self, *, a):
                 yield a, -a
 
-        out = self._run(self._handler(KwargsUDTF(), args=(), kwargs={"a": 0}), 
[7])
+        out = self._run(self._handler(KwargsUDTF, args=(), kwargs={"a": 0}), 
[7])
         self.assertEqual([(r["x"], r["y"]) for r in out], [(7, -7)])
 
     def test_skip_rest_of_input_table(self):
-        udtf = _RowUDTF(skip_after=1)
-        out = self._run(self._handler(udtf), [1, 2, 3])
+        class SkipUDTF(_RowUDTF):
+            _skip_after = 1
+
+        handler = self._handler(SkipUDTF)
+        out = self._run(handler, [1, 2, 3])
         self.assertEqual([(r["x"], r["y"]) for r in out], [(1, 2), (-1, -2)])
-        self.assertEqual(udtf.calls, ["eval", "eval", "terminate", "cleanup"])
+        self.assertEqual(handler._udtf.calls, ["eval", "eval", "terminate", 
"cleanup"])
 
     def test_none_result(self):
         class NoneUDTF:
             def eval(self, a):
                 return None
 
-        self.assertEqual(self._run(self._handler(NoneUDTF()), [1]), [])
+        self.assertEqual(self._run(self._handler(NoneUDTF), [1]), [])
 
     def test_non_nullable_column(self):
         class NullUDTF:
@@ -472,7 +491,7 @@ class ArrowTableUDTFHandlerTestsMixin:
             [StructField("x", LongType(), nullable=False), StructField("y", 
LongType())]
         )
         with self.assertRaises(PySparkRuntimeError) as ctx:
-            self._run(self._handler(NullUDTF(), return_type=return_type), [1])
+            self._run(self._handler(NullUDTF, return_type=return_type), [1])
         self.assertEqual(ctx.exception.getCondition(), "UDTF_EXEC_ERROR")
 
     def test_errors(self):
@@ -488,22 +507,25 @@ class ArrowTableUDTFHandlerTestsMixin:
             def eval(self, a):
                 yield a, a, a
 
-        for udtf, error_class in [
-            (Raises(), "UDTF_EXEC_ERROR"),
-            (NotIterable(), "UDTF_RETURN_NOT_ITERABLE"),
-            (WrongColumns(), "UDTF_RETURN_SCHEMA_MISMATCH"),
+        for udtf_cls, error_class in [
+            (Raises, "UDTF_EXEC_ERROR"),
+            (NotIterable, "UDTF_RETURN_NOT_ITERABLE"),
+            (WrongColumns, "UDTF_RETURN_SCHEMA_MISMATCH"),
         ]:
             with self.subTest(error_class=error_class):
                 with self.assertRaises(PySparkRuntimeError) as ctx:
-                    self._run(self._handler(udtf), [1])
+                    self._run(self._handler(udtf_cls), [1])
                 self.assertEqual(ctx.exception.getCondition(), error_class)
 
     def test_cleanup_on_error(self):
-        udtf = _RowUDTF()
-        udtf.eval = lambda a: 1  # not iterable
+        class NotIterable(_RowUDTF):
+            def eval(self, a):
+                return 1
+
+        handler = self._handler(NotIterable)
         with self.assertRaises(PySparkRuntimeError):
-            self._run(self._handler(udtf), [1])
-        self.assertEqual(udtf.calls, ["cleanup"])
+            self._run(handler, [1])
+        self.assertEqual(handler._udtf.calls, ["cleanup"])
 
 
 @unittest.skipIf(not have_pyarrow, pyarrow_requirement_message)
@@ -515,7 +537,7 @@ class 
ArrowTableUDTFArrowHandlerTests(ArrowTableUDTFHandlerTestsMixin, UDTFEvalT
             def eval(self, a):
                 yield a
 
-        out = self._run(self._handler(ScalarUDTF(), 
return_type=self._RETURN_TYPE), [1, 2])
+        out = self._run(self._handler(ScalarUDTF, 
return_type=self._RETURN_TYPE), [1, 2])
         self.assertEqual([r["x"] for r in out], [1, 2])
 
 
@@ -528,6 +550,73 @@ class ArrowTableUDTFLegacyPandasHandlerTests(
     handler_cls = ArrowTableUDTFLegacyPandasHandler if have_pyarrow else None
 
 
+class UDTFHandlerInitTests(UDTFEvalTypeHandlerTestCase):
+    def _new(self, handler_cls, udtf_cls, **kwargs):
+        """A new ``handler_cls`` for ``udtf_cls``."""
+        udtf = self._udtf_record(udtf_cls, self._RETURN_TYPE, **kwargs)
+        return handler_cls([udtf], RunnerConf({}), EvalConf({}))
+
+    def test_creates_udtf_instance(self):
+        class Udtf:
+            def eval(self, x):
+                yield (x,)
+
+        handler = self._new(TableUDTFHandler, Udtf)
+        self.assertIsInstance(handler._udtf, Udtf)
+        udtf = handler._udfs[0]
+        self.assertIsInstance(udtf, UDTF)
+        self.assertEqual(udtf.args_offsets, [0])
+        self.assertEqual(udtf.kwargs_offsets, {})
+        self.assertEqual(udtf.return_type, self._RETURN_TYPE)
+
+    def test_analyze_result_passed_to_init(self):
+        class Udtf:
+            def __init__(self, analyze_result):
+                self.analyze_result = analyze_result
+
+            def eval(self, x):
+                yield (x,)
+
+        analyze_result = AnalyzeResult(schema=self._RETURN_TYPE)
+        handler = self._new(TableUDTFHandler, Udtf, 
analyze_result=analyze_result)
+        self.assertEqual(handler._udtf.analyze_result, analyze_result)
+
+    def test_wrapped_in_partition_wrapper(self):
+        class Udtf:
+            def eval(self, row):
+                yield (row,)
+
+        handler = self._new(TableUDTFHandler, Udtf, 
partition_child_indexes=[0])
+        self.assertIsInstance(handler._udtf, UDTFWithPartitions)
+
+    @unittest.skipIf(not have_pyarrow, pyarrow_requirement_message)
+    def test_arrow_partition_wrapper(self):
+        class Udtf:
+            def eval(self, table):
+                yield table
+
+        handler = self._new(ArrowUDTFHandler, Udtf, 
partition_child_indexes=[0])
+        self.assertIsInstance(handler._udtf, ArrowUDTFWithPartition)
+
+    def test_invalid_udtf(self):
+        class NoEval:
+            pass
+
+        class Udtf:
+            def eval(self, x):
+                yield (x,)
+
+        with self.assertRaisesRegex(PySparkRuntimeError, "Invalid UDTF handler 
type"):
+            self._new(TableUDTFHandler, Udtf())
+        with self.assertRaisesRegex(PySparkRuntimeError, "'eval' method"):
+            self._new(TableUDTFHandler, NoEval)
+        with self.assertRaises(PySparkRuntimeError) as ctx:
+            self._new(TableUDTFHandler, Udtf, args=(0, 1))
+        self.assertEqual(
+            ctx.exception.getCondition(), 
"UDTF_EVAL_METHOD_ARGUMENTS_DO_NOT_MATCH_SIGNATURE"
+        )
+
+
 if __name__ == "__main__":
     from pyspark.testing import main
 
diff --git 
a/python/pyspark/eval_handlers/tests/test_vanilla_eval_type_handlers.py 
b/python/pyspark/eval_handlers/tests/test_vanilla_eval_type_handlers.py
index 72a4daa6fa37..fe6aafb75143 100644
--- a/python/pyspark/eval_handlers/tests/test_vanilla_eval_type_handlers.py
+++ b/python/pyspark/eval_handlers/tests/test_vanilla_eval_type_handlers.py
@@ -17,7 +17,7 @@
 
 """Tests for the plain Python (pickle) UDF eval type handler (``_vanilla``).
 
-The handler is built from ``UDF`` records, as ``read_single_udf`` returns 
them, and fed
+The handler is built from ``UDF`` records and fed
 pickled-row input via ``run(0, <rows>)``.
 """
 
@@ -45,11 +45,11 @@ class BatchedUDFHandlerTests(PySparkBaseTestCase):
         return BatchedUDFHandler(
             udfs=[
                 UDF(
-                    func=f,
                     raw_func=f,
                     args_offsets=arg_offsets,
                     kwargs_offsets=kwargs_offsets or {},
                     return_type=return_type or LongType(),
+                    result_id=0,
                 )
                 for arg_offsets, f in udfs
             ],
diff --git a/python/pyspark/worker.py b/python/pyspark/worker.py
index 6105b65f1aac..563b4d9dfe3d 100644
--- a/python/pyspark/worker.py
+++ b/python/pyspark/worker.py
@@ -23,42 +23,33 @@ pyspark.daemon. A task runs either a classic RDD command 
(eval type NON_UDF) or
 through the handler registered for their eval type in pyspark.eval_handlers.
 """
 
-import dataclasses
-import inspect
 import json
 import os
 import sys
 import warnings
-from typing import BinaryIO
+from typing import Any, BinaryIO, Callable
 
 from pyspark import shuffle
-from pyspark.accumulators import (
-    SpecialAccumulatorIds,
-    _accumulatorRegistry,
-    _deserialize_accumulator,
-)
+from pyspark.accumulators import _accumulatorRegistry
 from pyspark.errors import PySparkRuntimeError
-from pyspark.eval_handlers._base import get_eval_type_handler
+from pyspark.eval_handlers._base import EvalTypeHandler, get_eval_type_handler
 from pyspark.eval_handlers._typing import UDF, UDTF
-from pyspark.eval_handlers._udtf import TableUDTFHandler, UDTFEvalTypeHandler
+from pyspark.eval_handlers._udtf import UDTFHandlerBase
 from pyspark.logger.worker_io import capture_outputs
 from pyspark.messages import (
     SparkMessageReceiver,
     SparkSocketMessageReceiver,
 )
 from pyspark.serializers import (
+    FramedSerializer,
     SpecialLengths,
     write_int,
     write_with_length,
 )
-from pyspark.sql.types import (
-    StructType,
-    _parse_datatype_json_string,
-)
+from pyspark.sql.types import DataType, StructType, _parse_datatype_json_string
 from pyspark.taskcontext import BarrierTaskContext, TaskContext
 from pyspark.util import (
     PythonEvalType,
-    fail_on_stopiteration,
     handle_worker_exception,
     start_faulthandler_periodic_traceback,
     with_faulthandler,
@@ -86,217 +77,21 @@ def report_metrics(outfile: BinaryIO, metrics: dict[str, 
int]) -> None:
     write_with_length(payload, outfile)
 
 
-def chain(f, g):
+def chain(f: Callable[..., Any], g: Callable[..., Any]) -> Callable[..., Any]:
     """chain two functions together"""
     return lambda *a: g(f(*a))
 
 
-def wrap_perf_profiler(f, iter_based, result_id):
-    from pyspark.sql.profiler import ProfileResultsParam, 
ProfileResultsParamV2, WorkerPerfProfiler
-
-    accumulator = _deserialize_accumulator(
-        SpecialAccumulatorIds.SQL_UDF_PROFIER, None, ProfileResultsParam
-    )
-    accumulator_v2 = _deserialize_accumulator(
-        SpecialAccumulatorIds.SQL_UDF_PROFIER_V2, {}, ProfileResultsParamV2
-    )
-
-    if iter_based:
-
-        def profiling_func(*args, **kwargs):
-            iterator = iter(f(*args, **kwargs))
-            while True:
-                try:
-                    with WorkerPerfProfiler(accumulator, accumulator_v2, 
result_id):
-                        item = next(iterator)
-                    yield item
-                except StopIteration:
-                    break
-
-    else:
-
-        def profiling_func(*args, **kwargs):
-            with WorkerPerfProfiler(accumulator, accumulator_v2, result_id):
-                ret = f(*args, **kwargs)
-            return ret
-
-    return profiling_func
-
-
-def wrap_memory_profiler(f, iter_based, result_id):
-    import pyspark.memory_profiler_ext
-    from pyspark.sql.profiler import (
-        ProfileResultsParam,
-        ProfileResultsParamV2,
-        WorkerMemoryProfiler,
-    )
-
-    if not pyspark.memory_profiler_ext.has_memory_profiler:
-        return f
-
-    accumulator = _deserialize_accumulator(
-        SpecialAccumulatorIds.SQL_UDF_PROFIER, None, ProfileResultsParam
-    )
-
-    accumulator_v2 = _deserialize_accumulator(
-        SpecialAccumulatorIds.SQL_UDF_PROFIER_V2, {}, ProfileResultsParamV2
-    )
-
-    if iter_based:
-
-        def profiling_func(*args, **kwargs):
-            g = f(*args, **kwargs)
-            iterator = iter(g)
-
-            while True:
-                try:
-                    with WorkerMemoryProfiler(accumulator, accumulator_v2, 
result_id, g.gi_code):
-                        item = next(iterator)
-                    yield item
-                except StopIteration:
-                    break
-
-    else:
-
-        def profiling_func(*args, **kwargs):
-            with WorkerMemoryProfiler(accumulator, accumulator_v2, result_id, 
f):
-                ret = f(*args, **kwargs)
-            return ret
-
-    return profiling_func
-
-
-def read_single_udf(pickleSer, udf_info, handler_cls, runner_conf):
-    eval_type = handler_cls.eval_type
-    chained_func = None
-    for udf in udf_info.udfs:
+def unpickle_and_chain(
+    pickleSer: FramedSerializer, pickled_udfs: list[memoryview]
+) -> tuple[Callable[..., Any], DataType]:
+    """Unpickle the functions applied in turn to one column and chain them 
into one; returns the
+    chained function and the return type of the last one."""
+    chained_func, return_type = read_command(pickleSer, pickled_udfs[0])
+    for udf in pickled_udfs[1:]:
         f, return_type = read_command(pickleSer, udf)
-        if chained_func is None:
-            chained_func = f
-        else:
-            chained_func = chain(chained_func, f)
-
-    # If chained_func is from pyspark.sql.worker, it is to read/write data 
source.
-    # In this case, we check the data_source_profiler config.
-    module = getattr(chained_func, "__module__", "")
-    if isinstance(module, str) and module.startswith("pyspark.sql.worker."):
-        profiler = runner_conf.data_source_profiler
-    else:
-        profiler = runner_conf.udf_profiler
-    if profiler == "perf":
-        profiling_func = wrap_perf_profiler(
-            chained_func, handler_cls.is_iter_based, udf_info.result_id
-        )
-    elif profiler == "memory":
-        profiling_func = wrap_memory_profiler(
-            chained_func, handler_cls.is_iter_based, udf_info.result_id
-        )
-    else:
-        profiling_func = chained_func
-
-    if eval_type in (
-        PythonEvalType.SQL_SCALAR_PANDAS_ITER_UDF,
-        PythonEvalType.SQL_SCALAR_PANDAS_ITER_ELEMENTWISE_UDF,
-        PythonEvalType.SQL_ARROW_BATCHED_UDF,
-    ):
-        func = profiling_func
-    else:
-        # make sure StopIteration's raised in the user code are not ignored
-        # when they are processed in a for loop, raise them as RuntimeError's 
instead
-        func = fail_on_stopiteration(profiling_func)
-
-    # The last returnType will be the return type of UDF.
-    return UDF(
-        func=func,
-        raw_func=chained_func,
-        args_offsets=udf_info.args,
-        kwargs_offsets=udf_info.kwargs,
-        return_type=return_type,
-    )
-
-
-# Read a serialized user-defined table function (UDTF) and prepare it for 
evaluation: wrap
-# the constructor to receive the AnalyzeResult, instantiate the UDTF (wrapped 
in the handler's
-# partition_wrapper when its TABLE argument has PARTITION BY expressions), and 
check the 'eval'
-# signature against the call arguments.
-def read_single_udtf(pickleSer, udtf_info, handler_cls):
-    if udtf_info.pickled_analyze_result is not None:
-        pickled_analyze_result = 
pickleSer.loads(udtf_info.pickled_analyze_result)
-    else:
-        pickled_analyze_result = None
-    handler = read_command(pickleSer, udtf_info.handler)
-    if not isinstance(handler, type):
-        raise PySparkRuntimeError(
-            f"Invalid UDTF handler type. Expected a class (type 'type'), but "
-            f"got an instance of {type(handler).__name__}."
-        )
-
-    return_type = _parse_datatype_json_string(udtf_info.return_type)
-    if not isinstance(return_type, StructType):
-        raise PySparkRuntimeError(
-            f"The return type of a UDTF must be a struct type, but got 
{type(return_type)}."
-        )
-
-    # Pick how to construct the UDTF from the count of named positional 
__init__ parameters
-    # after self. With an AnalyzeResult, one parameter receives a copy of it, 
zero means the
-    # class is called with no arguments, and more are an error; without one, 
any is an error.
-    udtf_init_args = inspect.getfullargspec(handler)
-    if pickled_analyze_result is not None:
-        if len(udtf_init_args.args) > 2:
-            raise PySparkRuntimeError(
-                
errorClass="UDTF_CONSTRUCTOR_INVALID_IMPLEMENTS_ANALYZE_METHOD",
-                messageParameters={"name": udtf_info.name},
-            )
-        elif len(udtf_init_args.args) == 2:
-            prev_handler = handler
-
-            def construct_udtf():
-                # Here we pass the AnalyzeResult to the UDTF's __init__ method.
-                return 
prev_handler(dataclasses.replace(pickled_analyze_result))
-
-            handler = construct_udtf
-    elif len(udtf_init_args.args) > 1:
-        raise PySparkRuntimeError(
-            errorClass="UDTF_CONSTRUCTOR_INVALID_NO_ANALYZE_METHOD",
-            messageParameters={"name": udtf_info.name},
-        )
-
-    # Instantiate the UDTF class.
-    try:
-        if len(udtf_info.partition_child_indexes) > 0:
-            udtf = handler_cls.partition_wrapper(handler, 
udtf_info.partition_child_indexes)
-        else:
-            udtf = handler()
-    except Exception as e:
-        raise PySparkRuntimeError(
-            errorClass="UDTF_EXEC_ERROR",
-            messageParameters={"method_name": "__init__", "error": str(e)},
-        )
-
-    # Validate the UDTF
-    if not hasattr(udtf, "eval"):
-        raise PySparkRuntimeError(
-            "Failed to execute the user defined table function because it has 
not "
-            "implemented the 'eval' method. Please add the 'eval' method and 
try "
-            "the query again."
-        )
-
-    # Check that the arguments provided to the UDTF call match the expected 
parameters defined
-    # in the 'eval' method signature.
-    try:
-        inspect.signature(udtf.eval).bind(*udtf_info.args, **udtf_info.kwargs)
-    except TypeError as e:
-        raise PySparkRuntimeError(
-            errorClass="UDTF_EVAL_METHOD_ARGUMENTS_DO_NOT_MATCH_SIGNATURE",
-            messageParameters={"name": udtf_info.name, "reason": str(e)},
-        ) from None
-
-    return UDTF(
-        udtf=udtf,
-        args_offsets=udtf_info.args,
-        kwargs_offsets=udtf_info.kwargs,
-        return_type=return_type,
-    )
+        chained_func = chain(chained_func, f)
+    return chained_func, return_type
 
 
 def invoke_udf(message_receiver: SparkMessageReceiver, outfile: BinaryIO):
@@ -340,7 +135,8 @@ def invoke_udf(message_receiver: SparkMessageReceiver, 
outfile: BinaryIO):
         runner_conf = RunnerConf(init_info.runner_conf)
         eval_conf = EvalConf(init_info.eval_conf)
         # Only the classic RDD command carries a profiler of its own. UDF 
runners fold profiling
-        # into func at construction time (see read_single_udf); UDTF runners 
are not profiled.
+        # into func at construction time (see EvalTypeHandler.__init__); UDTF 
runners are not
+        # profiled.
         profiler = None
         if eval_type == PythonEvalType.NON_UDF:
             assert isinstance(init_info.udf_info, (bytes, memoryview))
@@ -351,16 +147,45 @@ def invoke_udf(message_receiver: SparkMessageReceiver, 
outfile: BinaryIO):
             handler_cls = get_eval_type_handler(eval_type, runner_conf)
             if handler_cls is None:
                 raise ValueError("Unknown eval type: {}".format(eval_type))
+            handler: EvalTypeHandler
             if isinstance(init_info.udf_info, UDTFInfo):
-                assert issubclass(handler_cls, (UDTFEvalTypeHandler, 
TableUDTFHandler))
-                udfs = [read_single_udtf(pickleSer, init_info.udf_info, 
handler_cls)]
+                assert issubclass(handler_cls, UDTFHandlerBase)
+                udtf_cls = read_command(pickleSer, init_info.udf_info.handler)
+                return_type = 
_parse_datatype_json_string(init_info.udf_info.return_type)
+                if not isinstance(return_type, StructType):
+                    raise PySparkRuntimeError(
+                        "The return type of a UDTF must be a struct type, but 
got "
+                        f"{type(return_type)}."
+                    )
+                if init_info.udf_info.pickled_analyze_result is not None:
+                    analyze_result = 
pickleSer.loads(init_info.udf_info.pickled_analyze_result)
+                else:
+                    analyze_result = None
+                udtf = UDTF(
+                    udtf_cls=udtf_cls,
+                    args_offsets=init_info.udf_info.args,
+                    kwargs_offsets=init_info.udf_info.kwargs,
+                    return_type=return_type,
+                    
partition_child_indexes=init_info.udf_info.partition_child_indexes,
+                    analyze_result=analyze_result,
+                    name=init_info.udf_info.name,
+                )
+                handler = handler_cls(udfs=[udtf], runner_conf=runner_conf, 
eval_conf=eval_conf)
             else:
                 assert isinstance(init_info.udf_info, list)
-                udfs = [
-                    read_single_udf(pickleSer, udf_info, handler_cls, 
runner_conf)
-                    for udf_info in init_info.udf_info
-                ]
-            handler = handler_cls(udfs=udfs, runner_conf=runner_conf, 
eval_conf=eval_conf)
+                udfs = []
+                for udf_info in init_info.udf_info:
+                    chained_func, return_type = unpickle_and_chain(pickleSer, 
udf_info.udfs)
+                    udfs.append(
+                        UDF(
+                            raw_func=chained_func,
+                            args_offsets=udf_info.args,
+                            kwargs_offsets=udf_info.kwargs,
+                            return_type=return_type,
+                            result_id=udf_info.result_id,
+                        )
+                    )
+                handler = handler_cls(udfs=udfs, runner_conf=runner_conf, 
eval_conf=eval_conf)
             func, serializer = handler.run, handler.serializer
             deserializer = serializer
 


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to