This is an automated email from the ASF dual-hosted git repository.
Yicong-Huang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 4cd8da26725e [SPARK-60503][PYTHON] Assign UDFs to eval type handlers
and move UDF wrapping onto them
4cd8da26725e is described below
commit 4cd8da26725e309844e8a720e50e131cb0274ee3
Author: Yicong Huang <[email protected]>
AuthorDate: Sat Oct 24 22:03:48 2026 +0000
[SPARK-60503][PYTHON] Assign UDFs to eval type handlers and move UDF
wrapping onto them
### What changes were proposed in this pull request?
The worker builds a `UDF` or `UDTF` record from what it unpickled, and
constructs the eval type handler from the records. Wrapping each UDF
(profiling, `fail_on_stopiteration`) and instantiating the UDTF (AnalyzeResult,
the partition wrapper) move from `worker.py` onto the handlers.
```python
if isinstance(init_info.udf_info, UDTFInfo):
udtf_info = init_info.udf_info
udtf_cls = read_command(pickleSer, udtf_info.handler)
return_type = _parse_datatype_json_string(udtf_info.return_type)
... # check return_type is a StructType; unpickle the AnalyzeResult
udtf = UDTF(
udtf_cls=udtf_cls,
args_offsets=udtf_info.args,
kwargs_offsets=udtf_info.kwargs,
return_type=return_type,
partition_child_indexes=udtf_info.partition_child_indexes,
analyze_result=analyze_result,
name=udtf_info.name,
)
handler = handler_cls(udfs=[udtf], runner_conf=runner_conf,
eval_conf=eval_conf)
else:
udfs = []
for udf_info in init_info.udf_info:
chained_func, return_type = unpickle_and_chain(pickleSer,
udf_info.udfs)
udfs.append(
UDF(
raw_func=chained_func,
args_offsets=udf_info.args,
kwargs_offsets=udf_info.kwargs,
return_type=return_type,
result_id=udf_info.result_id,
)
)
handler = handler_cls(udfs=udfs, runner_conf=runner_conf,
eval_conf=eval_conf)
```
- `worker.py`: `read_single_udf` and `read_single_udtf` are removed. All
unpickling stays in the worker: `unpickle_and_chain` unpickles the functions
chained on one column, and the UDTF class and AnalyzeResult are unpickled
before building the `UDTF` record.
- `UDF` and `UDTF` keep their field order and hold only what the worker
read: `UDF` drops the wrapped `func`, keeping the unwrapped chained function as
`raw_func`, and gains `result_id`; `UDTF.udtf_cls` (replacing the instance)
plus `partition_child_indexes`, `analyze_result` and `name`.
- `EvalTypeHandler.__init__` wraps each UDF's function for profiling
(`_with_profiler`, which selects `_with_perf_profiler` or
`_with_memory_profiler` by conf) and in `fail_on_stopiteration`, as
`self._funcs` at the indexes of `self._udfs`. The hardcoded
`fail_on_stopiteration` exemption set becomes a `_fail_on_stopiteration =
False` class attribute on the same handlers.
- New `UDTFHandlerBase`, the common base of the UDTF handlers. It owns
`partition_wrapper`, and its `__init__` instantiates the UDTF as `self._udtf`
(the rest of the former `read_single_udtf`).
- The handlers read the wrapped functions from `self._funcs`, the unwrapped
ones (signature, incremental Aggregator) from the record's `raw_func`, and the
UDTF instance from `self._udtf`, directly rather than through attributes that
only renamed them (`self._udf_func`, `self._grouped_udf`, `self._return_type`,
...).
### Why are the changes needed?
Part of SPARK-59415. How a UDF is wrapped depends on its eval type, so the
wrapping belongs on the handler, not in eval-type sets in the worker. Keeping
profiling on the base class also lets the UDTF handlers reuse it.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
New unit tests cover the handler's UDF wrapping (the
`fail_on_stopiteration` setting of every registered handler, profiler
selection), the UDTF instantiation (AnalyzeResult, the pickled and Arrow
partition wrappers, invalid UDTFs) and `unpickle_and_chain`. The handler tests
build their handlers from `UDF` and `UDTF` records.
These existing tests passed locally: `pyspark.eval_handlers.tests`,
`test_udf_profiler`, `test_memory_profiler`, `test_udtf`, `test_arrow_udtf`,
`test_pandas_udf_scalar`, `test_udf`, `test_arrow_python_udf`,
`test_python_datasource`, `test_pandas_grouped_map_with_state`,
`test_pandas_udf_grouped_agg`, `test_pandas_udf_window`,
`test_arrow_grouped_map`, `test_pandas_map`, `test_pandas_cogrouped_map`,
`test_pandas_grouped_map`, `test_pandas_transform_with_state`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #59744 from Yicong-Huang/simplify-udf-preparation.
Authored-by: Yicong Huang <[email protected]>
Signed-off-by: Yicong-Huang <[email protected]>
---
python/benchmarks/bench_eval_type.py | 2 +-
python/pyspark/eval_handlers/_arrow.py | 45 ++-
python/pyspark/eval_handlers/_base.py | 119 +++++++-
python/pyspark/eval_handlers/_elementwise.py | 21 +-
python/pyspark/eval_handlers/_pandas.py | 74 +++--
python/pyspark/eval_handlers/_stateful.py | 55 ++--
python/pyspark/eval_handlers/_typing.py | 31 ++-
python/pyspark/eval_handlers/_udtf.py | 150 +++++++---
python/pyspark/eval_handlers/_vanilla.py | 8 +-
.../tests/test_arrow_eval_type_handlers.py | 40 +--
.../tests/test_base_eval_type_handlers.py | 115 ++++++--
.../tests/test_elementwise_eval_type_handlers.py | 2 +-
.../tests/test_pandas_eval_type_handlers.py | 10 +-
.../tests/test_stateful_eval_type_handlers.py | 4 +-
.../tests/test_udtf_eval_type_handlers.py | 307 +++++++++++++--------
.../tests/test_vanilla_eval_type_handlers.py | 4 +-
python/pyspark/worker.py | 283 ++++---------------
17 files changed, 723 insertions(+), 547 deletions(-)
diff --git a/python/benchmarks/bench_eval_type.py
b/python/benchmarks/bench_eval_type.py
index f659216499d8..592654c43eb9 100644
--- a/python/benchmarks/bench_eval_type.py
+++ b/python/benchmarks/bench_eval_type.py
@@ -198,7 +198,7 @@ class MockProtocolWriter:
arg_offsets: list[int],
buf: io.BytesIO,
) -> None:
- """Write the ``read_single_udf`` portion of the protocol."""
+ """Write the ``UDFInfo`` portion of the protocol."""
write_int(1, buf) # num_udfs
write_int(len(arg_offsets), buf) # num_arg
for offset in arg_offsets:
diff --git a/python/pyspark/eval_handlers/_arrow.py
b/python/pyspark/eval_handlers/_arrow.py
index ed7fd87f6e3d..dd75347a0bf6 100644
--- a/python/pyspark/eval_handlers/_arrow.py
+++ b/python/pyspark/eval_handlers/_arrow.py
@@ -84,7 +84,6 @@ class
ArrowCoGroupedMapUDFHandler(CoGroupedEvalTypeHandler["pa.RecordBatch"]):
super().__init__(udfs, runner_conf, eval_conf)
assert len(udfs) == 1, "One COGROUPED_MAP_ARROW UDF expected here."
udf = udfs[0]
- self._cogrouped_udf = udf.func
# signature was lost when wrapping it
self._num_udf_args = len(inspect.getfullargspec(udf.raw_func).args)
parsed_offsets = extract_key_value_indexes(udf.args_offsets)
@@ -110,11 +109,11 @@ class
ArrowCoGroupedMapUDFHandler(CoGroupedEvalTypeHandler["pa.RecordBatch"]):
right_values = table_from_batches(right_batches,
self._right_val_cols)
if self._num_udf_args == 2:
- result = self._cogrouped_udf(left_values, right_values)
+ result = self._funcs[0](left_values, right_values)
else:
key_table = left_keys if left_keys.num_rows > 0 else right_keys
key = tuple(c[0] for c in key_table.columns)
- result = self._cogrouped_udf(key, left_values, right_values)
+ result = self._funcs[0](key, left_values, right_values)
verify_return_type(result, pa.Table)
# Verify types (and reorder by name when configured).
@@ -142,7 +141,6 @@ class
ArrowGroupedMapIterUDFHandler(GroupedEvalTypeHandler["pa.RecordBatch"]):
super().__init__(udfs, runner_conf, eval_conf)
assert len(udfs) == 1, "One GROUPED_MAP_ARROW_ITER UDF expected here."
udf = udfs[0]
- self._grouped_udf = udf.func
# signature was lost when wrapping it
self._num_udf_args = len(inspect.getfullargspec(udf.raw_func).args)
parsed_offsets = extract_key_value_indexes(udf.args_offsets)
@@ -181,10 +179,10 @@ class
ArrowGroupedMapIterUDFHandler(GroupedEvalTypeHandler["pa.RecordBatch"]):
# Call UDF with iterator of batches
if self._num_udf_args == 1:
- result = self._grouped_udf(value_batches)
+ result = self._funcs[0](value_batches)
else:
key = tuple(c[0] for c in keys.columns)
- result = self._grouped_udf(key, value_batches)
+ result = self._funcs[0](key, value_batches)
# Verify (and reorder by name when configured) each output batch
for batch in verify_return_type(result, Iterator[pa.RecordBatch]):
@@ -213,7 +211,6 @@ class
ArrowGroupedMapUDFHandler(GroupedEvalTypeHandler["pa.RecordBatch"]):
super().__init__(udfs, runner_conf, eval_conf)
assert len(udfs) == 1, "One GROUPED_MAP_ARROW UDF expected here."
udf = udfs[0]
- self._grouped_udf = udf.func
# signature was lost when wrapping it
self._num_udf_args = len(inspect.getfullargspec(udf.raw_func).args)
parsed_offsets = extract_key_value_indexes(udf.args_offsets)
@@ -251,10 +248,10 @@ class
ArrowGroupedMapUDFHandler(GroupedEvalTypeHandler["pa.RecordBatch"]):
# Call UDF
value_table = pa.Table.from_batches(value_batches)
if self._num_udf_args == 1:
- result = self._grouped_udf(value_table)
+ result = self._funcs[0](value_table)
else:
key = tuple(c[0] for c in keys.columns)
- result = self._grouped_udf(key, value_table)
+ result = self._funcs[0](key, value_table)
verify_return_type(result, pa.Table)
# Verify types (and reorder by name when configured).
@@ -292,11 +289,11 @@ class
ArrowGroupedAggUDFHandler(GroupedEvalTypeHandler["pa.RecordBatch"]):
for group in data:
concatenated = ArrowBatchTransformer.concat_batches(group)
results = [
- udf.func(
+ func(
*[concatenated.column(o) for o in udf.args_offsets],
**{k: concatenated.column(v) for k, v in
udf.kwargs_offsets.items()},
)
- for udf in self._udfs
+ for udf, func in zip(self._udfs, self._funcs)
]
result_arrays = [pa.array([r]) for r in results]
batch = pa.RecordBatch.from_arrays(result_arrays, self._col_names)
@@ -314,7 +311,6 @@ class
ArrowGroupedAggIterUDFHandler(GroupedEvalTypeHandler["pa.RecordBatch"]):
super().__init__(udfs, runner_conf, eval_conf)
assert len(udfs) == 1, "One GROUPED_AGG_ARROW_ITER UDF expected here."
udf = udfs[0]
- self._udf_func, self._args_offsets = udf.func, udf.args_offsets
self._return_schema = to_arrow_schema(
StructType([StructField("_0", udf.return_type)]),
timezone="UTC",
@@ -324,7 +320,7 @@ class
ArrowGroupedAggIterUDFHandler(GroupedEvalTypeHandler["pa.RecordBatch"]):
def run(self, split_index: int, data: Iterator[GroupedBatch]) ->
Iterator[pa.RecordBatch]:
import pyarrow as pa
- args_offsets = self._args_offsets
+ args_offsets = self._udfs[0].args_offsets
def extract_args(batch: pa.RecordBatch) -> Any:
args = tuple(batch.column(o) for o in args_offsets)
@@ -332,7 +328,7 @@ class
ArrowGroupedAggIterUDFHandler(GroupedEvalTypeHandler["pa.RecordBatch"]):
for group in data:
batch_iter = map(extract_args, group)
- result = self._udf_func(batch_iter)
+ result = self._funcs[0](batch_iter)
# Drain remaining batches to maintain stream position
for _ in batch_iter:
pass
@@ -367,8 +363,7 @@ class
ArrowWindowAggUDFHandler(GroupedEvalTypeHandler["pa.RecordBatch"]):
num_rows = concatenated.num_rows
result_arrays = []
- for udf_index, udf in enumerate(self._udfs):
- udf_func = udf.func
+ for udf_index, (udf, udf_func) in enumerate(zip(self._udfs,
self._funcs)):
args_offsets, kwargs_offsets = udf.args_offsets,
udf.kwargs_offsets
bound_type = self._window_bound_types[udf_index]
if bound_type == "unbounded":
@@ -688,7 +683,6 @@ class
ArrowMapUDFHandler(BatchEvalTypeHandler["pa.RecordBatch"]):
require_minimum_pyarrow_version()
super().__init__(udfs, runner_conf, eval_conf)
assert len(udfs) == 1, "One MAP_ARROW_ITER UDF expected here."
- self._udf_func = udfs[0].func
def run(self, split_index: int, data: Iterator[pa.RecordBatch]) ->
Iterator[pa.RecordBatch]:
import pyarrow as pa
@@ -697,7 +691,7 @@ class
ArrowMapUDFHandler(BatchEvalTypeHandler["pa.RecordBatch"]):
input_batches = map(ArrowBatchTransformer.flatten_struct, data)
# invoke the UDF
- output_batches = self._udf_func(input_batches)
+ output_batches = self._funcs[0](input_batches)
# The declared signature is Iterator[...], so a strict iterator is
required by
# default. With the legacy flag, accept any object Python can iterate
over -- via
@@ -731,7 +725,6 @@ class
ArrowScalarIterUDFHandler(BatchEvalTypeHandler["pa.RecordBatch"]):
super().__init__(udfs, runner_conf, eval_conf)
assert len(udfs) == 1, "One SCALAR_ARROW_ITER UDF expected here."
udf = udfs[0]
- self._udf_func, self._args_offsets = udf.func, udf.args_offsets
self._arrow_return_type = to_arrow_type(
udf.return_type, timezone="UTC",
prefers_large_types=runner_conf.use_large_var_types
)
@@ -739,7 +732,7 @@ class
ArrowScalarIterUDFHandler(BatchEvalTypeHandler["pa.RecordBatch"]):
def run(self, split_index: int, data: Iterator[pa.RecordBatch]) ->
Iterator[pa.RecordBatch]:
import pyarrow as pa
- args_offsets = self._args_offsets
+ args_offsets = self._udfs[0].args_offsets
num_input_rows = 0
def extract_args(batch: pa.RecordBatch) -> Any:
@@ -752,7 +745,7 @@ class
ArrowScalarIterUDFHandler(BatchEvalTypeHandler["pa.RecordBatch"]):
args_iter = map(extract_args, data)
# Call UDF and verify result type (iterator of pa.Array)
- verified_iter = verify_return_type(self._udf_func(args_iter),
Iterator[pa.Array])
+ verified_iter = verify_return_type(self._funcs[0](args_iter),
Iterator[pa.Array])
# Process results: enforce schema and assemble into RecordBatch
target_schema = pa.schema([pa.field("_0", self._arrow_return_type)])
@@ -797,11 +790,11 @@ class
ArrowScalarUDFHandler(BatchEvalTypeHandler["pa.RecordBatch"]):
for batch in data:
output_batch = pa.RecordBatch.from_arrays(
[
- udf.func(
+ func(
*[batch.column(o) for o in udf.args_offsets],
**{k: batch.column(v) for k, v in
udf.kwargs_offsets.items()},
)
- for udf in self._udfs
+ for udf, func in zip(self._udfs, self._funcs)
],
self._col_names,
)
@@ -820,6 +813,8 @@ class
ArrowBatchedUDFArrowHandler(BatchEvalTypeHandler["pa.RecordBatch"]):
eval_type = PythonEvalType.SQL_ARROW_BATCHED_UDF
variant_conf = "use_legacy_pandas_udf_conversion"
variant_value = False
+ # run wraps the row-by-row evaluation in fail_on_stopiteration itself.
+ _fail_on_stopiteration = False
def __init__(self, udfs: list[UDF], runner_conf: RunnerConf, eval_conf:
EvalConf) -> None:
require_minimum_pyarrow_version()
@@ -827,9 +822,9 @@ class
ArrowBatchedUDFArrowHandler(BatchEvalTypeHandler["pa.RecordBatch"]):
# --- UDF preparation ---
self._udf_infos = []
- for udf in udfs:
+ for udf, func in zip(udfs, self._funcs):
wrapped_func, args_kwargs_offsets = wrap_kwargs_support(
- udf.func, udf.args_offsets, udf.kwargs_offsets
+ func, udf.args_offsets, udf.kwargs_offsets
)
zero_arg = len(args_kwargs_offsets) == 0
self._udf_infos.append(
diff --git a/python/pyspark/eval_handlers/_base.py
b/python/pyspark/eval_handlers/_base.py
index a4793c5ad07a..d9bc7354b2b2 100644
--- a/python/pyspark/eval_handlers/_base.py
+++ b/python/pyspark/eval_handlers/_base.py
@@ -17,13 +17,13 @@
"""Base classes and registry for the eval type handlers.
-This leaf module imports only ``_typing`` and the serializers, so the package
-``__init__`` and the concrete-handler submodules can both import it.
+This leaf module imports only ``_typing``, the serializers and
``pyspark.util``, so the
+package ``__init__`` and the concrete-handler submodules can both import it.
"""
from abc import ABCMeta, abstractmethod
from collections.abc import Iterator
-from typing import TYPE_CHECKING, Any, ClassVar, Generic, Optional
+from typing import TYPE_CHECKING, Any, Callable, ClassVar, Generic, Optional
from pyspark.eval_handlers._typing import (
UDF,
@@ -39,6 +39,7 @@ from pyspark.sql.pandas.serializers import (
ArrowStreamGroupSerializer,
ArrowStreamSerializer,
)
+from pyspark.util import fail_on_stopiteration
if TYPE_CHECKING:
import pyarrow as pa # noqa: F401 # only in the batch category's
forward-ref subscript
@@ -149,6 +150,9 @@ class EvalTypeHandler(Generic[InputBatch, OutputBatch],
metaclass=_EvalTypeHandl
variant_value: ClassVar[Any] = None
# Whether the handler's UDFs return an iterator.
_iter_based: ClassVar[bool] = False
+ # Whether __init__ wraps the UDF in fail_on_stopiteration, so a
StopIteration raised in
+ # user code fails the task instead of silently ending the loop that calls
the UDF.
+ _fail_on_stopiteration: ClassVar[bool] = True
def __init__(
self, udfs: "list[UDF] | list[UDTF]", runner_conf: "RunnerConf",
eval_conf: "EvalConf"
@@ -156,6 +160,115 @@ class EvalTypeHandler(Generic[InputBatch, OutputBatch],
metaclass=_EvalTypeHandl
self._udfs: list[Any] = udfs
self._runner_conf = runner_conf
self._eval_conf = eval_conf
+ # Each UDF's function wrapped for profiling and, unless
_fail_on_stopiteration is False,
+ # in fail_on_stopiteration, at the indexes of _udfs; empty for a UDTF.
+ self._funcs: list[Callable[..., Any]] = []
+ for udf in udfs:
+ if isinstance(udf, UDF):
+ profiling_func = self._with_profiler(udf.raw_func,
udf.result_id)
+ self._funcs.append(
+ fail_on_stopiteration(profiling_func)
+ if self._fail_on_stopiteration
+ else profiling_func
+ )
+
+ def _with_profiler(self, f: Callable[..., Any], result_id: int) ->
Callable[..., Any]:
+ """Return ``f`` profiled by the profiler its config selects, or ``f``
itself when none is
+ set."""
+ # If f is from pyspark.sql.worker, it is to read/write data source.
+ # In this case, we check the data_source_profiler config.
+ module = getattr(f, "__module__", "")
+ if isinstance(module, str) and
module.startswith("pyspark.sql.worker."):
+ profiler = self._runner_conf.data_source_profiler
+ else:
+ profiler = self._runner_conf.udf_profiler
+ if profiler == "perf":
+ return self._with_perf_profiler(f, result_id)
+ elif profiler == "memory":
+ return self._with_memory_profiler(f, result_id)
+ else:
+ return f
+
+ def _with_perf_profiler(self, f: Callable[..., Any], result_id: int) ->
Callable[..., Any]:
+ from pyspark.accumulators import SpecialAccumulatorIds,
_deserialize_accumulator
+ from pyspark.sql.profiler import (
+ ProfileResultsParam,
+ ProfileResultsParamV2,
+ WorkerPerfProfiler,
+ )
+
+ accumulator = _deserialize_accumulator(
+ SpecialAccumulatorIds.SQL_UDF_PROFIER, None, ProfileResultsParam
+ )
+ accumulator_v2 = _deserialize_accumulator(
+ SpecialAccumulatorIds.SQL_UDF_PROFIER_V2, {}, ProfileResultsParamV2
+ )
+
+ if self._iter_based:
+
+ def profiling_func(*args: Any, **kwargs: Any) -> Any:
+ iterator = iter(f(*args, **kwargs))
+ while True:
+ try:
+ with WorkerPerfProfiler(accumulator, accumulator_v2,
result_id):
+ item = next(iterator)
+ yield item
+ except StopIteration:
+ break
+
+ else:
+
+ def profiling_func(*args: Any, **kwargs: Any) -> Any:
+ with WorkerPerfProfiler(accumulator, accumulator_v2,
result_id):
+ ret = f(*args, **kwargs)
+ return ret
+
+ return profiling_func
+
+ def _with_memory_profiler(self, f: Callable[..., Any], result_id: int) ->
Callable[..., Any]:
+ import pyspark.memory_profiler_ext
+ from pyspark.accumulators import SpecialAccumulatorIds,
_deserialize_accumulator
+ from pyspark.sql.profiler import (
+ ProfileResultsParam,
+ ProfileResultsParamV2,
+ WorkerMemoryProfiler,
+ )
+
+ if not pyspark.memory_profiler_ext.has_memory_profiler:
+ return f
+
+ accumulator = _deserialize_accumulator(
+ SpecialAccumulatorIds.SQL_UDF_PROFIER, None, ProfileResultsParam
+ )
+
+ accumulator_v2 = _deserialize_accumulator(
+ SpecialAccumulatorIds.SQL_UDF_PROFIER_V2, {}, ProfileResultsParamV2
+ )
+
+ if self._iter_based:
+
+ def profiling_func(*args: Any, **kwargs: Any) -> Any:
+ g = f(*args, **kwargs)
+ iterator = iter(g)
+
+ while True:
+ try:
+ with WorkerMemoryProfiler(
+ accumulator, accumulator_v2, result_id, g.gi_code
+ ):
+ item = next(iterator)
+ yield item
+ except StopIteration:
+ break
+
+ else:
+
+ def profiling_func(*args: Any, **kwargs: Any) -> Any:
+ with WorkerMemoryProfiler(accumulator, accumulator_v2,
result_id, f):
+ ret = f(*args, **kwargs)
+ return ret
+
+ return profiling_func
@property
@abstractmethod
diff --git a/python/pyspark/eval_handlers/_elementwise.py
b/python/pyspark/eval_handlers/_elementwise.py
index 2a836cabaf6e..c5a1ce4e3dd3 100644
--- a/python/pyspark/eval_handlers/_elementwise.py
+++ b/python/pyspark/eval_handlers/_elementwise.py
@@ -200,7 +200,7 @@ class
ArrowElementwiseUDFHandler(BatchEvalTypeHandler["pa.RecordBatch"]):
for udf_index, udf in enumerate(udfs):
udf_return_type = udf.return_type
wrapped_func, args_kwargs_offsets = wrap_kwargs_support(
- udf.func, udf.args_offsets, udf.kwargs_offsets
+ self._funcs[udf_index], udf.args_offsets, udf.kwargs_offsets
)
depth = nesting[udf_index] if nesting is not None else 1
# Each argument arrives as ``array^depth<T>``; convert its leaves
with the element type
@@ -395,7 +395,7 @@ class
_ScalarElementwiseUDFHandler(_VectorizedElementwiseUDFHandler):
for udf_index, udf in enumerate(udfs):
udf_return_type = udf.return_type
wrapped_func, args_kwargs_offsets = wrap_kwargs_support(
- udf.func, udf.args_offsets, udf.kwargs_offsets
+ self._funcs[udf_index], udf.args_offsets, udf.kwargs_offsets
)
udf_output_schema = _elementwise_output_schema(udf_return_type,
runner_conf)
depth = nesting[udf_index] if nesting is not None else 1
@@ -518,8 +518,6 @@ class
_ScalarIterElementwiseUDFHandler(_VectorizedElementwiseUDFHandler):
assert len(udfs) == 1, "One SCALAR_*_ITER_ELEMENTWISE UDF expected
here."
udf = udfs[0]
- self._udf_func, self._args_offsets = udf.func, udf.args_offsets
- self._return_type = udf.return_type
assert not udf.kwargs_offsets, "Iterator UDFs do not take keyword
arguments."
# Each argument arrives as ``array^depth<T>``; the vectorized function
must see the leaf
@@ -528,9 +526,9 @@ class
_ScalarIterElementwiseUDFHandler(_VectorizedElementwiseUDFHandler):
nesting = eval_conf.elementwise_nesting
self._depth = nesting[0] if nesting is not None else 1
self._udf_input_schema = _elementwise_input_schema(
- eval_conf.input_type, self._args_offsets, self._depth
+ eval_conf.input_type, udf.args_offsets, self._depth
)
- self._udf_output_schema =
_elementwise_output_schema(self._return_type, runner_conf)
+ self._udf_output_schema = _elementwise_output_schema(udf.return_type,
runner_conf)
@abstractmethod
def _verify_return_type(self, results: Any) -> Iterator[Any]:
@@ -552,7 +550,7 @@ class
_ScalarIterElementwiseUDFHandler(_VectorizedElementwiseUDFHandler):
# shapes re-nest this batch's rows. The user function sees the
flat leaves as a
# pandas Series / DataFrame (pandas) or a pa.Array (Arrow), each
with its leaf type.
flat_batch, list_lengths_by_level, large_list_by_level =
_elementwise_flatten_inputs(
- batch, self._args_offsets, self._depth
+ batch, self._udfs[0].args_offsets, self._depth
)
pending_shapes.append((list_lengths_by_level, large_list_by_level,
flat_batch.num_rows))
num_input_elements += flat_batch.num_rows
@@ -561,7 +559,7 @@ class
_ScalarIterElementwiseUDFHandler(_VectorizedElementwiseUDFHandler):
flat_args_iter = map(extract_flat, data)
- verified_iter =
self._verify_return_type(self._udf_func(flat_args_iter))
+ verified_iter =
self._verify_return_type(self._funcs[0](flat_args_iter))
# Buffer the streamed flat element results and emit an ``array<R>``
row as soon as the
# shape at the head of the FIFO is fully covered. A row whose length
is 0 (an empty
@@ -603,7 +601,9 @@ class
_ScalarIterElementwiseUDFHandler(_VectorizedElementwiseUDFHandler):
def process_results() -> Iterator[pa.RecordBatch]:
nonlocal pending_chunks, pending_len, empty_schema,
num_output_elements
for result in verified_iter:
- chunk = self._to_flat_batch(result, self._return_type,
self._udf_output_schema)
+ chunk = self._to_flat_batch(
+ result, self._udfs[0].return_type, self._udf_output_schema
+ )
num_output_elements += chunk.num_rows
# Fail fast if the UDF over-produces, before the buffer grows
unbounded (the
# base iterator paths do the same via verify_output_row_limit).
@@ -653,13 +653,14 @@ class PandasScalarIterElementwiseUDFHandler(
an iterator of pandas Series (DataFrames for struct types)."""
eval_type = PythonEvalType.SQL_SCALAR_PANDAS_ITER_ELEMENTWISE_UDF
+ _fail_on_stopiteration = False
def _verify_return_type(self, results: Any) -> Iterator[Any]:
import pandas as pd
pandas_iter_type = (
Iterator[pd.DataFrame]
- if isinstance(self._return_type, StructType)
+ if isinstance(self._udfs[0].return_type, StructType)
else Iterator[pd.Series]
)
return verify_return_type(results, pandas_iter_type)
diff --git a/python/pyspark/eval_handlers/_pandas.py
b/python/pyspark/eval_handlers/_pandas.py
index 744c785520e4..9a503ee425ab 100644
--- a/python/pyspark/eval_handlers/_pandas.py
+++ b/python/pyspark/eval_handlers/_pandas.py
@@ -102,9 +102,9 @@ class
PandasScalarUDFHandler(BatchEvalTypeHandler["pa.RecordBatch"]):
# Process: evaluate each UDF column-wise on pandas Series.
results = []
- for udf in self._udfs:
+ for udf, func in zip(self._udfs, self._funcs):
return_type = udf.return_type
- result = udf.func(
+ result = func(
*[pandas_columns[o] for o in udf.args_offsets],
**{k: pandas_columns[v] for k, v in
udf.kwargs_offsets.items()},
)
@@ -150,6 +150,7 @@ class
PandasScalarIterUDFHandler(BatchEvalTypeHandler["pa.RecordBatch"]):
eval_type = PythonEvalType.SQL_SCALAR_PANDAS_ITER_UDF
_iter_based = True
+ _fail_on_stopiteration = False
def __init__(self, udfs: list[UDF], runner_conf: "RunnerConf", eval_conf:
"EvalConf") -> None:
import pandas as pd
@@ -159,12 +160,10 @@ class
PandasScalarIterUDFHandler(BatchEvalTypeHandler["pa.RecordBatch"]):
super().__init__(udfs, runner_conf, eval_conf)
assert len(udfs) == 1, "One SCALAR_PANDAS_ITER UDF expected here."
udf = udfs[0]
- self._udf_func, self._args_offsets = udf.func, udf.args_offsets
- self._return_type = udf.return_type
- self._return_schema = StructType([StructField("_0",
self._return_type)])
+ self._return_schema = StructType([StructField("_0", udf.return_type)])
self._expected_iter_type = (
Iterator[pd.DataFrame]
- if isinstance(self._return_type, StructType)
+ if isinstance(udf.return_type, StructType)
else Iterator[pd.Series]
)
@@ -182,18 +181,21 @@ class
PandasScalarIterUDFHandler(BatchEvalTypeHandler["pa.RecordBatch"]):
prefer_int_ext_dtype=self._runner_conf.prefer_int_ext_dtype,
df_for_struct=True,
)
- args = tuple(pandas_columns[o] for o in self._args_offsets)
+ args = tuple(pandas_columns[o] for o in self._udfs[0].args_offsets)
num_input_rows += batch.num_rows
return args[0] if len(args) == 1 else args
# Extract args from input batches (streaming), then call the UDF and
verify the
# result type (iterator of pd.Series / pd.DataFrame).
args_iter = map(extract_args, data)
- verified_iter = verify_return_type(self._udf_func(args_iter),
self._expected_iter_type)
+ verified_iter = verify_return_type(self._funcs[0](args_iter),
self._expected_iter_type)
def convert(result: Any) -> "pa.RecordBatch":
verify_pandas_result(
- result, self._return_type, assign_cols_by_name=True,
truncate_return_schema=True
+ result,
+ self._udfs[0].return_type,
+ assign_cols_by_name=True,
+ truncate_return_schema=True,
)
return PandasToArrowConversion.from_pandas(
[result],
@@ -230,11 +232,10 @@ class
PandasMapUDFHandler(BatchEvalTypeHandler["pa.RecordBatch"]):
require_minimum_pyarrow_version()
super().__init__(udfs, runner_conf, eval_conf)
assert len(udfs) == 1, "One MAP_PANDAS_ITER UDF expected here."
- self._udf_func, self._return_type = udfs[0].func, udfs[0].return_type
- self._return_schema = StructType([StructField("_0",
self._return_type)])
+ self._return_schema = StructType([StructField("_0",
udfs[0].return_type)])
self._expected_iter_type = (
Iterator[pd.DataFrame]
- if isinstance(self._return_type, StructType)
+ if isinstance(udfs[0].return_type, StructType)
else Iterator[pd.Series]
)
@@ -250,7 +251,7 @@ class
PandasMapUDFHandler(BatchEvalTypeHandler["pa.RecordBatch"]):
df_for_struct=True,
)[0]
- result = self._udf_func(dataframe_iter())
+ result = self._funcs[0](dataframe_iter())
# The declared signature is Iterator[...], so a strict iterator is
required by
# default. With the legacy flag, accept any object Python can iterate
over -- via
# iter(...), which honors both __iter__ and the sequence protocol
(__getitem__) --
@@ -267,7 +268,7 @@ class
PandasMapUDFHandler(BatchEvalTypeHandler["pa.RecordBatch"]):
for df in verify_return_type(result, self._expected_iter_type):
verify_pandas_result(
- df, self._return_type, assign_cols_by_name=True,
truncate_return_schema=True
+ df, self._udfs[0].return_type, assign_cols_by_name=True,
truncate_return_schema=True
)
yield PandasToArrowConversion.from_pandas(
[df],
@@ -295,13 +296,12 @@ class
PandasCoGroupedMapUDFHandler(CoGroupedEvalTypeHandler["pa.RecordBatch"]):
super().__init__(udfs, runner_conf, eval_conf)
assert len(udfs) == 1, "One COGROUPED_MAP_PANDAS UDF expected here."
udf = udfs[0]
- self._cogrouped_udf, self._return_type = udf.func, udf.return_type
# signature was lost when wrapping it
self._num_udf_args = len(inspect.getfullargspec(udf.raw_func).args)
parsed_offsets = extract_key_value_indexes(udf.args_offsets)
self._left_key_offsets, self._left_value_offsets = parsed_offsets[0]
self._right_key_offsets, self._right_value_offsets = parsed_offsets[1]
- self._return_schema = StructType([StructField("_0",
self._return_type)])
+ self._return_schema = StructType([StructField("_0", udf.return_type)])
def run(self, split_index: int, data: "Iterator[CoGroupedBatch]") ->
"Iterator[pa.RecordBatch]":
"""Apply cogroupBy Pandas UDF.
@@ -330,7 +330,7 @@ class
PandasCoGroupedMapUDFHandler(CoGroupedEvalTypeHandler["pa.RecordBatch"]):
right_df = pd.concat([right_series[o] for o in
self._right_value_offsets], axis=1)
if self._num_udf_args == 2:
- result = self._cogrouped_udf(left_df, right_df)
+ result = self._funcs[0](left_df, right_df)
else:
key_series = (
[left_series[o] for o in self._left_key_offsets]
@@ -338,14 +338,14 @@ class
PandasCoGroupedMapUDFHandler(CoGroupedEvalTypeHandler["pa.RecordBatch"]):
else [right_series[o] for o in self._right_key_offsets]
)
key = tuple(s.iloc[0] for s in key_series)
- result = self._cogrouped_udf(key, left_df, right_df)
+ result = self._funcs[0](key, left_df, right_df)
del left_batches, right_batches, left_table, right_table
del left_series, right_series, left_df, right_df
verify_pandas_result(
result,
- self._return_type,
+ self._udfs[0].return_type,
self._runner_conf.assign_cols_by_name,
truncate_return_schema=False,
)
@@ -378,7 +378,6 @@ class
PandasGroupedMapIterUDFHandler(GroupedEvalTypeHandler["pa.RecordBatch"]):
super().__init__(udfs, runner_conf, eval_conf)
assert len(udfs) == 1, "One GROUPED_MAP_PANDAS_ITER UDF expected here."
udf = udfs[0]
- self._grouped_udf, self._return_type = udf.func, udf.return_type
# signature was lost when wrapping it
self._num_udf_args = len(inspect.getfullargspec(udf.raw_func).args)
parsed_offsets = extract_key_value_indexes(udf.args_offsets)
@@ -386,7 +385,7 @@ class
PandasGroupedMapIterUDFHandler(GroupedEvalTypeHandler["pa.RecordBatch"]):
"Expected one pair of offsets for GROUPED_MAP_PANDAS_ITER UDF."
)
self._key_offsets, self._value_offsets = parsed_offsets[0]
- self._return_schema = StructType([StructField("_0",
self._return_type)])
+ self._return_schema = StructType([StructField("_0", udf.return_type)])
def run(self, split_index: int, data: "Iterator[GroupedBatch]") ->
"Iterator[pa.RecordBatch]":
"""Apply groupBy Pandas UDF (iterator variant)."""
@@ -414,15 +413,15 @@ class
PandasGroupedMapIterUDFHandler(GroupedEvalTypeHandler["pa.RecordBatch"]):
yield pd.concat([series[o] for o in value_offsets], axis=1)
if self._num_udf_args == 1:
- result = self._grouped_udf(dataframe_iter())
+ result = self._funcs[0](dataframe_iter())
else:
key = tuple(first_series[o].iloc[0] for o in self._key_offsets)
- result = self._grouped_udf(key, dataframe_iter())
+ result = self._funcs[0](key, dataframe_iter())
for df in result:
verify_pandas_result(
df,
- self._return_type,
+ self._udfs[0].return_type,
self._runner_conf.assign_cols_by_name,
truncate_return_schema=False,
)
@@ -459,13 +458,12 @@ class
PandasGroupedMapUDFHandler(GroupedEvalTypeHandler["pa.RecordBatch"]):
super().__init__(udfs, runner_conf, eval_conf)
assert len(udfs) == 1, "One GROUPED_MAP_PANDAS UDF expected here."
udf = udfs[0]
- self._grouped_udf, self._return_type = udf.func, udf.return_type
# signature was lost when wrapping it
self._num_udf_args = len(inspect.getfullargspec(udf.raw_func).args)
parsed_offsets = extract_key_value_indexes(udf.args_offsets)
assert len(parsed_offsets) == 1, "Expected one pair of offsets for
GROUPED_MAP_PANDAS UDF."
self._key_offsets, self._value_offsets = parsed_offsets[0]
- self._return_schema = StructType([StructField("_0",
self._return_type)])
+ self._return_schema = StructType([StructField("_0", udf.return_type)])
def run(self, split_index: int, data: "Iterator[GroupedBatch]") ->
"Iterator[pa.RecordBatch]":
# Bound each output RecordBatch toward the byte cap (no limit when
unset).
@@ -501,16 +499,16 @@ class
PandasGroupedMapUDFHandler(GroupedEvalTypeHandler["pa.RecordBatch"]):
value_df = pd.concat([all_series[o] for o in self._value_offsets],
axis=1)
if self._num_udf_args == 1:
- result = self._grouped_udf(value_df)
+ result = self._funcs[0](value_df)
else:
key = tuple(all_series[o].iloc[0] for o in self._key_offsets)
- result = self._grouped_udf(key, value_df)
+ result = self._funcs[0](key, value_df)
del all_batches, table, all_series, value_df
verify_pandas_result(
result,
- self._return_type,
+ self._udfs[0].return_type,
self._runner_conf.assign_cols_by_name,
truncate_return_schema=False,
)
@@ -559,11 +557,11 @@ class
PandasGroupedAggUDFHandler(GroupedEvalTypeHandler["pa.RecordBatch"]):
prefer_int_ext_dtype=self._runner_conf.prefer_int_ext_dtype,
)
results = [
- udf.func(
+ func(
*[all_series[o] for o in udf.args_offsets],
**{k: all_series[v] for k, v in
udf.kwargs_offsets.items()},
)
- for udf in self._udfs
+ for udf, func in zip(self._udfs, self._funcs)
]
yield PandasToArrowConversion.from_pandas(
[pd.Series([r]) for r in results],
@@ -591,13 +589,12 @@ class
PandasGroupedAggIterUDFHandler(GroupedEvalTypeHandler["pa.RecordBatch"]):
super().__init__(udfs, runner_conf, eval_conf)
assert len(udfs) == 1, "One GROUPED_AGG_PANDAS_ITER UDF expected here."
udf = udfs[0]
- self._udf_func, self._args_offsets = udf.func, udf.args_offsets
self._return_schema = StructType([StructField("_0", udf.return_type)])
def run(self, split_index: int, data: "Iterator[GroupedBatch]") ->
"Iterator[pa.RecordBatch]":
import pandas as pd
- args_offsets = self._args_offsets
+ args_offsets = self._udfs[0].args_offsets
def extract_series(batch: "pa.RecordBatch") -> Any:
# Convert one RecordBatch to a pandas Series per column, then
select args:
@@ -613,7 +610,7 @@ class
PandasGroupedAggIterUDFHandler(GroupedEvalTypeHandler["pa.RecordBatch"]):
for group in data:
series_iter = map(extract_series, group)
- result = self._udf_func(series_iter)
+ result = self._funcs[0](series_iter)
# Drain remaining batches to maintain stream position
for _ in series_iter:
pass
@@ -663,8 +660,7 @@ class
PandasWindowAggUDFHandler(GroupedEvalTypeHandler["pa.RecordBatch"]):
num_rows = table.num_rows
result_series = []
- for udf_index, udf in enumerate(self._udfs):
- udf_func = udf.func
+ for udf_index, (udf, udf_func) in enumerate(zip(self._udfs,
self._funcs)):
args_offsets, kwargs_offsets = udf.args_offsets,
udf.kwargs_offsets
bound_type = self._window_bound_types[udf_index]
if bound_type == "unbounded":
@@ -730,6 +726,8 @@ class
ArrowBatchedUDFLegacyPandasHandler(BatchEvalTypeHandler["pa.RecordBatch"])
eval_type = PythonEvalType.SQL_ARROW_BATCHED_UDF
variant_conf = "use_legacy_pandas_udf_conversion"
variant_value = True
+ # run wraps the row-by-row evaluation in fail_on_stopiteration itself.
+ _fail_on_stopiteration = False
def __init__(self, udfs: list[UDF], runner_conf: "RunnerConf", eval_conf:
"EvalConf") -> None:
require_minimum_pandas_version()
@@ -738,10 +736,10 @@ class
ArrowBatchedUDFLegacyPandasHandler(BatchEvalTypeHandler["pa.RecordBatch"])
# --- UDF preparation ---
self._udf_infos = []
- for udf in udfs:
+ for udf, func in zip(udfs, self._funcs):
udf_return_type = udf.return_type
wrapped_func, args_kwargs_offsets = wrap_kwargs_support(
- udf.func, udf.args_offsets, udf.kwargs_offsets
+ func, udf.args_offsets, udf.kwargs_offsets
)
zero_arg = len(args_kwargs_offsets) == 0
# Legacy coerces String/Binary for Arrow compatibility
diff --git a/python/pyspark/eval_handlers/_stateful.py
b/python/pyspark/eval_handlers/_stateful.py
index e8a1f49660ce..87f38371cf6c 100644
--- a/python/pyspark/eval_handlers/_stateful.py
+++ b/python/pyspark/eval_handlers/_stateful.py
@@ -107,7 +107,6 @@ class
PandasGroupedMapWithStateUDFHandler(BatchEvalTypeHandler["pa.RecordBatch"]
# See FlatMapGroupsInPandasWithStateExec for how arg_offsets are used
to
# distinguish between grouping attributes and data attributes.
- self._func, self._return_type = udfs[0].func, udfs[0].return_type
parsed_offsets = extract_key_value_indexes(udfs[0].args_offsets)
self._key_offsets = parsed_offsets[0][0]
self._value_offsets = parsed_offsets[0][1]
@@ -228,13 +227,13 @@ class
PandasGroupedMapWithStateUDFHandler(BatchEvalTypeHandler["pa.RecordBatch"]
# The number of columns of the result must match the return type,
# but an empty result with no columns at all is acceptable.
if not (
- len(result.columns) == len(self._return_type)
+ len(result.columns) == len(self._udfs[0].return_type)
or (len(result.columns) == 0 and result.empty)
):
raise PySparkRuntimeError(
errorClass="RESULT_COLUMN_SCHEMA_MISMATCH",
messageParameters={
- "expected": str(len(self._return_type)),
+ "expected": str(len(self._udfs[0].return_type)),
"actual": str(len(result.columns)),
},
)
@@ -262,7 +261,7 @@ class
PandasGroupedMapWithStateUDFHandler(BatchEvalTypeHandler["pa.RecordBatch"]
else:
values = (pd.concat(x, axis=1) for x in value_series_gen)
- result_iter = self._func(key, values, state)
+ result_iter = self._funcs[0](key, values, state)
if isinstance(result_iter, pd.DataFrame):
raise PySparkTypeError(
@@ -393,7 +392,11 @@ class
PandasGroupedMapWithStateUDFHandler(BatchEvalTypeHandler["pa.RecordBatch"]
# Flush a batch once the record threshold is exceeded.
if pdf_data_cnt > self._arrow_max_records_per_batch:
yield construct_record_batch(
- pdfs, pdf_data_cnt, self._return_type, state_pdfs,
state_data_cnt
+ pdfs,
+ pdf_data_cnt,
+ self._udfs[0].return_type,
+ state_pdfs,
+ state_data_cnt,
)
pdfs = []
state_pdfs = []
@@ -408,7 +411,7 @@ class
PandasGroupedMapWithStateUDFHandler(BatchEvalTypeHandler["pa.RecordBatch"]
# Flush the trailing batch if it has any data or state left.
if pdf_data_cnt > 0 or state_data_cnt > 0:
yield construct_record_batch(
- pdfs, pdf_data_cnt, self._return_type, state_pdfs,
state_data_cnt
+ pdfs, pdf_data_cnt, self._udfs[0].return_type, state_pdfs,
state_data_cnt
)
@@ -426,7 +429,6 @@ class
PandasTransformWithStateUDFHandler(BatchEvalTypeHandler["pa.RecordBatch"])
super().__init__(udfs, runner_conf, eval_conf)
assert len(udfs) == 1, "One TRANSFORM_WITH_STATE_PANDAS UDF expected
here."
- self._func, self._return_type = udfs[0].func, udfs[0].return_type
# See TransformWithStateInPandasExec for how arg_offsets are used to
# distinguish between grouping attributes and data attributes
@@ -437,7 +439,7 @@ class
PandasTransformWithStateUDFHandler(BatchEvalTypeHandler["pa.RecordBatch"])
self._key_offsets = parsed_offsets[0][0]
self._value_offsets = parsed_offsets[0][1]
- self._output_schema = StructType([StructField("_0",
self._return_type)])
+ self._output_schema = StructType([StructField("_0",
udfs[0].return_type)])
self._stateful_processor_api_client = StatefulProcessorApiClient(
eval_conf.state_server_socket_port, # type: ignore[arg-type]
@@ -515,7 +517,7 @@ class
PandasTransformWithStateUDFHandler(BatchEvalTypeHandler["pa.RecordBatch"])
def convert_results(result_iter: Iterable[pd.DataFrame]) ->
Iterator[pa.RecordBatch]:
for result in result_iter:
- if isinstance(self._return_type, StructType) and not
isinstance(
+ if isinstance(self._udfs[0].return_type, StructType) and not
isinstance(
result, pd.DataFrame
):
raise PySparkValueError(
@@ -538,7 +540,7 @@ class
PandasTransformWithStateUDFHandler(BatchEvalTypeHandler["pa.RecordBatch"])
# This must be a generator expression - do not materialize.
values_gen = (df.iloc[:, self._value_offsets] for _, df in group)
yield from convert_results(
- self._func(
+ self._funcs[0](
self._stateful_processor_api_client,
TransformWithStateInPandasFuncMode.PROCESS_DATA,
key,
@@ -547,7 +549,7 @@ class
PandasTransformWithStateUDFHandler(BatchEvalTypeHandler["pa.RecordBatch"])
)
yield from convert_results(
- self._func(
+ self._funcs[0](
self._stateful_processor_api_client,
TransformWithStateInPandasFuncMode.PROCESS_TIMER,
None,
@@ -556,7 +558,7 @@ class
PandasTransformWithStateUDFHandler(BatchEvalTypeHandler["pa.RecordBatch"])
)
yield from convert_results(
- self._func(
+ self._funcs[0](
self._stateful_processor_api_client,
TransformWithStateInPandasFuncMode.COMPLETE,
None,
@@ -579,7 +581,6 @@ class
PandasTransformWithStateInitStateUDFHandler(BatchEvalTypeHandler["pa.Recor
super().__init__(udfs, runner_conf, eval_conf)
assert len(udfs) == 1, "One TRANSFORM_WITH_STATE_PANDAS_INIT_STATE UDF
expected here."
- self._func, self._return_type = udfs[0].func, udfs[0].return_type
# See TransformWithStateInPandasExec for how arg_offsets are used to
# distinguish between grouping attributes and data attributes.
@@ -591,7 +592,7 @@ class
PandasTransformWithStateInitStateUDFHandler(BatchEvalTypeHandler["pa.Recor
parsed_offsets = extract_key_value_indexes(udfs[0].args_offsets)
self._key_offsets = parsed_offsets[0][0]
self._init_key_offsets = parsed_offsets[1][0]
- self._output_schema = StructType([StructField("_0",
self._return_type)])
+ self._output_schema = StructType([StructField("_0",
udfs[0].return_type)])
self._stateful_processor_api_client = StatefulProcessorApiClient(
eval_conf.state_server_socket_port, # type: ignore[arg-type]
@@ -728,7 +729,7 @@ class
PandasTransformWithStateInitStateUDFHandler(BatchEvalTypeHandler["pa.Recor
# TODO(SPARK-49100): add verification that elements in result_iter
are
# indeed of type pd.DataFrame and conform to assigned cols
for result in result_iter:
- if isinstance(self._return_type, StructType) and not
isinstance(
+ if isinstance(self._udfs[0].return_type, StructType) and not
isinstance(
result, pd.DataFrame
):
raise PySparkValueError(
@@ -755,7 +756,7 @@ class
PandasTransformWithStateInitStateUDFHandler(BatchEvalTypeHandler["pa.Recor
state_values = (data_df for _, data_df, _ in group_data if not
data_df.empty)
init_states = (init_df for _, _, init_df in group_init if not
init_df.empty)
yield from convert_results(
- self._func(
+ self._funcs[0](
self._stateful_processor_api_client,
TransformWithStateInPandasFuncMode.PROCESS_DATA,
key,
@@ -765,7 +766,7 @@ class
PandasTransformWithStateInitStateUDFHandler(BatchEvalTypeHandler["pa.Recor
)
yield from convert_results(
- self._func(
+ self._funcs[0](
self._stateful_processor_api_client,
TransformWithStateInPandasFuncMode.PROCESS_TIMER,
None,
@@ -775,7 +776,7 @@ class
PandasTransformWithStateInitStateUDFHandler(BatchEvalTypeHandler["pa.Recor
)
yield from convert_results(
- self._func(
+ self._funcs[0](
self._stateful_processor_api_client,
TransformWithStateInPandasFuncMode.COMPLETE,
None,
@@ -798,7 +799,6 @@ class
RowTransformWithStateUDFHandler(BatchEvalTypeHandler["pa.RecordBatch"]):
super().__init__(udfs, runner_conf, eval_conf)
assert len(udfs) == 1, "One TRANSFORM_WITH_STATE_PYTHON_ROW UDF
expected here."
- self._func, self._return_type = udfs[0].func, udfs[0].return_type
# See TransformWithStateInPySparkExec for how arg_offsets are used to
# distinguish between grouping attributes and data attributes.
@@ -850,7 +850,7 @@ class
RowTransformWithStateUDFHandler(BatchEvalTypeHandler["pa.RecordBatch"]):
# Convert spark type to arrow type
# TODO: we need to make this configurable, currently using default
values.
arrow_type = to_arrow_type(
- self._return_type,
+ self._udfs[0].return_type,
timezone="UTC",
prefers_large_types=False,
)
@@ -865,7 +865,7 @@ class
RowTransformWithStateUDFHandler(BatchEvalTypeHandler["pa.RecordBatch"]):
# This must be a generator expression - do not materialize.
values_gen = map(lambda x: x[1], group)
yield from convert_results(
- self._func(
+ self._funcs[0](
self._stateful_processor_api_client,
TransformWithStateInPandasFuncMode.PROCESS_DATA,
key,
@@ -874,7 +874,7 @@ class
RowTransformWithStateUDFHandler(BatchEvalTypeHandler["pa.RecordBatch"]):
)
yield from convert_results(
- self._func(
+ self._funcs[0](
self._stateful_processor_api_client,
TransformWithStateInPandasFuncMode.PROCESS_TIMER,
None,
@@ -883,7 +883,7 @@ class
RowTransformWithStateUDFHandler(BatchEvalTypeHandler["pa.RecordBatch"]):
)
yield from convert_results(
- self._func(
+ self._funcs[0](
self._stateful_processor_api_client,
TransformWithStateInPandasFuncMode.COMPLETE,
None,
@@ -905,7 +905,6 @@ class
RowTransformWithStateInitStateUDFHandler(BatchEvalTypeHandler["pa.RecordBa
super().__init__(udfs, runner_conf, eval_conf)
assert len(udfs) == 1, "One TRANSFORM_WITH_STATE_PYTHON_ROW_INIT_STATE
UDF expected here."
- self._func, self._return_type = udfs[0].func, udfs[0].return_type
# See TransformWithStateInPandasExec for how arg_offsets are used to
# distinguish between grouping attributes and data attributes.
@@ -1031,7 +1030,7 @@ class
RowTransformWithStateInitStateUDFHandler(BatchEvalTypeHandler["pa.RecordBa
# Convert spark type to arrow type
# TODO: we need to make this configurable, currently using default
values.
arrow_type = to_arrow_type(
- self._return_type,
+ self._udfs[0].return_type,
timezone="UTC",
prefers_large_types=False,
)
@@ -1046,7 +1045,7 @@ class
RowTransformWithStateInitStateUDFHandler(BatchEvalTypeHandler["pa.RecordBa
# These must be generator expressions - do not materialize.
for _, (values_gen, init_states_gen) in group:
yield from convert_results(
- self._func(
+ self._funcs[0](
self._stateful_processor_api_client,
TransformWithStateInPandasFuncMode.PROCESS_DATA,
key,
@@ -1056,7 +1055,7 @@ class
RowTransformWithStateInitStateUDFHandler(BatchEvalTypeHandler["pa.RecordBa
)
yield from convert_results(
- self._func(
+ self._funcs[0](
self._stateful_processor_api_client,
TransformWithStateInPandasFuncMode.PROCESS_TIMER,
None,
@@ -1066,7 +1065,7 @@ class
RowTransformWithStateInitStateUDFHandler(BatchEvalTypeHandler["pa.RecordBa
)
yield from convert_results(
- self._func(
+ self._funcs[0](
self._stateful_processor_api_client,
TransformWithStateInPandasFuncMode.COMPLETE,
None,
diff --git a/python/pyspark/eval_handlers/_typing.py
b/python/pyspark/eval_handlers/_typing.py
index 8d533bd1278f..d01180bd5544 100644
--- a/python/pyspark/eval_handlers/_typing.py
+++ b/python/pyspark/eval_handlers/_typing.py
@@ -21,14 +21,15 @@ The Arrow element types are forward refs so pyarrow stays a
type-checking-only
import at runtime, as elsewhere in the worker.
"""
-from collections.abc import Callable, Iterator
+from collections.abc import Iterator
from dataclasses import dataclass
-from typing import TYPE_CHECKING, Any, Protocol, TypeVar
+from typing import TYPE_CHECKING, Any, Optional, Protocol, TypeVar
if TYPE_CHECKING:
import pyarrow as pa
from pyspark.sql.types import StructType
+ from pyspark.sql.udtf import AnalyzeResult
# Grouped-category input element, matching what the serializer yields: a group
# serializer yields a lazy iterator; a co-group serializer materializes each
side.
@@ -42,19 +43,19 @@ OutputBatch = TypeVar("OutputBatch")
@dataclass(frozen=True)
class UDF:
- """One UDF as read by ``read_single_udf``; each handler takes the fields
it needs."""
+ """One UDF as the worker reads it; each handler takes the fields it
needs."""
- # The chained function wrapped for profiling and, for most eval types, in
- # fail_on_stopiteration.
- func: Callable[..., Any]
- # The chained function before any wrapping, for handlers that need the
pickled object
- # itself (an incremental Aggregator) or its signature.
+ # The chained function before any wrapping: the pickled object itself (an
incremental
+ # Aggregator) with its signature. EvalTypeHandler.__init__ keeps it
wrapped for profiling and,
+ # for most eval types, in fail_on_stopiteration, as the handler's _funcs.
raw_func: Any
args_offsets: list[int]
kwargs_offsets: dict[str, int]
# The DataType of the last function in the chain; a StructType for the
eval types that
# return rows.
return_type: Any
+ # The id the profiler reports the UDF's results under.
+ result_id: int
class UDTFInstance(Protocol):
@@ -66,11 +67,17 @@ class UDTFInstance(Protocol):
@dataclass(frozen=True)
class UDTF:
- """The UDTF as read by ``read_single_udtf``."""
+ """The UDTF as the worker reads it; ``UDTFHandlerBase.__init__``
instantiates it as the
+ handler's ``_udtf``."""
- # Wrapped in the handler's partition_wrapper when its TABLE argument has
PARTITION BY
- # expressions.
- udtf: UDTFInstance
+ # The user's UDTF class, as unpickled.
+ udtf_cls: type
args_offsets: list[int]
kwargs_offsets: dict[str, int]
return_type: "StructType"
+ # The child indexes of the TABLE argument's PARTITION BY expressions, if
any.
+ partition_child_indexes: list[int]
+ # Passed to the UDTF's __init__ when it takes one argument.
+ analyze_result: Optional["AnalyzeResult"]
+ # The UDTF's name in error messages.
+ name: str
diff --git a/python/pyspark/eval_handlers/_udtf.py
b/python/pyspark/eval_handlers/_udtf.py
index c17b7e82653f..2b1b67b348e2 100644
--- a/python/pyspark/eval_handlers/_udtf.py
+++ b/python/pyspark/eval_handlers/_udtf.py
@@ -18,8 +18,8 @@
"""Handlers for the user-defined table function (UDTF) eval types, and the
wrappers that
implement TABLE argument PARTITION BY semantics on top of a UDTF instance.
-Each handler receives the single ``UDTF`` record ``read_single_udtf`` returns,
which holds
-the instantiated UDTF.
+Each handler receives the single ``UDTF`` record the worker reads, and
instantiates the
+UDTF from it.
pyarrow is imported lazily (inside the handlers and the type-checking block)
so the module
stays importable and its handlers register without pyarrow installed.
@@ -27,6 +27,8 @@ stays importable and its handlers register without pyarrow
installed.
from __future__ import annotations
+import dataclasses
+import inspect
import itertools
from abc import ABCMeta, abstractmethod
from collections.abc import Iterable, Iterator
@@ -34,10 +36,14 @@ from typing import TYPE_CHECKING, Any, Callable, ClassVar,
Optional, Tuple
from pyspark.errors import PySparkRuntimeError, PySparkTypeError,
PySparkValueError
from pyspark.eval_handlers._base import BatchEvalTypeHandler, EvalTypeHandler
-from pyspark.eval_handlers._typing import UDTF, InputBatch, OutputBatch
+from pyspark.eval_handlers._typing import UDTF, InputBatch, OutputBatch,
UDTFInstance
from pyspark.eval_handlers.utils import wrap_kwargs_support
from pyspark.eval_handlers.verification import verify_arrow_result,
verify_pandas_result
-from pyspark.serializers import BatchedSerializer, CPickleSerializer,
Serializer
+from pyspark.serializers import (
+ BatchedSerializer,
+ CPickleSerializer,
+ Serializer,
+)
from pyspark.sql.conversion import (
ArrowBatchTransformer,
ArrowTableToRowsConversion,
@@ -592,27 +598,101 @@ def build_null_checker(return_type: StructType) ->
Optional[Callable[[Any], None
return check
-class UDTFEvalTypeHandler(EvalTypeHandler[InputBatch, OutputBatch]):
- """Base for the UDTF eval types.
+class UDTFHandlerBase(EvalTypeHandler[InputBatch, OutputBatch]):
+ """Base for every UDTF eval type handler; ``__init__`` creates the UDTF
instance as
+ ``_udtf``."""
+
+ # Wraps the UDTF when its TABLE argument has PARTITION BY expressions.
+ partition_wrapper: ClassVar[type[PartitionedUDTF]] = UDTFWithPartitions
+
+ def __init__(self, udfs: list[UDTF], runner_conf: RunnerConf, eval_conf:
EvalConf) -> None:
+ """Create, as ``_udtf``, the instance of the UDTF's class (wrapped in
+ ``partition_wrapper`` when its TABLE argument has PARTITION BY
expressions, its
+ constructor given the AnalyzeResult when it takes one), and check its
'eval' signature
+ against the call arguments."""
+ super().__init__(udfs, runner_conf, eval_conf)
+ assert len(udfs) == 1, "One UDTF expected here."
+ udtf = udfs[0]
+ if not isinstance(udtf.udtf_cls, type):
+ raise PySparkRuntimeError(
+ f"Invalid UDTF handler type. Expected a class (type 'type'),
but "
+ f"got an instance of {type(udtf.udtf_cls).__name__}."
+ )
+
+ # Pick how to construct the UDTF from the count of named positional
__init__ parameters
+ # after self. With an AnalyzeResult, one parameter receives a copy of
it, zero means the
+ # class is called with no arguments, and more are an error; without
one, any is an error.
+ handler: Callable[[], Any] = udtf.udtf_cls
+ udtf_init_args = inspect.getfullargspec(udtf.udtf_cls)
+ if udtf.analyze_result is not None:
+ if len(udtf_init_args.args) > 2:
+ raise PySparkRuntimeError(
+
errorClass="UDTF_CONSTRUCTOR_INVALID_IMPLEMENTS_ANALYZE_METHOD",
+ messageParameters={"name": udtf.name},
+ )
+ elif len(udtf_init_args.args) == 2:
+ analyze_result = udtf.analyze_result
+
+ def construct_udtf() -> Any:
+ # Here we pass the AnalyzeResult to the UDTF's __init__
method.
+ return udtf.udtf_cls(dataclasses.replace(analyze_result))
+
+ handler = construct_udtf
+ elif len(udtf_init_args.args) > 1:
+ raise PySparkRuntimeError(
+ errorClass="UDTF_CONSTRUCTOR_INVALID_NO_ANALYZE_METHOD",
+ messageParameters={"name": udtf.name},
+ )
+
+ # Instantiate the UDTF class.
+ try:
+ if len(udtf.partition_child_indexes) > 0:
+ instance = self.partition_wrapper(handler,
udtf.partition_child_indexes)
+ else:
+ instance = handler()
+ except Exception as e:
+ raise PySparkRuntimeError(
+ errorClass="UDTF_EXEC_ERROR",
+ messageParameters={"method_name": "__init__", "error": str(e)},
+ )
+
+ # Validate the UDTF
+ if not hasattr(instance, "eval"):
+ raise PySparkRuntimeError(
+ "Failed to execute the user defined table function because it
has not "
+ "implemented the 'eval' method. Please add the 'eval' method
and try "
+ "the query again."
+ )
+
+ # Check that the arguments provided to the UDTF call match the
expected parameters
+ # defined in the 'eval' method signature.
+ try:
+ inspect.signature(instance.eval).bind(*udtf.args_offsets,
**udtf.kwargs_offsets)
+ except TypeError as e:
+ raise PySparkRuntimeError(
+ errorClass="UDTF_EVAL_METHOD_ARGUMENTS_DO_NOT_MATCH_SIGNATURE",
+ messageParameters={"name": udtf.name, "reason": str(e)},
+ ) from None
+
+ self._udtf: UDTFInstance = instance
+
+
+class UDTFEvalTypeHandler(UDTFHandlerBase[InputBatch, OutputBatch]):
+ """Base for the Arrow UDTF eval types.
``run`` calls ``_eval_batch`` per input batch, then ``_eval_terminate``
once at the end (also
when ``eval`` raised ``SkipRestOfInputTableException``), and always calls
the UDTF's
``cleanup``. Subclasses implement the two hooks and pick the serializer.
"""
- # Wraps the UDTF when its TABLE argument has PARTITION BY expressions; see
read_single_udtf.
- partition_wrapper: ClassVar[type[PartitionedUDTF]] = UDTFWithPartitions
-
def __init__(self, udfs: list[UDTF], runner_conf: RunnerConf, eval_conf:
EvalConf) -> None:
super().__init__(udfs, runner_conf, eval_conf)
assert len(udfs) == 1, "One UDTF expected here."
- udtf = udfs[0].udtf
- self._return_type = udfs[0].return_type
self._eval_method, self._args_kwargs_offsets = wrap_kwargs_support(
- udtf.eval, udfs[0].args_offsets, udfs[0].kwargs_offsets
+ self._udtf.eval, udfs[0].args_offsets, udfs[0].kwargs_offsets
)
- self._terminate_method = getattr(udtf, "terminate", None)
- self._cleanup_method = getattr(udtf, "cleanup", None)
+ self._terminate_method = getattr(self._udtf, "terminate", None)
+ self._cleanup_method = getattr(self._udtf, "cleanup", None)
@abstractmethod
def _eval_batch(self, batch: InputBatch) -> Iterator[OutputBatch]:
@@ -638,33 +718,30 @@ class UDTFEvalTypeHandler(EvalTypeHandler[InputBatch,
OutputBatch]):
cleanup()
-class TableUDTFHandler(EvalTypeHandler[Any, tuple]):
+class TableUDTFHandler(UDTFHandlerBase[Any, tuple]):
"""SQL_TABLE_UDF, the Python UDTF: input rows arrive pickled one at a time
and the UDTF's
``eval`` is called once per row; the results of each call are converted to
the internal
representation and returned to the executor as one tuple."""
eval_type = PythonEvalType.SQL_TABLE_UDF
- # Wraps the UDTF when its TABLE argument has PARTITION BY expressions; see
read_single_udtf.
- partition_wrapper: ClassVar[type[PartitionedUDTF]] = UDTFWithPartitions
def __init__(self, udfs: list[UDTF], runner_conf: RunnerConf, eval_conf:
EvalConf) -> None:
super().__init__(udfs, runner_conf, eval_conf)
assert len(udfs) == 1, "One UDTF expected here."
- udtf, return_type = udfs[0].udtf, udfs[0].return_type
- self._check_output_row_against_schema = build_null_checker(return_type)
+ self._check_output_row_against_schema =
build_null_checker(udfs[0].return_type)
eval_func_kwargs_support, self._args_kwargs_offsets =
wrap_kwargs_support(
- udtf.eval, udfs[0].args_offsets, udfs[0].kwargs_offsets
+ self._udtf.eval, udfs[0].args_offsets, udfs[0].kwargs_offsets
)
- self._eval = self._wrap_udtf(eval_func_kwargs_support, return_type)
+ self._eval = self._wrap_udtf(eval_func_kwargs_support,
udfs[0].return_type)
self._terminate: Optional[Callable[..., tuple]]
- if hasattr(udtf, "terminate"):
- self._terminate = self._wrap_udtf(getattr(udtf, "terminate"),
return_type)
+ if hasattr(self._udtf, "terminate"):
+ self._terminate = self._wrap_udtf(getattr(self._udtf,
"terminate"), udfs[0].return_type)
else:
self._terminate = None
- self._cleanup = getattr(udtf, "cleanup") if hasattr(udtf, "cleanup")
else None
+ self._cleanup = getattr(self._udtf, "cleanup") if hasattr(self._udtf,
"cleanup") else None
@property
def serializer(self) -> Serializer:
@@ -778,9 +855,11 @@ class ArrowUDTFHandler(
super().__init__(udfs, runner_conf, eval_conf)
arrow_return_type = to_arrow_type(
- self._return_type, timezone="UTC",
prefers_large_types=runner_conf.use_large_var_types
+ self._udfs[0].return_type,
+ timezone="UTC",
+ prefers_large_types=runner_conf.use_large_var_types,
)
- self._return_type_size = len(self._return_type)
+ self._return_type_size = len(self._udfs[0].return_type)
self._target_schema = pa.schema(list(arrow_return_type))
self._table_arg_offsets = (
@@ -885,8 +964,8 @@ class ArrowTableUDTFHandler(
def __init__(self, udfs: list[UDTF], runner_conf: RunnerConf, eval_conf:
EvalConf) -> None:
super().__init__(udfs, runner_conf, eval_conf)
- self._return_type_size = len(self._return_type)
- self._check_output_row_against_schema =
build_null_checker(self._return_type)
+ self._return_type_size = len(self._udfs[0].return_type)
+ self._check_output_row_against_schema =
build_null_checker(self._udfs[0].return_type)
@abstractmethod
def _to_columns(self, batch: pa.RecordBatch) -> list:
@@ -941,7 +1020,7 @@ class
ArrowTableUDTFLegacyPandasHandler(ArrowTableUDTFHandler):
super().__init__(udfs, runner_conf, eval_conf)
# The output pandas DataFrame is converted as a single struct column
named
# "_0" against this schema.
- self._output_schema = StructType([StructField("_0",
self._return_type)])
+ self._output_schema = StructType([StructField("_0",
self._udfs[0].return_type)])
def _verify_result(self, result: Any, method_name: str) -> Any:
import pandas as pd
@@ -973,7 +1052,10 @@ class
ArrowTableUDTFLegacyPandasHandler(ArrowTableUDTFHandler):
# Verify the type and the schema of the result.
verify_pandas_result(
- result, self._return_type, assign_cols_by_name=False,
truncate_return_schema=False
+ result,
+ self._udfs[0].return_type,
+ assign_cols_by_name=False,
+ truncate_return_schema=False,
)
return result
@@ -1050,7 +1132,9 @@ class ArrowTableUDTFArrowHandler(ArrowTableUDTFHandler):
super().__init__(udfs, runner_conf, eval_conf)
self._arrow_return_type = to_arrow_type(
- self._return_type, timezone="UTC",
prefers_large_types=runner_conf.use_large_var_types
+ self._udfs[0].return_type,
+ timezone="UTC",
+ prefers_large_types=runner_conf.use_large_var_types,
)
self._converters = [
ArrowTableToRowsConversion._create_converter(
@@ -1133,14 +1217,14 @@ class ArrowTableUDTFArrowHandler(ArrowTableUDTFHandler):
errorClass="UDTF_ARROW_DATA_CONVERSION_ERROR",
messageParameters={
"data": str(data),
- "schema": self._return_type.simpleString(),
+ "schema": self._udfs[0].return_type.simpleString(),
"arrow_schema": str(arrow_return_type),
},
) from original_exception
try:
table = LocalDataToArrowConversion.convert(
- data, self._return_type, self._runner_conf.use_large_var_types
+ data, self._udfs[0].return_type,
self._runner_conf.use_large_var_types
)
except PySparkValueError as e:
if e.getErrorClass() == "AXIS_LENGTH_MISMATCH":
diff --git a/python/pyspark/eval_handlers/_vanilla.py
b/python/pyspark/eval_handlers/_vanilla.py
index 2ad0e4767fe5..b20da0f2c839 100644
--- a/python/pyspark/eval_handlers/_vanilla.py
+++ b/python/pyspark/eval_handlers/_vanilla.py
@@ -45,15 +45,13 @@ class BatchedUDFHandler(EvalTypeHandler[Any, Any]):
def __init__(self, udfs: "list[UDF]", runner_conf: "RunnerConf",
eval_conf: "EvalConf") -> None:
super().__init__(udfs, runner_conf, eval_conf)
- self._udf_infos = [self._prepare_udf(udf) for udf in udfs]
+ self._udf_infos = [self._prepare_udf(udf, func) for udf, func in
zip(udfs, self._funcs)]
@staticmethod
- def _prepare_udf(udf: "UDF") -> tuple[list[int], Callable[..., Any]]:
+ def _prepare_udf(udf: "UDF", func: Callable[..., Any]) -> tuple[list[int],
Callable[..., Any]]:
"""Apply kwargs binding, and convert each result to the internal
representation only
when the return type requires it."""
- func, args_kwargs_offsets = wrap_kwargs_support(
- udf.func, udf.args_offsets, udf.kwargs_offsets
- )
+ func, args_kwargs_offsets = wrap_kwargs_support(func,
udf.args_offsets, udf.kwargs_offsets)
if udf.return_type.needConversion():
toInternal = udf.return_type.toInternal
return args_kwargs_offsets, lambda *a: toInternal(func(*a))
diff --git
a/python/pyspark/eval_handlers/tests/test_arrow_eval_type_handlers.py
b/python/pyspark/eval_handlers/tests/test_arrow_eval_type_handlers.py
index 63ce1c546314..1c29e79b98d4 100644
--- a/python/pyspark/eval_handlers/tests/test_arrow_eval_type_handlers.py
+++ b/python/pyspark/eval_handlers/tests/test_arrow_eval_type_handlers.py
@@ -34,7 +34,7 @@ from pyspark.eval_handlers.utils import hashable_grouping_key
from pyspark.sql.pandas.serializers import ArrowStreamCoGroupSerializer,
ArrowStreamSerializer
from pyspark.sql.types import DoubleType, LongType, StringType, StructField,
StructType
from pyspark.testing.utils import PySparkBaseTestCase, have_pyarrow,
pyarrow_requirement_message
-from pyspark.util import PythonEvalType, fail_on_stopiteration
+from pyspark.util import PythonEvalType
from pyspark.worker_metrics import WorkerMetrics
with patch.dict(os.environ, {"SPARK_PYTHON_RUNTIME": "PYTHON_WORKER"}):
@@ -125,27 +125,23 @@ class ArrowEvalTypeHandlerTestCase(PySparkBaseTestCase):
return offsets
@staticmethod
- def _udf(func, args_offsets, kwargs_offsets=None, return_type=None,
raw_func=None):
- """A ``UDF`` record; ``raw_func`` defaults to ``func``."""
+ def _udf(func, args_offsets, kwargs_offsets=None, return_type=None):
+ """A ``UDF`` record as the worker builds it."""
return UDF(
- func=func,
- raw_func=func if raw_func is None else raw_func,
+ raw_func=func,
args_offsets=args_offsets,
kwargs_offsets=kwargs_offsets or {},
return_type=return_type or LongType(),
+ result_id=0,
)
@classmethod
def _grouped_handler(cls, handler_cls, udf, arg_offsets):
- """Build a grouped/cogrouped-map handler with the shared return type.
As from
- ``read_single_udf``, ``func`` is wrapped (losing the signature) while
``raw_func``
- keeps it."""
+ """Build a grouped/cogrouped-map handler with the shared return type.
The function the
+ handler wraps loses its signature, so the handler reads it from the
record's
+ ``raw_func``."""
return handler_cls(
- udfs=[
- cls._udf(
- lambda *a: udf(*a), arg_offsets,
return_type=cls._RETURN_TYPE, raw_func=udf
- )
- ],
+ udfs=[cls._udf(udf, arg_offsets, return_type=cls._RETURN_TYPE)],
runner_conf=RunnerConf({}),
eval_conf=None,
)
@@ -609,10 +605,9 @@ class
ArrowGroupedAggIncrementalPartialUDFHandlerTests(ArrowEvalTypeHandlerTestC
_KEY_SCHEMA = StructType([StructField("k", LongType())])
def _handler(self, cap=10000):
- # As from read_single_udf: func is wrapped, raw_func is the Aggregator
itself.
- agg = _SumAggregator()
+ # The handler takes the Aggregator itself from the record's raw_func.
return ArrowGroupedAggIncrementalPartialUDFHandler(
- udfs=[self._udf(fail_on_stopiteration(agg), [1], raw_func=agg)],
+ udfs=[self._udf(_SumAggregator(), [1])],
runner_conf=RunnerConf({"spark.sql.execution.arrow.maxRecordsPerBatch":
str(cap)}),
eval_conf=EvalConf({"grouping_key_schema":
self._KEY_SCHEMA.json()}),
)
@@ -669,9 +664,8 @@ class
ArrowGroupedAggIncrementalPartialUDFHandlerTests(ArrowEvalTypeHandlerTestC
[pa.array([float("nan"), float("nan")]), pa.array([1, 2],
type=pa.int64())],
["k", "v"],
)
- agg = _SumAggregator()
handler = ArrowGroupedAggIncrementalPartialUDFHandler(
- udfs=[self._udf(fail_on_stopiteration(agg), [1], raw_func=agg)],
+ udfs=[self._udf(_SumAggregator(), [1])],
runner_conf=RunnerConf({}),
eval_conf=EvalConf(
{"grouping_key_schema": StructType([StructField("k",
DoubleType())]).json()}
@@ -692,12 +686,9 @@ class
ArrowGroupedAggIncrementalFinalUDFHandlerTests(ArrowEvalTypeHandlerTestCas
return pa.RecordBatch.from_arrays([pa.array(buffers,
type=self._BUFFER_TYPE)], ["buf"])
def _handler(self):
- # As from read_single_udf: func is wrapped, raw_func is the Aggregator
itself.
- agg = _SumAggregator()
+ # The handler takes the Aggregator itself from the record's raw_func.
return ArrowGroupedAggIncrementalFinalUDFHandler(
- udfs=[self._udf(fail_on_stopiteration(agg), [0], raw_func=agg)],
- runner_conf=RunnerConf({}),
- eval_conf=None,
+ udfs=[self._udf(_SumAggregator(), [0])],
runner_conf=RunnerConf({}), eval_conf=None
)
def test_merges_partial_buffers_per_group(self):
@@ -722,8 +713,7 @@ class
ArrowGroupedAggIncrementalFinalUDFHandlerTests(ArrowEvalTypeHandlerTestCas
class ArrowWindowAggIncrementalUDFHandlerTests(ArrowEvalTypeHandlerTestCase):
def _handler(self, agg, bound_type, args_offsets):
return ArrowWindowAggIncrementalUDFHandler(
- # As from read_single_udf: func is wrapped, raw_func is the
Aggregator itself.
- udfs=[self._udf(fail_on_stopiteration(agg), args_offsets,
raw_func=agg)],
+ udfs=[self._udf(agg, args_offsets)],
runner_conf=RunnerConf({"window_bound_types": bound_type}),
eval_conf=None,
)
diff --git a/python/pyspark/eval_handlers/tests/test_base_eval_type_handlers.py
b/python/pyspark/eval_handlers/tests/test_base_eval_type_handlers.py
index fdd0d76ebaf0..ba611ba866ab 100644
--- a/python/pyspark/eval_handlers/tests/test_base_eval_type_handlers.py
+++ b/python/pyspark/eval_handlers/tests/test_base_eval_type_handlers.py
@@ -22,7 +22,6 @@ Handler-flavor tests live alongside their module, e.g. the
Arrow handlers in
"""
import os
-from types import SimpleNamespace
from unittest.mock import patch
from pyspark.errors import PySparkRuntimeError
@@ -41,13 +40,12 @@ from pyspark.sql.pandas.serializers import (
ArrowStreamGroupSerializer,
ArrowStreamSerializer,
)
-from pyspark.sql.types import LongType
+from pyspark.sql.types import LongType, StringType
from pyspark.testing.utils import PySparkBaseTestCase
from pyspark.util import PythonEvalType
with patch.dict(os.environ, {"SPARK_PYTHON_RUNTIME": "PYTHON_WORKER"}):
- from pyspark.worker import read_single_udf
- from pyspark.worker_message import UDFInfo
+ from pyspark.worker import unpickle_and_chain
from pyspark.worker_util import RunnerConf, pickleSer
@@ -272,31 +270,110 @@ class EvalTypeHandlerTests(PySparkBaseTestCase):
self.assertNotIn(unused_eval_type, _eval_type_handlers)
-class ReadSingleUDFTests(PySparkBaseTestCase):
- def test_returns_udf_record(self):
- def stop(x):
- raise StopIteration(x)
-
- udf_info = UDFInfo(
- udfs=[CloudPickleSerializer().dumps((stop, LongType()))],
- args=[1],
- kwargs={"y": 2},
- result_id=0,
+class HandlerUDFsTests(PySparkBaseTestCase):
+ class _Handler(BatchEvalTypeHandler["pa.RecordBatch"]):
+ def run(self, split_index, data):
+ return data
+
+ @staticmethod
+ def _stop(x):
+ raise StopIteration(x)
+
+ def _new(self, handler_cls, runner_conf=None):
+ return handler_cls(
+ [
+ UDF(
+ raw_func=self._stop,
+ args_offsets=[1],
+ kwargs_offsets={"y": 2},
+ return_type=LongType(),
+ result_id=0,
+ )
+ ],
+ runner_conf or RunnerConf({}),
+ None,
)
- # read_single_udf only reads these two attributes of the handler class.
- handler_cls =
SimpleNamespace(eval_type=PythonEvalType.SQL_BATCHED_UDF, is_iter_based=False)
- udf = read_single_udf(pickleSer, udf_info, handler_cls, RunnerConf({}))
+
+ def test_wraps_udfs(self):
+ handler = self._new(self._Handler)
+ udf = handler._udfs[0]
self.assertIsInstance(udf, UDF)
self.assertEqual(udf.args_offsets, [1])
self.assertEqual(udf.kwargs_offsets, {"y": 2})
self.assertEqual(udf.return_type, LongType())
- # func is wrapped in fail_on_stopiteration; raw_func is the unwrapped
function.
+ # The handler keeps the function wrapped in fail_on_stopiteration; the
record keeps it
+ # unwrapped.
with self.assertRaises(PySparkRuntimeError):
- udf.func(1)
+ handler._funcs[0](1)
with self.assertRaises(StopIteration):
udf.raw_func(1)
+ def test_fail_on_stopiteration_per_handler(self):
+ not_wrapped = {
+ PythonEvalType.SQL_SCALAR_PANDAS_ITER_UDF,
+ PythonEvalType.SQL_SCALAR_PANDAS_ITER_ELEMENTWISE_UDF,
+ PythonEvalType.SQL_ARROW_BATCHED_UDF,
+ }
+ for _, variants in list(_eval_type_handlers.values()):
+ for handler_cls in variants.values():
+ with self.subTest(handler=handler_cls.__name__):
+ self.assertEqual(
+ handler_cls._fail_on_stopiteration,
+ handler_cls.eval_type not in not_wrapped,
+ )
+
+ class _NotWrapped(self._Handler):
+ _fail_on_stopiteration = False
+
+ with self.assertRaises(StopIteration):
+ self._new(_NotWrapped)._funcs[0](1)
+
+ def test_profiler_selected_by_conf(self):
+ def f(x):
+ return x
+
+ def data_source_f(x):
+ return x
+
+ data_source_f.__module__ = "pyspark.sql.worker.plan_data_source_read"
+
+ udf_perf = self._Handler([],
RunnerConf({"spark.sql.pyspark.udf.profiler": "perf"}), None)
+ data_source_perf = self._Handler(
+ [], RunnerConf({"spark.sql.pyspark.dataSource.profiler": "perf"}),
None
+ )
+ udf_memory = self._Handler(
+ [], RunnerConf({"spark.sql.pyspark.udf.profiler": "memory"}), None
+ )
+ no_profiler = self._Handler([], RunnerConf({}), None)
+ with patch.object(
+ EvalTypeHandler, "_with_perf_profiler", side_effect=lambda func,
result_id: "perf"
+ ) as perf:
+ self.assertEqual(udf_perf._with_profiler(f, 0), "perf")
+ self.assertIs(data_source_perf._with_profiler(f, 0), f)
+ self.assertIs(udf_perf._with_profiler(data_source_f, 0),
data_source_f)
+ self.assertEqual(data_source_perf._with_profiler(data_source_f,
0), "perf")
+ self.assertEqual(perf.call_count, 2)
+ with patch.object(
+ EvalTypeHandler, "_with_memory_profiler", side_effect=lambda func,
result_id: "memory"
+ ):
+ self.assertEqual(udf_memory._with_profiler(f, 0), "memory")
+ self.assertIs(no_profiler._with_profiler(f, 0), f)
+
+
+class UnpickleAndChainTests(PySparkBaseTestCase):
+ def test_chains_in_order(self):
+ ser = CloudPickleSerializer()
+ func, return_type = unpickle_and_chain(
+ pickleSer,
+ [
+ ser.dumps((lambda x: x + 1, LongType())),
+ ser.dumps((lambda x: str(x * 2), StringType())),
+ ],
+ )
+ self.assertEqual(func(3), "8")
+ self.assertEqual(return_type, StringType())
+
if __name__ == "__main__":
from pyspark.testing import main
diff --git
a/python/pyspark/eval_handlers/tests/test_elementwise_eval_type_handlers.py
b/python/pyspark/eval_handlers/tests/test_elementwise_eval_type_handlers.py
index a03a4726a878..9dbf346607bf 100644
--- a/python/pyspark/eval_handlers/tests/test_elementwise_eval_type_handlers.py
+++ b/python/pyspark/eval_handlers/tests/test_elementwise_eval_type_handlers.py
@@ -102,11 +102,11 @@ class
ElementwiseEvalTypeHandlerTestCase(PySparkBaseTestCase):
return handler_cls(
udfs=[
UDF(
- func=func,
raw_func=func,
args_offsets=args_offsets,
kwargs_offsets=kwargs_offsets,
return_type=return_type,
+ result_id=0,
)
for func, args_offsets, kwargs_offsets, return_type in udfs
],
diff --git
a/python/pyspark/eval_handlers/tests/test_pandas_eval_type_handlers.py
b/python/pyspark/eval_handlers/tests/test_pandas_eval_type_handlers.py
index 9b22dcae9d82..2f419d87b0f1 100644
--- a/python/pyspark/eval_handlers/tests/test_pandas_eval_type_handlers.py
+++ b/python/pyspark/eval_handlers/tests/test_pandas_eval_type_handlers.py
@@ -92,16 +92,16 @@ class PandasEvalTypeHandlerTestCase(PySparkBaseTestCase):
@classmethod
def _grouped_handler(cls, handler_cls, udf, arg_offsets, runner_conf=None):
- """Build a grouped/cogrouped-map handler. As from ``read_single_udf``,
``func`` is
- wrapped (losing the signature) while ``raw_func`` keeps it."""
+ """Build a grouped/cogrouped-map handler. The function the handler
wraps loses its
+ signature, so the handler reads it from the record's ``raw_func``."""
return handler_cls(
udfs=[
UDF(
- func=lambda *a: udf(*a),
raw_func=udf,
args_offsets=arg_offsets,
kwargs_offsets={},
return_type=cls._GROUPED_RETURN_TYPE,
+ result_id=0,
)
],
runner_conf=runner_conf or RunnerConf({}),
@@ -115,13 +115,13 @@ class PandasEvalTypeHandlerTestCase(PySparkBaseTestCase):
@staticmethod
def _udf(func, return_type=None, args=(0,), kwargs=None):
- """A ``UDF`` record for a pandas UDF."""
+ """A ``UDF`` record for a pandas UDF, as the worker builds it."""
return UDF(
- func=func,
raw_func=func,
args_offsets=list(args),
kwargs_offsets=dict(kwargs or {}),
return_type=return_type or LongType(),
+ result_id=0,
)
@staticmethod
diff --git
a/python/pyspark/eval_handlers/tests/test_stateful_eval_type_handlers.py
b/python/pyspark/eval_handlers/tests/test_stateful_eval_type_handlers.py
index 6f8646926f01..678561e64d77 100644
--- a/python/pyspark/eval_handlers/tests/test_stateful_eval_type_handlers.py
+++ b/python/pyspark/eval_handlers/tests/test_stateful_eval_type_handlers.py
@@ -119,11 +119,11 @@ class
TransformWithStateHandlerTestCase(PySparkBaseTestCase):
return handler_cls(
udfs=[
UDF(
- func=udf,
raw_func=udf,
args_offsets=arg_offsets,
kwargs_offsets={},
return_type=cls._RETURN_TYPE,
+ result_id=0,
)
],
runner_conf=runner_conf or RunnerConf({}),
@@ -237,11 +237,11 @@ class
PandasGroupedMapWithStateUDFHandlerTests(PySparkBaseTestCase):
return PandasGroupedMapWithStateUDFHandler(
udfs=[
UDF(
- func=udf,
raw_func=udf,
args_offsets=cls._ARG_OFFSETS,
kwargs_offsets={},
return_type=cls._RETURN_TYPE,
+ result_id=0,
)
],
runner_conf=runner_conf or RunnerConf({}),
diff --git a/python/pyspark/eval_handlers/tests/test_udtf_eval_type_handlers.py
b/python/pyspark/eval_handlers/tests/test_udtf_eval_type_handlers.py
index 038a5a2cba04..7d9760921418 100644
--- a/python/pyspark/eval_handlers/tests/test_udtf_eval_type_handlers.py
+++ b/python/pyspark/eval_handlers/tests/test_udtf_eval_type_handlers.py
@@ -17,9 +17,8 @@
"""Tests for the UDTF eval type handlers (``_udtf``).
-The handler gets the ``UDTF`` record ``read_single_udtf`` returns; each test
builds that
-record around a UDTF instance, calls ``run(0, <input batches>)`` and asserts
on the output
-batches.
+Each test builds the handler from the ``UDTF`` record of a UDTF class, calls
+``run(0, <input batches>)`` and asserts on the output batches.
"""
import os
@@ -34,6 +33,7 @@ from pyspark.serializers import BatchedSerializer,
CPickleSerializer
from pyspark.sql.functions import SkipRestOfInputTableException
from pyspark.sql.pandas.serializers import ArrowStreamSerializer
from pyspark.sql.types import LongType, StructField, StructType
+from pyspark.sql.udtf import AnalyzeResult
from pyspark.testing.utils import (
PySparkBaseTestCase,
have_pandas,
@@ -62,9 +62,11 @@ if have_pyarrow:
class _RecordingUDTF:
"""Arrow UDTF that echoes its TABLE argument and records the lifecycle
calls."""
- def __init__(self, skip_after=None):
+ # Raise SkipRestOfInputTableException on the eval call after this many;
None never skips.
+ _skip_after = None
+
+ def __init__(self):
self.calls = []
- self._skip_after = skip_after
def eval(self, table):
self.calls.append("eval")
@@ -83,9 +85,11 @@ class _RecordingUDTF:
class _RowUDTF:
"""Row-wise UDTF that emits its argument and its double, and records the
lifecycle calls."""
- def __init__(self, skip_after=None):
+ # Raise SkipRestOfInputTableException on the eval call after this many;
None never skips.
+ _skip_after = None
+
+ def __init__(self):
self.calls = []
- self._skip_after = skip_after
def eval(self, a):
self.calls.append("eval")
@@ -102,10 +106,30 @@ class _RowUDTF:
class UDTFEvalTypeHandlerTestCase(PySparkBaseTestCase):
- """Shared return type for the UDTF handler tests."""
+ """Shared return type and helpers for the UDTF handler tests."""
_RETURN_TYPE = StructType([StructField("x", LongType())])
+ @staticmethod
+ def _udtf_record(
+ udtf_cls,
+ return_type,
+ args=(0,),
+ kwargs=None,
+ partition_child_indexes=(),
+ analyze_result=None,
+ ):
+ """The ``UDTF`` record the worker builds for ``udtf_cls``."""
+ return UDTF(
+ udtf_cls=udtf_cls,
+ args_offsets=list(args),
+ kwargs_offsets=kwargs or {},
+ return_type=return_type,
+ partition_child_indexes=list(partition_child_indexes),
+ analyze_result=analyze_result,
+ name="test_udtf",
+ )
+
@unittest.skipIf(not have_pyarrow, pyarrow_requirement_message)
class UDTFEvalTypeHandlerRegistrationTests(PySparkBaseTestCase):
@@ -136,19 +160,9 @@ class
UDTFEvalTypeHandlerRegistrationTests(PySparkBaseTestCase):
class TableUDTFHandlerTests(UDTFEvalTypeHandlerTestCase):
_ROW_RETURN_TYPE = StructType([StructField("x", LongType()),
StructField("y", LongType())])
- def _handler(self, udtf, args=(0,), kwargs=None, return_type=None):
- return TableUDTFHandler(
- udfs=[
- UDTF(
- udtf=udtf,
- args_offsets=list(args),
- kwargs_offsets=kwargs or {},
- return_type=return_type or self._ROW_RETURN_TYPE,
- )
- ],
- runner_conf=RunnerConf({}),
- eval_conf=EvalConf({}),
- )
+ def _handler(self, udtf_cls, args=(0,), kwargs=None, return_type=None):
+ udtf = self._udtf_record(udtf_cls, return_type or
self._ROW_RETURN_TYPE, args, kwargs)
+ return TableUDTFHandler(udfs=[udtf], runner_conf=RunnerConf({}),
eval_conf=EvalConf({}))
def _run(self, handler, *rows):
# Each input row is a list of argument values; each output is the
tuple of rows one
@@ -156,45 +170,48 @@ class TableUDTFHandlerTests(UDTFEvalTypeHandlerTestCase):
return list(handler.run(0, iter([list(r) for r in rows])))
def test_serializer_and_partition_wrapper(self):
- ser = self._handler(_RowUDTF()).serializer
+ ser = self._handler(_RowUDTF).serializer
self.assertIsInstance(ser, BatchedSerializer)
self.assertIsInstance(ser.serializer, CPickleSerializer)
self.assertEqual(ser.batchSize, 1)
self.assertIs(TableUDTFHandler.partition_wrapper, UDTFWithPartitions)
def test_eval_terminate_cleanup(self):
- udtf = _RowUDTF()
- out = self._run(self._handler(udtf), [1], [2])
+ handler = self._handler(_RowUDTF)
+ out = self._run(handler, [1], [2])
self.assertEqual(out, [((1, 2),), ((2, 4),), ((-1, -2),)])
- self.assertEqual(udtf.calls, ["eval", "eval", "terminate", "cleanup"])
+ self.assertEqual(handler._udtf.calls, ["eval", "eval", "terminate",
"cleanup"])
def test_no_terminate_or_cleanup(self):
class EvalOnly:
def eval(self, a):
yield a, a
- self.assertEqual(self._run(self._handler(EvalOnly()), [1]), [((1,
1),)])
+ self.assertEqual(self._run(self._handler(EvalOnly), [1]), [((1, 1),)])
def test_kwargs(self):
class KwargsUDTF:
def eval(self, *, a):
yield a, -a
- out = self._run(self._handler(KwargsUDTF(), args=(), kwargs={"a": 0}),
[7])
+ out = self._run(self._handler(KwargsUDTF, args=(), kwargs={"a": 0}),
[7])
self.assertEqual(out, [((7, -7),)])
def test_skip_rest_of_input_table(self):
- udtf = _RowUDTF(skip_after=1)
- out = self._run(self._handler(udtf), [1], [2], [3])
+ class SkipUDTF(_RowUDTF):
+ _skip_after = 1
+
+ handler = self._handler(SkipUDTF)
+ out = self._run(handler, [1], [2], [3])
self.assertEqual(out, [((1, 2),), ((-1, -2),)])
- self.assertEqual(udtf.calls, ["eval", "eval", "terminate", "cleanup"])
+ self.assertEqual(handler._udtf.calls, ["eval", "eval", "terminate",
"cleanup"])
def test_none_result(self):
class NoneUDTF:
def eval(self, a):
return None
- self.assertEqual(self._run(self._handler(NoneUDTF()), [1]), [()])
+ self.assertEqual(self._run(self._handler(NoneUDTF), [1]), [()])
def test_non_nullable_column(self):
class NullUDTF:
@@ -205,7 +222,7 @@ class TableUDTFHandlerTests(UDTFEvalTypeHandlerTestCase):
[StructField("x", LongType(), nullable=False), StructField("y",
LongType())]
)
with self.assertRaises(PySparkRuntimeError) as ctx:
- self._run(self._handler(NullUDTF(), return_type=return_type), [1])
+ self._run(self._handler(NullUDTF, return_type=return_type), [1])
self.assertEqual(ctx.exception.getCondition(), "UDTF_EXEC_ERROR")
def test_errors(self):
@@ -225,44 +242,39 @@ class TableUDTFHandlerTests(UDTFEvalTypeHandlerTestCase):
def eval(self, a):
yield a
- for udtf, error_class in [
- (Raises(), "UDTF_EXEC_ERROR"),
- (NotIterable(), "UDTF_RETURN_NOT_ITERABLE"),
- (WrongColumns(), "UDTF_RETURN_SCHEMA_MISMATCH"),
- (NotARow(), "UDTF_INVALID_OUTPUT_ROW_TYPE"),
+ for udtf_cls, error_class in [
+ (Raises, "UDTF_EXEC_ERROR"),
+ (NotIterable, "UDTF_RETURN_NOT_ITERABLE"),
+ (WrongColumns, "UDTF_RETURN_SCHEMA_MISMATCH"),
+ (NotARow, "UDTF_INVALID_OUTPUT_ROW_TYPE"),
]:
with self.subTest(error_class=error_class):
with self.assertRaises(PySparkRuntimeError) as ctx:
- self._run(self._handler(udtf), [1])
+ self._run(self._handler(udtf_cls), [1])
self.assertEqual(ctx.exception.getCondition(), error_class)
def test_cleanup_on_error(self):
- udtf = _RowUDTF()
- udtf.eval = lambda a: 1 # not iterable
+ class NotIterable(_RowUDTF):
+ def eval(self, a):
+ return 1
+
+ handler = self._handler(NotIterable)
with self.assertRaises(PySparkRuntimeError):
- self._run(self._handler(udtf), [1])
- self.assertEqual(udtf.calls, ["cleanup"])
+ self._run(handler, [1])
+ self.assertEqual(handler._udtf.calls, ["cleanup"])
@unittest.skipIf(not have_pyarrow, pyarrow_requirement_message)
class ArrowUDTFHandlerTests(UDTFEvalTypeHandlerTestCase):
- def _handler(self, udtf, args=(0,), kwargs=None, table_arg_offsets="0"):
+ def _handler(
+ self, udtf_cls, args=(0,), kwargs=None, table_arg_offsets="0",
partition_child_indexes=()
+ ):
conf = {} if table_arg_offsets is None else {"table_arg_offsets":
table_arg_offsets}
- return ArrowUDTFHandler(
- udfs=[
- UDTF(
- udtf=udtf,
- args_offsets=list(args),
- kwargs_offsets=kwargs or {},
- return_type=self._RETURN_TYPE,
- )
- ],
- runner_conf=RunnerConf({}),
- eval_conf=EvalConf(conf),
- )
+ udtf = self._udtf_record(udtf_cls, self._RETURN_TYPE, args, kwargs,
partition_child_indexes)
+ return ArrowUDTFHandler(udfs=[udtf], runner_conf=RunnerConf({}),
eval_conf=EvalConf(conf))
def test_serializer(self):
- self.assertIsInstance(self._handler(_RecordingUDTF()).serializer,
ArrowStreamSerializer)
+ self.assertIsInstance(self._handler(_RecordingUDTF).serializer,
ArrowStreamSerializer)
def test_partition_wrapper(self):
self.assertIs(UDTFEvalTypeHandler.partition_wrapper,
UDTFWithPartitions)
@@ -272,16 +284,20 @@ class ArrowUDTFHandlerTests(UDTFEvalTypeHandlerTestCase):
# Column 1 is the projected PARTITION BY key; the UDTF only sees
column 0.
created = []
- def create_udtf():
- created.append(_RecordingUDTF(skip_after=1))
- return created[-1]
+ class Udtf(_RecordingUDTF):
+ _skip_after = 1
+
+ def __init__(self):
+ super().__init__()
+ created.append(self)
- udtf = ArrowUDTFWithPartition(create_udtf, [1])
batches = [
ArrowBatchTransformer.wrap_struct(pa.record_batch({"x": [1, 2, 3],
"k": [0, 0, 1]})),
ArrowBatchTransformer.wrap_struct(pa.record_batch({"x": [4, 5],
"k": [1, 2]})),
]
- out = list(self._handler(udtf).run(0, iter(batches)))
+ handler = self._handler(Udtf, partition_child_indexes=[1])
+ self.assertIsInstance(handler._udtf, ArrowUDTFWithPartition)
+ out = list(handler.run(0, iter(batches)))
# Partition k=1 spans both batches; its second eval call is skipped.
self.assertEqual(
[r["x"] for b in out for r in b.column(0).to_pylist()], [1, 2, -1,
3, -1, 5, -1]
@@ -296,9 +312,9 @@ class ArrowUDTFHandlerTests(UDTFEvalTypeHandlerTestCase):
)
def test_eval_terminate_cleanup(self):
- udtf = _RecordingUDTF()
+ handler = self._handler(_RecordingUDTF)
out = list(
- self._handler(udtf).run(
+ handler.run(
0,
iter(
[
@@ -309,7 +325,7 @@ class ArrowUDTFHandlerTests(UDTFEvalTypeHandlerTestCase):
)
)
self.assertEqual([r["x"] for b in out for r in
b.column(0).to_pylist()], [1, 2, 3, -1])
- self.assertEqual(udtf.calls, ["eval", "eval", "terminate", "cleanup"])
+ self.assertEqual(handler._udtf.calls, ["eval", "eval", "terminate",
"cleanup"])
def test_kwargs(self):
class KwargsUDTF:
@@ -317,7 +333,7 @@ class ArrowUDTFHandlerTests(UDTFEvalTypeHandlerTestCase):
yield pa.table({"x": table.column(0)})
out = list(
- self._handler(KwargsUDTF(), args=(), kwargs={"table": 0}).run(
+ self._handler(KwargsUDTF, args=(), kwargs={"table": 0}).run(
0,
iter([ArrowBatchTransformer.wrap_struct(pa.record_batch({"x": [7]}))])
)
)
@@ -330,11 +346,14 @@ class ArrowUDTFHandlerTests(UDTFEvalTypeHandlerTestCase):
yield pa.table({"x": pa.compute.multiply(arr, 10)})
batch = pa.RecordBatch.from_arrays([pa.array([1, 2],
type=pa.int64())], ["a"])
- out = list(self._handler(ScalarUDTF(), table_arg_offsets=None).run(0,
iter([batch])))
+ out = list(self._handler(ScalarUDTF, table_arg_offsets=None).run(0,
iter([batch])))
self.assertEqual([r["x"] for b in out for r in
b.column(0).to_pylist()], [10, 20])
def test_skip_rest_of_input_table(self):
- udtf = _RecordingUDTF(skip_after=1)
+ class SkipUDTF(_RecordingUDTF):
+ _skip_after = 1
+
+ handler = self._handler(SkipUDTF)
batches = iter(
[
ArrowBatchTransformer.wrap_struct(pa.record_batch({"x": [1]})),
@@ -342,9 +361,9 @@ class ArrowUDTFHandlerTests(UDTFEvalTypeHandlerTestCase):
ArrowBatchTransformer.wrap_struct(pa.record_batch({"x": [3]})),
]
)
- out = list(self._handler(udtf).run(0, batches))
+ out = list(handler.run(0, batches))
self.assertEqual([r["x"] for b in out for r in
b.column(0).to_pylist()], [1, -1])
- self.assertEqual(udtf.calls, ["eval", "eval", "terminate", "cleanup"])
+ self.assertEqual(handler._udtf.calls, ["eval", "eval", "terminate",
"cleanup"])
# The remaining input is left unconsumed.
self.assertEqual(len(list(batches)), 1)
@@ -354,7 +373,7 @@ class ArrowUDTFHandlerTests(UDTFEvalTypeHandlerTestCase):
return None
out = list(
- self._handler(NoneUDTF()).run(
+ self._handler(NoneUDTF).run(
0,
iter([ArrowBatchTransformer.wrap_struct(pa.record_batch({"x": [1]}))])
)
)
@@ -378,16 +397,16 @@ class ArrowUDTFHandlerTests(UDTFEvalTypeHandlerTestCase):
def eval(self, table):
yield pa.table({"x": [1], "y": [2]})
- for udtf, error_class in [
- (Raises(), "UDTF_EXEC_ERROR"),
- (NotIterable(), "UDTF_RETURN_NOT_ITERABLE"),
- (NotArrow(), "UDTF_ARROW_TYPE_CONVERSION_ERROR"),
- (WrongColumns(), "UDTF_RETURN_SCHEMA_MISMATCH"),
+ for udtf_cls, error_class in [
+ (Raises, "UDTF_EXEC_ERROR"),
+ (NotIterable, "UDTF_RETURN_NOT_ITERABLE"),
+ (NotArrow, "UDTF_ARROW_TYPE_CONVERSION_ERROR"),
+ (WrongColumns, "UDTF_RETURN_SCHEMA_MISMATCH"),
]:
with self.subTest(error_class=error_class):
with self.assertRaises(PySparkRuntimeError) as ctx:
list(
- self._handler(udtf).run(
+ self._handler(udtf_cls).run(
0,
iter([ArrowBatchTransformer.wrap_struct(pa.record_batch({"x": [1]}))]),
)
@@ -395,15 +414,18 @@ class ArrowUDTFHandlerTests(UDTFEvalTypeHandlerTestCase):
self.assertEqual(ctx.exception.getCondition(), error_class)
def test_cleanup_on_error(self):
- udtf = _RecordingUDTF()
- udtf.eval = lambda table: 1 # not iterable
+ class NotIterable(_RecordingUDTF):
+ def eval(self, table):
+ return 1
+
+ handler = self._handler(NotIterable)
with self.assertRaises(PySparkRuntimeError):
list(
- self._handler(udtf).run(
+ handler.run(
0,
iter([ArrowBatchTransformer.wrap_struct(pa.record_batch({"x": [1]}))])
)
)
- self.assertEqual(udtf.calls, ["cleanup"])
+ self.assertEqual(handler._udtf.calls, ["cleanup"])
class ArrowTableUDTFHandlerTestsMixin:
@@ -414,16 +436,10 @@ class ArrowTableUDTFHandlerTestsMixin:
_ROW_RETURN_TYPE = StructType([StructField("x", LongType()),
StructField("y", LongType())])
_ROW_INPUT_TYPE = StructType([StructField("a", LongType())])
- def _handler(self, udtf, args=(0,), kwargs=None, return_type=None):
+ def _handler(self, udtf_cls, args=(0,), kwargs=None, return_type=None):
+ udtf = self._udtf_record(udtf_cls, return_type or
self._ROW_RETURN_TYPE, args, kwargs)
return self.handler_cls(
- udfs=[
- UDTF(
- udtf=udtf,
- args_offsets=list(args),
- kwargs_offsets=kwargs or {},
- return_type=return_type or self._ROW_RETURN_TYPE,
- )
- ],
+ udfs=[udtf],
runner_conf=RunnerConf({}),
eval_conf=EvalConf({"input_type": self._ROW_INPUT_TYPE.json()}),
)
@@ -433,35 +449,38 @@ class ArrowTableUDTFHandlerTestsMixin:
return [r for b in handler.run(0, iter(batches)) for r in
b.column(0).to_pylist()]
def test_serializer_and_partition_wrapper(self):
- self.assertIsInstance(self._handler(_RowUDTF()).serializer,
ArrowStreamSerializer)
+ self.assertIsInstance(self._handler(_RowUDTF).serializer,
ArrowStreamSerializer)
self.assertIs(self.handler_cls.partition_wrapper, UDTFWithPartitions)
def test_eval_terminate_cleanup(self):
- udtf = _RowUDTF()
- out = self._run(self._handler(udtf), [1, 2], [3])
+ handler = self._handler(_RowUDTF)
+ out = self._run(handler, [1, 2], [3])
self.assertEqual([(r["x"], r["y"]) for r in out], [(1, 2), (2, 4), (3,
6), (-1, -2)])
- self.assertEqual(udtf.calls, ["eval", "eval", "eval", "terminate",
"cleanup"])
+ self.assertEqual(handler._udtf.calls, ["eval", "eval", "eval",
"terminate", "cleanup"])
def test_kwargs(self):
class KwargsUDTF:
def eval(self, *, a):
yield a, -a
- out = self._run(self._handler(KwargsUDTF(), args=(), kwargs={"a": 0}),
[7])
+ out = self._run(self._handler(KwargsUDTF, args=(), kwargs={"a": 0}),
[7])
self.assertEqual([(r["x"], r["y"]) for r in out], [(7, -7)])
def test_skip_rest_of_input_table(self):
- udtf = _RowUDTF(skip_after=1)
- out = self._run(self._handler(udtf), [1, 2, 3])
+ class SkipUDTF(_RowUDTF):
+ _skip_after = 1
+
+ handler = self._handler(SkipUDTF)
+ out = self._run(handler, [1, 2, 3])
self.assertEqual([(r["x"], r["y"]) for r in out], [(1, 2), (-1, -2)])
- self.assertEqual(udtf.calls, ["eval", "eval", "terminate", "cleanup"])
+ self.assertEqual(handler._udtf.calls, ["eval", "eval", "terminate",
"cleanup"])
def test_none_result(self):
class NoneUDTF:
def eval(self, a):
return None
- self.assertEqual(self._run(self._handler(NoneUDTF()), [1]), [])
+ self.assertEqual(self._run(self._handler(NoneUDTF), [1]), [])
def test_non_nullable_column(self):
class NullUDTF:
@@ -472,7 +491,7 @@ class ArrowTableUDTFHandlerTestsMixin:
[StructField("x", LongType(), nullable=False), StructField("y",
LongType())]
)
with self.assertRaises(PySparkRuntimeError) as ctx:
- self._run(self._handler(NullUDTF(), return_type=return_type), [1])
+ self._run(self._handler(NullUDTF, return_type=return_type), [1])
self.assertEqual(ctx.exception.getCondition(), "UDTF_EXEC_ERROR")
def test_errors(self):
@@ -488,22 +507,25 @@ class ArrowTableUDTFHandlerTestsMixin:
def eval(self, a):
yield a, a, a
- for udtf, error_class in [
- (Raises(), "UDTF_EXEC_ERROR"),
- (NotIterable(), "UDTF_RETURN_NOT_ITERABLE"),
- (WrongColumns(), "UDTF_RETURN_SCHEMA_MISMATCH"),
+ for udtf_cls, error_class in [
+ (Raises, "UDTF_EXEC_ERROR"),
+ (NotIterable, "UDTF_RETURN_NOT_ITERABLE"),
+ (WrongColumns, "UDTF_RETURN_SCHEMA_MISMATCH"),
]:
with self.subTest(error_class=error_class):
with self.assertRaises(PySparkRuntimeError) as ctx:
- self._run(self._handler(udtf), [1])
+ self._run(self._handler(udtf_cls), [1])
self.assertEqual(ctx.exception.getCondition(), error_class)
def test_cleanup_on_error(self):
- udtf = _RowUDTF()
- udtf.eval = lambda a: 1 # not iterable
+ class NotIterable(_RowUDTF):
+ def eval(self, a):
+ return 1
+
+ handler = self._handler(NotIterable)
with self.assertRaises(PySparkRuntimeError):
- self._run(self._handler(udtf), [1])
- self.assertEqual(udtf.calls, ["cleanup"])
+ self._run(handler, [1])
+ self.assertEqual(handler._udtf.calls, ["cleanup"])
@unittest.skipIf(not have_pyarrow, pyarrow_requirement_message)
@@ -515,7 +537,7 @@ class
ArrowTableUDTFArrowHandlerTests(ArrowTableUDTFHandlerTestsMixin, UDTFEvalT
def eval(self, a):
yield a
- out = self._run(self._handler(ScalarUDTF(),
return_type=self._RETURN_TYPE), [1, 2])
+ out = self._run(self._handler(ScalarUDTF,
return_type=self._RETURN_TYPE), [1, 2])
self.assertEqual([r["x"] for r in out], [1, 2])
@@ -528,6 +550,73 @@ class ArrowTableUDTFLegacyPandasHandlerTests(
handler_cls = ArrowTableUDTFLegacyPandasHandler if have_pyarrow else None
+class UDTFHandlerInitTests(UDTFEvalTypeHandlerTestCase):
+ def _new(self, handler_cls, udtf_cls, **kwargs):
+ """A new ``handler_cls`` for ``udtf_cls``."""
+ udtf = self._udtf_record(udtf_cls, self._RETURN_TYPE, **kwargs)
+ return handler_cls([udtf], RunnerConf({}), EvalConf({}))
+
+ def test_creates_udtf_instance(self):
+ class Udtf:
+ def eval(self, x):
+ yield (x,)
+
+ handler = self._new(TableUDTFHandler, Udtf)
+ self.assertIsInstance(handler._udtf, Udtf)
+ udtf = handler._udfs[0]
+ self.assertIsInstance(udtf, UDTF)
+ self.assertEqual(udtf.args_offsets, [0])
+ self.assertEqual(udtf.kwargs_offsets, {})
+ self.assertEqual(udtf.return_type, self._RETURN_TYPE)
+
+ def test_analyze_result_passed_to_init(self):
+ class Udtf:
+ def __init__(self, analyze_result):
+ self.analyze_result = analyze_result
+
+ def eval(self, x):
+ yield (x,)
+
+ analyze_result = AnalyzeResult(schema=self._RETURN_TYPE)
+ handler = self._new(TableUDTFHandler, Udtf,
analyze_result=analyze_result)
+ self.assertEqual(handler._udtf.analyze_result, analyze_result)
+
+ def test_wrapped_in_partition_wrapper(self):
+ class Udtf:
+ def eval(self, row):
+ yield (row,)
+
+ handler = self._new(TableUDTFHandler, Udtf,
partition_child_indexes=[0])
+ self.assertIsInstance(handler._udtf, UDTFWithPartitions)
+
+ @unittest.skipIf(not have_pyarrow, pyarrow_requirement_message)
+ def test_arrow_partition_wrapper(self):
+ class Udtf:
+ def eval(self, table):
+ yield table
+
+ handler = self._new(ArrowUDTFHandler, Udtf,
partition_child_indexes=[0])
+ self.assertIsInstance(handler._udtf, ArrowUDTFWithPartition)
+
+ def test_invalid_udtf(self):
+ class NoEval:
+ pass
+
+ class Udtf:
+ def eval(self, x):
+ yield (x,)
+
+ with self.assertRaisesRegex(PySparkRuntimeError, "Invalid UDTF handler
type"):
+ self._new(TableUDTFHandler, Udtf())
+ with self.assertRaisesRegex(PySparkRuntimeError, "'eval' method"):
+ self._new(TableUDTFHandler, NoEval)
+ with self.assertRaises(PySparkRuntimeError) as ctx:
+ self._new(TableUDTFHandler, Udtf, args=(0, 1))
+ self.assertEqual(
+ ctx.exception.getCondition(),
"UDTF_EVAL_METHOD_ARGUMENTS_DO_NOT_MATCH_SIGNATURE"
+ )
+
+
if __name__ == "__main__":
from pyspark.testing import main
diff --git
a/python/pyspark/eval_handlers/tests/test_vanilla_eval_type_handlers.py
b/python/pyspark/eval_handlers/tests/test_vanilla_eval_type_handlers.py
index 72a4daa6fa37..fe6aafb75143 100644
--- a/python/pyspark/eval_handlers/tests/test_vanilla_eval_type_handlers.py
+++ b/python/pyspark/eval_handlers/tests/test_vanilla_eval_type_handlers.py
@@ -17,7 +17,7 @@
"""Tests for the plain Python (pickle) UDF eval type handler (``_vanilla``).
-The handler is built from ``UDF`` records, as ``read_single_udf`` returns
them, and fed
+The handler is built from ``UDF`` records and fed
pickled-row input via ``run(0, <rows>)``.
"""
@@ -45,11 +45,11 @@ class BatchedUDFHandlerTests(PySparkBaseTestCase):
return BatchedUDFHandler(
udfs=[
UDF(
- func=f,
raw_func=f,
args_offsets=arg_offsets,
kwargs_offsets=kwargs_offsets or {},
return_type=return_type or LongType(),
+ result_id=0,
)
for arg_offsets, f in udfs
],
diff --git a/python/pyspark/worker.py b/python/pyspark/worker.py
index 6105b65f1aac..563b4d9dfe3d 100644
--- a/python/pyspark/worker.py
+++ b/python/pyspark/worker.py
@@ -23,42 +23,33 @@ pyspark.daemon. A task runs either a classic RDD command
(eval type NON_UDF) or
through the handler registered for their eval type in pyspark.eval_handlers.
"""
-import dataclasses
-import inspect
import json
import os
import sys
import warnings
-from typing import BinaryIO
+from typing import Any, BinaryIO, Callable
from pyspark import shuffle
-from pyspark.accumulators import (
- SpecialAccumulatorIds,
- _accumulatorRegistry,
- _deserialize_accumulator,
-)
+from pyspark.accumulators import _accumulatorRegistry
from pyspark.errors import PySparkRuntimeError
-from pyspark.eval_handlers._base import get_eval_type_handler
+from pyspark.eval_handlers._base import EvalTypeHandler, get_eval_type_handler
from pyspark.eval_handlers._typing import UDF, UDTF
-from pyspark.eval_handlers._udtf import TableUDTFHandler, UDTFEvalTypeHandler
+from pyspark.eval_handlers._udtf import UDTFHandlerBase
from pyspark.logger.worker_io import capture_outputs
from pyspark.messages import (
SparkMessageReceiver,
SparkSocketMessageReceiver,
)
from pyspark.serializers import (
+ FramedSerializer,
SpecialLengths,
write_int,
write_with_length,
)
-from pyspark.sql.types import (
- StructType,
- _parse_datatype_json_string,
-)
+from pyspark.sql.types import DataType, StructType, _parse_datatype_json_string
from pyspark.taskcontext import BarrierTaskContext, TaskContext
from pyspark.util import (
PythonEvalType,
- fail_on_stopiteration,
handle_worker_exception,
start_faulthandler_periodic_traceback,
with_faulthandler,
@@ -86,217 +77,21 @@ def report_metrics(outfile: BinaryIO, metrics: dict[str,
int]) -> None:
write_with_length(payload, outfile)
-def chain(f, g):
+def chain(f: Callable[..., Any], g: Callable[..., Any]) -> Callable[..., Any]:
"""chain two functions together"""
return lambda *a: g(f(*a))
-def wrap_perf_profiler(f, iter_based, result_id):
- from pyspark.sql.profiler import ProfileResultsParam,
ProfileResultsParamV2, WorkerPerfProfiler
-
- accumulator = _deserialize_accumulator(
- SpecialAccumulatorIds.SQL_UDF_PROFIER, None, ProfileResultsParam
- )
- accumulator_v2 = _deserialize_accumulator(
- SpecialAccumulatorIds.SQL_UDF_PROFIER_V2, {}, ProfileResultsParamV2
- )
-
- if iter_based:
-
- def profiling_func(*args, **kwargs):
- iterator = iter(f(*args, **kwargs))
- while True:
- try:
- with WorkerPerfProfiler(accumulator, accumulator_v2,
result_id):
- item = next(iterator)
- yield item
- except StopIteration:
- break
-
- else:
-
- def profiling_func(*args, **kwargs):
- with WorkerPerfProfiler(accumulator, accumulator_v2, result_id):
- ret = f(*args, **kwargs)
- return ret
-
- return profiling_func
-
-
-def wrap_memory_profiler(f, iter_based, result_id):
- import pyspark.memory_profiler_ext
- from pyspark.sql.profiler import (
- ProfileResultsParam,
- ProfileResultsParamV2,
- WorkerMemoryProfiler,
- )
-
- if not pyspark.memory_profiler_ext.has_memory_profiler:
- return f
-
- accumulator = _deserialize_accumulator(
- SpecialAccumulatorIds.SQL_UDF_PROFIER, None, ProfileResultsParam
- )
-
- accumulator_v2 = _deserialize_accumulator(
- SpecialAccumulatorIds.SQL_UDF_PROFIER_V2, {}, ProfileResultsParamV2
- )
-
- if iter_based:
-
- def profiling_func(*args, **kwargs):
- g = f(*args, **kwargs)
- iterator = iter(g)
-
- while True:
- try:
- with WorkerMemoryProfiler(accumulator, accumulator_v2,
result_id, g.gi_code):
- item = next(iterator)
- yield item
- except StopIteration:
- break
-
- else:
-
- def profiling_func(*args, **kwargs):
- with WorkerMemoryProfiler(accumulator, accumulator_v2, result_id,
f):
- ret = f(*args, **kwargs)
- return ret
-
- return profiling_func
-
-
-def read_single_udf(pickleSer, udf_info, handler_cls, runner_conf):
- eval_type = handler_cls.eval_type
- chained_func = None
- for udf in udf_info.udfs:
+def unpickle_and_chain(
+ pickleSer: FramedSerializer, pickled_udfs: list[memoryview]
+) -> tuple[Callable[..., Any], DataType]:
+ """Unpickle the functions applied in turn to one column and chain them
into one; returns the
+ chained function and the return type of the last one."""
+ chained_func, return_type = read_command(pickleSer, pickled_udfs[0])
+ for udf in pickled_udfs[1:]:
f, return_type = read_command(pickleSer, udf)
- if chained_func is None:
- chained_func = f
- else:
- chained_func = chain(chained_func, f)
-
- # If chained_func is from pyspark.sql.worker, it is to read/write data
source.
- # In this case, we check the data_source_profiler config.
- module = getattr(chained_func, "__module__", "")
- if isinstance(module, str) and module.startswith("pyspark.sql.worker."):
- profiler = runner_conf.data_source_profiler
- else:
- profiler = runner_conf.udf_profiler
- if profiler == "perf":
- profiling_func = wrap_perf_profiler(
- chained_func, handler_cls.is_iter_based, udf_info.result_id
- )
- elif profiler == "memory":
- profiling_func = wrap_memory_profiler(
- chained_func, handler_cls.is_iter_based, udf_info.result_id
- )
- else:
- profiling_func = chained_func
-
- if eval_type in (
- PythonEvalType.SQL_SCALAR_PANDAS_ITER_UDF,
- PythonEvalType.SQL_SCALAR_PANDAS_ITER_ELEMENTWISE_UDF,
- PythonEvalType.SQL_ARROW_BATCHED_UDF,
- ):
- func = profiling_func
- else:
- # make sure StopIteration's raised in the user code are not ignored
- # when they are processed in a for loop, raise them as RuntimeError's
instead
- func = fail_on_stopiteration(profiling_func)
-
- # The last returnType will be the return type of UDF.
- return UDF(
- func=func,
- raw_func=chained_func,
- args_offsets=udf_info.args,
- kwargs_offsets=udf_info.kwargs,
- return_type=return_type,
- )
-
-
-# Read a serialized user-defined table function (UDTF) and prepare it for
evaluation: wrap
-# the constructor to receive the AnalyzeResult, instantiate the UDTF (wrapped
in the handler's
-# partition_wrapper when its TABLE argument has PARTITION BY expressions), and
check the 'eval'
-# signature against the call arguments.
-def read_single_udtf(pickleSer, udtf_info, handler_cls):
- if udtf_info.pickled_analyze_result is not None:
- pickled_analyze_result =
pickleSer.loads(udtf_info.pickled_analyze_result)
- else:
- pickled_analyze_result = None
- handler = read_command(pickleSer, udtf_info.handler)
- if not isinstance(handler, type):
- raise PySparkRuntimeError(
- f"Invalid UDTF handler type. Expected a class (type 'type'), but "
- f"got an instance of {type(handler).__name__}."
- )
-
- return_type = _parse_datatype_json_string(udtf_info.return_type)
- if not isinstance(return_type, StructType):
- raise PySparkRuntimeError(
- f"The return type of a UDTF must be a struct type, but got
{type(return_type)}."
- )
-
- # Pick how to construct the UDTF from the count of named positional
__init__ parameters
- # after self. With an AnalyzeResult, one parameter receives a copy of it,
zero means the
- # class is called with no arguments, and more are an error; without one,
any is an error.
- udtf_init_args = inspect.getfullargspec(handler)
- if pickled_analyze_result is not None:
- if len(udtf_init_args.args) > 2:
- raise PySparkRuntimeError(
-
errorClass="UDTF_CONSTRUCTOR_INVALID_IMPLEMENTS_ANALYZE_METHOD",
- messageParameters={"name": udtf_info.name},
- )
- elif len(udtf_init_args.args) == 2:
- prev_handler = handler
-
- def construct_udtf():
- # Here we pass the AnalyzeResult to the UDTF's __init__ method.
- return
prev_handler(dataclasses.replace(pickled_analyze_result))
-
- handler = construct_udtf
- elif len(udtf_init_args.args) > 1:
- raise PySparkRuntimeError(
- errorClass="UDTF_CONSTRUCTOR_INVALID_NO_ANALYZE_METHOD",
- messageParameters={"name": udtf_info.name},
- )
-
- # Instantiate the UDTF class.
- try:
- if len(udtf_info.partition_child_indexes) > 0:
- udtf = handler_cls.partition_wrapper(handler,
udtf_info.partition_child_indexes)
- else:
- udtf = handler()
- except Exception as e:
- raise PySparkRuntimeError(
- errorClass="UDTF_EXEC_ERROR",
- messageParameters={"method_name": "__init__", "error": str(e)},
- )
-
- # Validate the UDTF
- if not hasattr(udtf, "eval"):
- raise PySparkRuntimeError(
- "Failed to execute the user defined table function because it has
not "
- "implemented the 'eval' method. Please add the 'eval' method and
try "
- "the query again."
- )
-
- # Check that the arguments provided to the UDTF call match the expected
parameters defined
- # in the 'eval' method signature.
- try:
- inspect.signature(udtf.eval).bind(*udtf_info.args, **udtf_info.kwargs)
- except TypeError as e:
- raise PySparkRuntimeError(
- errorClass="UDTF_EVAL_METHOD_ARGUMENTS_DO_NOT_MATCH_SIGNATURE",
- messageParameters={"name": udtf_info.name, "reason": str(e)},
- ) from None
-
- return UDTF(
- udtf=udtf,
- args_offsets=udtf_info.args,
- kwargs_offsets=udtf_info.kwargs,
- return_type=return_type,
- )
+ chained_func = chain(chained_func, f)
+ return chained_func, return_type
def invoke_udf(message_receiver: SparkMessageReceiver, outfile: BinaryIO):
@@ -340,7 +135,8 @@ def invoke_udf(message_receiver: SparkMessageReceiver,
outfile: BinaryIO):
runner_conf = RunnerConf(init_info.runner_conf)
eval_conf = EvalConf(init_info.eval_conf)
# Only the classic RDD command carries a profiler of its own. UDF
runners fold profiling
- # into func at construction time (see read_single_udf); UDTF runners
are not profiled.
+ # into func at construction time (see EvalTypeHandler.__init__); UDTF
runners are not
+ # profiled.
profiler = None
if eval_type == PythonEvalType.NON_UDF:
assert isinstance(init_info.udf_info, (bytes, memoryview))
@@ -351,16 +147,45 @@ def invoke_udf(message_receiver: SparkMessageReceiver,
outfile: BinaryIO):
handler_cls = get_eval_type_handler(eval_type, runner_conf)
if handler_cls is None:
raise ValueError("Unknown eval type: {}".format(eval_type))
+ handler: EvalTypeHandler
if isinstance(init_info.udf_info, UDTFInfo):
- assert issubclass(handler_cls, (UDTFEvalTypeHandler,
TableUDTFHandler))
- udfs = [read_single_udtf(pickleSer, init_info.udf_info,
handler_cls)]
+ assert issubclass(handler_cls, UDTFHandlerBase)
+ udtf_cls = read_command(pickleSer, init_info.udf_info.handler)
+ return_type =
_parse_datatype_json_string(init_info.udf_info.return_type)
+ if not isinstance(return_type, StructType):
+ raise PySparkRuntimeError(
+ "The return type of a UDTF must be a struct type, but
got "
+ f"{type(return_type)}."
+ )
+ if init_info.udf_info.pickled_analyze_result is not None:
+ analyze_result =
pickleSer.loads(init_info.udf_info.pickled_analyze_result)
+ else:
+ analyze_result = None
+ udtf = UDTF(
+ udtf_cls=udtf_cls,
+ args_offsets=init_info.udf_info.args,
+ kwargs_offsets=init_info.udf_info.kwargs,
+ return_type=return_type,
+
partition_child_indexes=init_info.udf_info.partition_child_indexes,
+ analyze_result=analyze_result,
+ name=init_info.udf_info.name,
+ )
+ handler = handler_cls(udfs=[udtf], runner_conf=runner_conf,
eval_conf=eval_conf)
else:
assert isinstance(init_info.udf_info, list)
- udfs = [
- read_single_udf(pickleSer, udf_info, handler_cls,
runner_conf)
- for udf_info in init_info.udf_info
- ]
- handler = handler_cls(udfs=udfs, runner_conf=runner_conf,
eval_conf=eval_conf)
+ udfs = []
+ for udf_info in init_info.udf_info:
+ chained_func, return_type = unpickle_and_chain(pickleSer,
udf_info.udfs)
+ udfs.append(
+ UDF(
+ raw_func=chained_func,
+ args_offsets=udf_info.args,
+ kwargs_offsets=udf_info.kwargs,
+ return_type=return_type,
+ result_id=udf_info.result_id,
+ )
+ )
+ handler = handler_cls(udfs=udfs, runner_conf=runner_conf,
eval_conf=eval_conf)
func, serializer = handler.run, handler.serializer
deserializer = serializer
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]