Yicong Huang created SPARK-60532:
------------------------------------
Summary: Chain UDF functions in EvalTypeHandler and enforce
single-UDF handlers in the base
Key: SPARK-60532
URL: https://issues.apache.org/jira/browse/SPARK-60532
Project: Spark
Issue Type: Sub-task
Components: PySpark
Affects Versions: 5.0.0
Reporter: Yicong Huang
Move the chaining of nested UDF calls (ChainedPythonFunctions) from worker.py
into EvalTypeHandler.__init__, so the worker only deserializes the functions
and the UDF record carries them unchained. Add a _single_udf class flag on
EvalTypeHandler: handlers that take exactly one UDF (grouped/cogrouped map, map
iter, scalar iter, grouped agg iter, scalar iter elementwise, stateful, UDTF)
declare it, and the base fails fast on any other count and exposes the UDF and
its functions as _udf, _raw_func and _func, replacing the per-handler udfs[0]
asserts and lookups. No behavior change.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
