[
https://issues.apache.org/jira/browse/SPARK-59049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haiyang Sun updated SPARK-59049:
--------------------------------
Description:
PySpark can currently chain compatible UDFs and execute them in a single worker
round trip. This requires both:
- Planner logic that groups the UDF expressions into one execution node.
- Worker support for executing multiple UDFs together.
To generalize this behavior for external UDFs, chaining should be modeled as an
explicit engine-worker capability:
1. Not every worker supports UDF chaining. The worker specification should
advertise this capability so the engine can skip the chaining optimization for
unsupported workers.
2. Chaining should have a language-neutral protocol representation. A generic
planning rule can combine UDFs into an expression tree with metadata describing
how inputs are mapped. Workers implementing the protocol can then execute the
combined UDF in one round trip. The representation should support simple linear
chains initially while allowing more complex trees in the future.
3. Language-specific client and payload serialization should remain independent
of chaining. The planner should package the component UDFs as one logical
composite UDF, which downstream planning treats like any other UDF and maps to
one UDF session.
4. External UDF planning should enforce a one-to-one mapping between a Spark
execution node and a UDF session. Each node therefore contains one logical UDF,
which may be either a single UDF or a composite chained UDF.
> Support udf chaining logic for external udfs
> --------------------------------------------
>
> Key: SPARK-59049
> URL: https://issues.apache.org/jira/browse/SPARK-59049
> Project: Spark
> Issue Type: Sub-task
> Components: UDF
> Affects Versions: 4.3.0
> Reporter: Haiyang Sun
> Priority: Major
>
> PySpark can currently chain compatible UDFs and execute them in a single
> worker round trip. This requires both:
> - Planner logic that groups the UDF expressions into one execution node.
> - Worker support for executing multiple UDFs together.
> To generalize this behavior for external UDFs, chaining should be modeled as
> an explicit engine-worker capability:
> 1. Not every worker supports UDF chaining. The worker specification should
> advertise this capability so the engine can skip the chaining optimization
> for unsupported workers.
> 2. Chaining should have a language-neutral protocol representation. A generic
> planning rule can combine UDFs into an expression tree with metadata
> describing how inputs are mapped. Workers implementing the protocol can then
> execute the combined UDF in one round trip. The representation should support
> simple linear chains initially while allowing more complex trees in the
> future.
> 3. Language-specific client and payload serialization should remain
> independent of chaining. The planner should package the component UDFs as one
> logical composite UDF, which downstream planning treats like any other UDF
> and maps to one UDF session.
> 4. External UDF planning should enforce a one-to-one mapping between a Spark
> execution node and a UDF session. Each node therefore contains one logical
> UDF, which may be either a single UDF or a composite chained UDF.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
