[
https://issues.apache.org/jira/browse/SPARK-55278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haiyang Sun updated SPARK-55278:
--------------------------------
Summary: [SPIP] Language-agnostic UDF Protocol for Spark (was:
Language-agnostic UDF Protocol for Spark)
> [SPIP] Language-agnostic UDF Protocol for Spark
> -----------------------------------------------
>
> Key: SPARK-55278
> URL: https://issues.apache.org/jira/browse/SPARK-55278
> Project: Spark
> Issue Type: Improvement
> Components: Connect, PySpark
> Affects Versions: 4.2
> Reporter: Haiyang Sun
> Priority: Major
>
> Run user-provided code in Spark {*}consistently across many programming
> languages{*}.
> Today, Spark Connect allows users to write queries from multiple languages,
> but support for user-defined functions is incomplete. In practice, only
> Python has a mature solution, and it relies on language-specific mechanisms
> that do not generalize to other languages such as
> [Go|https://github.com/apache/spark-connect-go] /
> [Rust|https://github.com/apache/spark-connect-rust] /
> [Swift|https://github.com/apache/spark-connect-swift] /
> [.NET|https://github.com/GoEddie/spark-connect-dotnet] (where UDF is not
> supported).
> Our objective is to define a *unified API and execution protocol* for
> user-defined functions that run outside the Spark engine process via
> inter-process communication (IPC). This allows Spark to interact with
> external workers in a consistent way, regardless of the language used to
> implement the function.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
