+1 (non-binding) El mié, 25 feb 2026, 19:15, vaquar khan <[email protected]> escribió:
> +1 (non-binding) > > On Wed, 25 Feb 2026 at 11:33, Daniel Tenedorio <[email protected]> > wrote: > >> +1 (non-binding), this should make Spark's interfaces better and simplify >> the PySpark UDF protocols. Thanks for preparing this! >> >> On 2026/02/25 16:12:08 Herman van Hovell via dev wrote: >> > Hi Spark devs, >> > >> > I would like to call for a vote on the SPIP: Language-Agnostic UDF >> > Execution Protocol for Spark. >> > >> > Summary: >> > >> > The SPIP proposes a structured, language-agnostic execution protocol for >> > running user-defined functions (UDFs) in Spark across multiple >> programming >> > languages. >> > >> > Today, Spark Connect allows users to write queries from multiple >> languages, >> > but support for user-defined functions remains incomplete. In practice, >> > only Scala / Java / Python / R have working support, and it relies on >> > language-specific mechanisms that do not generalize well to other >> languages >> > such as Go <https://github.com/apache/spark-connect-go>, Rust >> > <https://github.com/apache/spark-connect-rust>, Swift >> > <https://github.com/apache/spark-connect-swift>, TypeScript >> > <https://github.com/BaldrVivaldelli/ts-spark-connector> or .NET >> > <https://github.com/GoEddie/spark-connect-dotnet>, where UDF support is >> > currently unavailable. There are also legacy limitations around the >> > existing PySpark worker.py implementation that can be improved with the >> > proposal. >> > >> > This proposal aims to define a unified API and execution protocol for >> UDFs >> > that run outside the Spark executor process and communicate with Spark >> via >> > inter-process communication (IPC). The goal is to enable Spark to >> interact >> > with external workers in a consistent and extensible way, regardless of >> the >> > implementation language. >> > >> > Links: >> > >> > SPIP Doc: >> > >> https://docs.google.com/document/d/19Whzq127QxVt2Luk0EClgaDtcpBsFUp67NcVdKKyPF8/edit?tab=t.0 >> > >> > JIRA: https://issues.apache.org/jira/browse/SPARK-55278 >> > >> > Discussion Thread: >> > https://lists.apache.org/thread/9t4svsnd71j7sb4r4scf2xhh8dvp3b43 >> > >> > Please vote on the SPIP for the next 72 hours: >> > >> > [ ] +1: Accept the proposal as an official SPIP >> > >> > [ ] +0 >> > >> > [ ] -1: I don’t think this is a good idea because… >> > >> > Thanks to everyone who participated in the discussion and provided >> valuable >> > feedback. >> > >> >> --------------------------------------------------------------------- >> To unsubscribe e-mail: [email protected] >> >>
