+1 On Wed, Mar 18, 2026, 4:53 PM Hyukjin Kwon <[email protected]> wrote:
> +1 > > On Thu, 19 Mar 2026 at 06:42, Szehon Ho <[email protected]> wrote: > >> +1 (non binding) >> >> Thanks >> Szehon >> >> On Wed, Mar 18, 2026 at 2:17 PM Yicong Huang <[email protected]> >> wrote: >> >>> +1 (non-binding) >>> >>> >>> On Wed, Mar 18, 2026 at 1:49 PM DB Tsai <[email protected]> wrote: >>> >>>> +1 >>>> >>>> DB Tsai | https://www.dbtsai.com/ | PGP 42E5B25A8F7A82C1 >>>> >>>> On Mar 18, 2026, at 12:24 PM, Chao Sun <[email protected]> wrote: >>>> >>>> +1 (binding) >>>> >>>> On Wed, Mar 18, 2026 at 12:00 PM Herman van Hovell via dev < >>>> [email protected]> wrote: >>>> >>>>> +1 >>>>> >>>>> On Wed, Mar 18, 2026 at 2:57 PM Daniel Tenedorio <[email protected]> >>>>> wrote: >>>>> >>>>>> +1 (non-binding) >>>>>> >>>>>> Thank you for your persistence working on this proposal and figuring >>>>>> out the details. >>>>>> >>>>>> On 2026/03/18 09:32:40 Haiyang Sun via dev wrote: >>>>>> > Hi Spark devs, >>>>>> > >>>>>> > I would like to call for *a new vote following the previous >>>>>> attempt* for the >>>>>> > *SPIP: Language-Agnostic UDF Execution Protocol for Spark *after >>>>>> addressing >>>>>> > comments and providing a supplementary design document for worker >>>>>> > specification. >>>>>> > >>>>>> > The SPIP proposes a structured, language-agnostic framework for >>>>>> running >>>>>> > user-defined functions (UDFs) in Spark across multiple programming >>>>>> languages >>>>>> > >>>>>> > Today, Spark Connect allows users to write queries from multiple >>>>>> languages, >>>>>> > but support for user-defined functions remains incomplete. In >>>>>> practice, >>>>>> > only Scala, Java, Python have working support, and this relies on >>>>>> > language-specific mechanisms that do not generalize well to other >>>>>> languages >>>>>> > such as Go <https://github.com/apache/spark-connect-go> / Rust >>>>>> > <https://github.com/apache/spark-connect-rust> / Swift >>>>>> > <https://github.com/apache/spark-connect-swift> / TypeScript >>>>>> > <https://github.com/BaldrVivaldelli/ts-spark-connector> where UDF >>>>>> support >>>>>> > is currently unavailable. In addition, there are legacy limitations >>>>>> in the >>>>>> > existing PySpark worker implementation that make it difficult to >>>>>> evolve the >>>>>> > system or extend it to new languages. >>>>>> > >>>>>> > The proposal introduces two related components: >>>>>> > >>>>>> > >>>>>> > 1. >>>>>> > >>>>>> > *A unified UDF execution protocol* >>>>>> > >>>>>> > The proposal defines a structured API and execution protocol for >>>>>> running >>>>>> > UDFs outside the Spark executor process and communicating with >>>>>> Spark via >>>>>> > inter-process communication (IPC). This protocol enables Spark >>>>>> to interact >>>>>> > with external UDF workers in a consistent and extensible way, >>>>>> regardless of >>>>>> > the implementation language. >>>>>> > 2. >>>>>> > >>>>>> > *A worker specification for provisioning and lifecycle >>>>>> management.* >>>>>> > >>>>>> > To support multi-language execution environments, the proposal >>>>>> also >>>>>> > introduces a worker specification describing how UDF workers can >>>>>> be >>>>>> > installed, started, connected to, and terminated. This document >>>>>> complements >>>>>> > the SPIP by outlining how workers can be provisioned and managed >>>>>> in a >>>>>> > consistent way. >>>>>> > >>>>>> > Note that this SPIP can help enable UDF support for languages that >>>>>> > currently do not support UDFs. For languages that already have UDF >>>>>> > implementations (especially Python), the goal is not to replace >>>>>> existing >>>>>> > implementations immediately, but to provide a framework that may >>>>>> allow them >>>>>> > to gradually evolve toward more language-agnostic abstractions over >>>>>> time. >>>>>> > >>>>>> > More details can be found in the SPIP document and the >>>>>> supplementary design >>>>>> > for worker specification: >>>>>> > >>>>>> > SPIP: >>>>>> > >>>>>> https://docs.google.com/document/d/19Whzq127QxVt2Luk0EClgaDtcpBsFUp67NcVdKKyPF8 >>>>>> > >>>>>> > Worker specification design document: >>>>>> > >>>>>> https://docs.google.com/document/d/1Dx9NqHRNuUpatH9DYoFF9cmvUl2fqHT4Rjbyw4EGLHs >>>>>> > >>>>>> > Discussion Thread: >>>>>> > https://lists.apache.org/thread/9t4svsnd71j7sb4r4scf2xhh8dvp3b43 >>>>>> > >>>>>> > Previous vote and discussion thread: >>>>>> > https://lists.apache.org/thread/81xghrfwvopp274rgyxfthsstb2xmkz1 >>>>>> > >>>>>> > *Please vote on adopting this proposal.* >>>>>> > >>>>>> > [ ] +1: Accept the proposal as an official SPIP >>>>>> > >>>>>> > [ ] +0: No opinion >>>>>> > >>>>>> > [ ] -1: Disapprove (please explain why) >>>>>> > >>>>>> > The vote will remain open for *at least 72 hours. * >>>>>> > >>>>>> > Thanks to everyone who participated in the discussion and provided >>>>>> valuable >>>>>> > feedback! >>>>>> > >>>>>> > >>>>>> > Best regards, >>>>>> > >>>>>> > Haiyang >>>>>> > >>>>>> >>>>>> --------------------------------------------------------------------- >>>>>> To unsubscribe e-mail: [email protected] >>>>>> >>>>>> >>>>
