+1 (non-binding) Looking forward to having this!
On Thu, Mar 19, 2026 at 7:07 AM vaquar khan <[email protected]> wrote: > +1 > > On Wed, Mar 18, 2026, 4:53 PM Hyukjin Kwon <[email protected]> wrote: > >> +1 >> >> On Thu, 19 Mar 2026 at 06:42, Szehon Ho <[email protected]> wrote: >> >>> +1 (non binding) >>> >>> Thanks >>> Szehon >>> >>> On Wed, Mar 18, 2026 at 2:17 PM Yicong Huang <[email protected]> >>> wrote: >>> >>>> +1 (non-binding) >>>> >>>> >>>> On Wed, Mar 18, 2026 at 1:49 PM DB Tsai <[email protected]> wrote: >>>> >>>>> +1 >>>>> >>>>> DB Tsai | https://www.dbtsai.com/ | PGP 42E5B25A8F7A82C1 >>>>> >>>>> On Mar 18, 2026, at 12:24 PM, Chao Sun <[email protected]> wrote: >>>>> >>>>> +1 (binding) >>>>> >>>>> On Wed, Mar 18, 2026 at 12:00 PM Herman van Hovell via dev < >>>>> [email protected]> wrote: >>>>> >>>>>> +1 >>>>>> >>>>>> On Wed, Mar 18, 2026 at 2:57 PM Daniel Tenedorio <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> +1 (non-binding) >>>>>>> >>>>>>> Thank you for your persistence working on this proposal and figuring >>>>>>> out the details. >>>>>>> >>>>>>> On 2026/03/18 09:32:40 Haiyang Sun via dev wrote: >>>>>>> > Hi Spark devs, >>>>>>> > >>>>>>> > I would like to call for *a new vote following the previous >>>>>>> attempt* for the >>>>>>> > *SPIP: Language-Agnostic UDF Execution Protocol for Spark *after >>>>>>> addressing >>>>>>> > comments and providing a supplementary design document for worker >>>>>>> > specification. >>>>>>> > >>>>>>> > The SPIP proposes a structured, language-agnostic framework for >>>>>>> running >>>>>>> > user-defined functions (UDFs) in Spark across multiple programming >>>>>>> languages >>>>>>> > >>>>>>> > Today, Spark Connect allows users to write queries from multiple >>>>>>> languages, >>>>>>> > but support for user-defined functions remains incomplete. In >>>>>>> practice, >>>>>>> > only Scala, Java, Python have working support, and this relies on >>>>>>> > language-specific mechanisms that do not generalize well to other >>>>>>> languages >>>>>>> > such as Go <https://github.com/apache/spark-connect-go> / Rust >>>>>>> > <https://github.com/apache/spark-connect-rust> / Swift >>>>>>> > <https://github.com/apache/spark-connect-swift> / TypeScript >>>>>>> > <https://github.com/BaldrVivaldelli/ts-spark-connector> where >>>>>>> UDF support >>>>>>> > is currently unavailable. In addition, there are legacy >>>>>>> limitations in the >>>>>>> > existing PySpark worker implementation that make it difficult to >>>>>>> evolve the >>>>>>> > system or extend it to new languages. >>>>>>> > >>>>>>> > The proposal introduces two related components: >>>>>>> > >>>>>>> > >>>>>>> > 1. >>>>>>> > >>>>>>> > *A unified UDF execution protocol* >>>>>>> > >>>>>>> > The proposal defines a structured API and execution protocol >>>>>>> for running >>>>>>> > UDFs outside the Spark executor process and communicating with >>>>>>> Spark via >>>>>>> > inter-process communication (IPC). This protocol enables Spark >>>>>>> to interact >>>>>>> > with external UDF workers in a consistent and extensible way, >>>>>>> regardless of >>>>>>> > the implementation language. >>>>>>> > 2. >>>>>>> > >>>>>>> > *A worker specification for provisioning and lifecycle >>>>>>> management.* >>>>>>> > >>>>>>> > To support multi-language execution environments, the proposal >>>>>>> also >>>>>>> > introduces a worker specification describing how UDF workers >>>>>>> can be >>>>>>> > installed, started, connected to, and terminated. This document >>>>>>> complements >>>>>>> > the SPIP by outlining how workers can be provisioned and >>>>>>> managed in a >>>>>>> > consistent way. >>>>>>> > >>>>>>> > Note that this SPIP can help enable UDF support for languages that >>>>>>> > currently do not support UDFs. For languages that already have UDF >>>>>>> > implementations (especially Python), the goal is not to replace >>>>>>> existing >>>>>>> > implementations immediately, but to provide a framework that may >>>>>>> allow them >>>>>>> > to gradually evolve toward more language-agnostic abstractions >>>>>>> over time. >>>>>>> > >>>>>>> > More details can be found in the SPIP document and the >>>>>>> supplementary design >>>>>>> > for worker specification: >>>>>>> > >>>>>>> > SPIP: >>>>>>> > >>>>>>> https://docs.google.com/document/d/19Whzq127QxVt2Luk0EClgaDtcpBsFUp67NcVdKKyPF8 >>>>>>> > >>>>>>> > Worker specification design document: >>>>>>> > >>>>>>> https://docs.google.com/document/d/1Dx9NqHRNuUpatH9DYoFF9cmvUl2fqHT4Rjbyw4EGLHs >>>>>>> > >>>>>>> > Discussion Thread: >>>>>>> > https://lists.apache.org/thread/9t4svsnd71j7sb4r4scf2xhh8dvp3b43 >>>>>>> > >>>>>>> > Previous vote and discussion thread: >>>>>>> > https://lists.apache.org/thread/81xghrfwvopp274rgyxfthsstb2xmkz1 >>>>>>> > >>>>>>> > *Please vote on adopting this proposal.* >>>>>>> > >>>>>>> > [ ] +1: Accept the proposal as an official SPIP >>>>>>> > >>>>>>> > [ ] +0: No opinion >>>>>>> > >>>>>>> > [ ] -1: Disapprove (please explain why) >>>>>>> > >>>>>>> > The vote will remain open for *at least 72 hours. * >>>>>>> > >>>>>>> > Thanks to everyone who participated in the discussion and provided >>>>>>> valuable >>>>>>> > feedback! >>>>>>> > >>>>>>> > >>>>>>> > Best regards, >>>>>>> > >>>>>>> > Haiyang >>>>>>> > >>>>>>> >>>>>>> --------------------------------------------------------------------- >>>>>>> To unsubscribe e-mail: [email protected] >>>>>>> >>>>>>> >>>>>
