This vote passes. I left it open for a bit longer because Monday was a holiday for some folks.
The votes are: +1s: Huaxin Gao * Hyukjin Kwon* Felix Cheung* Jungtaek Lim Rishab Joshi Chao Sun* Liu Cao Wenchen Fan* Kent Yao* Kousuke Saruta* Peter Toth Dongjoon Hyun* John Zhuge 0s: -1s: None * Binding All of the work will be behind a feature flag so that if after the experiment it turns out not to be a success it can be removed later. Thank you everyone who voted, read the document, left feedback, etc. I'll start filing some smaller JIRAs to break the work down into smaller pieces. On Wed, Jan 14, 2026 at 5:54 PM John Zhuge <[email protected]> wrote: > +1 (non-binding) Promising. Let's maintain focus while recognizing the > potential architectural shift. > > On Wed, Jan 14, 2026 at 3:44 PM Dongjoon Hyun <[email protected]> wrote: > >> +1 >> >> On 2026/01/14 11:24:50 Peter Toth wrote: >> > +1 (non-binding) >> > >> > On Wed, Jan 14, 2026 at 7:25 AM Kousuke Saruta <[email protected]> >> wrote: >> > >> > > +1 >> > > >> > > 2026年1月14日(水) 15:03 Kent Yao <[email protected]>: >> > > >> > >> +1 (binding) >> > >> >> > >> Wenchen Fan <[email protected]> 于2026年1月14日周三 13:06写道： >> > >> >> > >>> +1 (binding) >> > >>> >> > >>> On Wed, Jan 14, 2026 at 11:54 AM Liu Cao <[email protected]> >> wrote: >> > >>> >> > >>>> +1 (non-binding) >> > >>>> >> > >>>> The initial scope seems reasonable; very interested in the long >> term >> > >>>> aspect in accelerator integrations. >> > >>>> >> > >>>> On Tue, Jan 13, 2026 at 19:18 Rishab Joshi <[email protected]> >> > >>>> wrote: >> > >>>> >> > >>>>> +1 (non-binding). >> > >>>>> Interesting problem. Looking forward to seeing the project mature >> and >> > >>>>> evolve. >> > >>>>> >> > >>>>> On Tue, Jan 13, 2026 at 7:05 PM Jungtaek Lim < >> > >>>>> [email protected]> wrote: >> > >>>>> >> > >>>>>> +1 (non-binding) >> > >>>>>> >> > >>>>>> I can't imagine the coverage of UDF for this SPIP at this point, >> but >> > >>>>>> I think this is the way to go if things work out, so definitely >> good to >> > >>>>>> explore. >> > >>>>>> >> > >>>>>> On Wed, Jan 14, 2026 at 12:00 PM Felix Cheung < >> > >>>>>> [email protected]> wrote: >> > >>>>>> >> > >>>>>>> +1 worth a shot, it would be great to have some improvement in >> this >> > >>>>>>> space >> > >>>>>>> >> > >>>>>>> >> > >>>>>>> ------------------------------ >> > >>>>>>> *From:* Hyukjin Kwon <[email protected]> >> > >>>>>>> *Sent:* Tuesday, January 13, 2026 6:50:19 PM >> > >>>>>>> *To:* [email protected] <[email protected]> >> > >>>>>>> *Subject:* Re: [VOTE] SPIP: Improving Spark SQL UDFs with >> > >>>>>>> Transpilation >> > >>>>>>> >> > >>>>>>> I'm +1 on this as long as we can isolate the implementation and >> turn >> > >>>>>>> it off by default till it's proven that it works and it's ready. >> > >>>>>>> >> > >>>>>>> I am not fully confident that it will work but I think it >> should be >> > >>>>>>> worth trying. >> > >>>>>>> >> > >>>>>>> On Tue, Jan 13, 2026 at 6:29 AM huaxin gao < >> [email protected]> >> > >>>>>>> wrote: >> > >>>>>>> >> > >>>>>>> +1 >> > >>>>>>> >> > >>>>>>> On Mon, Jan 12, 2026 at 12:24 PM Holden Karau < >> > >>>>>>> [email protected]> wrote: >> > >>>>>>> >> > >>>>>>> Hi Folks, >> > >>>>>>> >> > >>>>>>> Discussion on the SPIP Spark SQL UDF transpilation >> > >>>>>>> < >> https://lists.apache.org/thread/xj8qfqvo5f9o188984mwh2kcg0fnqs9c> >> > >>>>>>> seems to have settled, so I'm now bringing it for a vote. The >> normal >> > >>>>>>> requirements is that an SPIP vote is open for at least 72 >> hours, but given >> > >>>>>>> that some are just returning from the winter holidays, I plan >> to leave this >> > >>>>>>> vote open until Sunday Jan 18th. >> > >>>>>>> >> > >>>>>>> From the original discussion: >> > >>>>>>> >> > >>>>>>> It's been a few years since we last looked at transpilation, and >> > >>>>>>> with the growth of Pandas on Spark I think it's time we revisit >> it. I've >> > >>>>>>> got a JIRA filed < >> https://issues.apache.org/jira/browse/SPARK-54783> >> > >>>>>>> some rough proof of concept code >> > >>>>>>> <https://github.com/apache/spark/pull/53547> (I think doing the >> > >>>>>>> transpilation Python side instead of Scala side makes more >> sense, but was >> > >>>>>>> interesting to play with), and of course everyones favourite a >> > >>>>>>> design doc. >> > >>>>>>> < >> https://docs.google.com/document/d/1cHc6tiR4yO3hppTzrK1F1w9RwyEPMvaeEuL2ub2LURg/edit?usp=sharing> >> (I >> > >>>>>>> also have a collection of YouTube streams playing with the idea >> > >>>>>>> <https://www.youtube.com/@HoldenKarau/streams> if anyone wants >> to >> > >>>>>>> follow along on that journey). >> > >>>>>>> >> > >>>>>>> Cheers, >> > >>>>>>> >> > >>>>>>> Holden :) >> > >>>>>>> >> > >>>>>>> -- >> > >>>>>>> Twitter: https://twitter.com/holdenkarau >> > >>>>>>> Fight Health Insurance: https://www.fighthealthinsurance.com/ >> > >>>>>>> <https://www.fighthealthinsurance.com/?q=hk_email> >> > >>>>>>> Books (Learning Spark, High Performance Spark, etc.): >> > >>>>>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >> > >>>>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >> > >>>>>>> Pronouns: she/her >> > >>>>>>> >> > >>>>>>> >> > >>>>> >> > >>>>> -- >> > >>>>> Regards >> > >>>>> Rishab Joshi >> > >>>>> >> > >>>> -- >> > >>>> >> > >>>> Liu Cao >> > >>>> >> > >>>> >> > >>>> >> > >> >> --------------------------------------------------------------------- >> To unsubscribe e-mail: [email protected] >> >> > > -- > John Zhuge > -- Twitter: https://twitter.com/holdenkarau Fight Health Insurance: https://www.fighthealthinsurance.com/ <https://www.fighthealthinsurance.com/?q=hk_email> Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> YouTube Live Streams: https://www.youtube.com/user/holdenkarau Pronouns: she/her
