+1, Interesting problem to solve in python. Catalyst udf Optimization. Thanks Anand
On Wed, Jan 21, 2026, 5:15 PM Holden Karau <[email protected]> wrote: > This vote passes. I left it open for a bit longer because Monday was a > holiday for some folks. > > The votes are: > > +1s: > Huaxin Gao * > Hyukjin Kwon* > Felix Cheung* > Jungtaek Lim > Rishab Joshi > Chao Sun* > Liu Cao > Wenchen Fan* > Kent Yao* > Kousuke Saruta* > Peter Toth > Dongjoon Hyun* > John Zhuge > > 0s: > > -1s: > > None > > * Binding > > All of the work will be behind a feature flag so that if after the > experiment it turns out not to be a success it can be removed later. > > Thank you everyone who voted, read the document, left feedback, etc. I'll > start filing some smaller JIRAs to break the work down into smaller pieces. > > On Wed, Jan 14, 2026 at 5:54 PM John Zhuge <[email protected]> wrote: > >> +1 (non-binding) Promising. Let's maintain focus while recognizing the >> potential architectural shift. >> >> On Wed, Jan 14, 2026 at 3:44 PM Dongjoon Hyun <[email protected]> >> wrote: >> >>> +1 >>> >>> On 2026/01/14 11:24:50 Peter Toth wrote: >>> > +1 (non-binding) >>> > >>> > On Wed, Jan 14, 2026 at 7:25 AM Kousuke Saruta <[email protected]> >>> wrote: >>> > >>> > > +1 >>> > > >>> > > 2026年1月14日(水) 15:03 Kent Yao <[email protected]>: >>> > > >>> > >> +1 (binding) >>> > >> >>> > >> Wenchen Fan <[email protected]> 于2026年1月14日周三 13:06写道： >>> > >> >>> > >>> +1 (binding) >>> > >>> >>> > >>> On Wed, Jan 14, 2026 at 11:54 AM Liu Cao <[email protected]> >>> wrote: >>> > >>> >>> > >>>> +1 (non-binding) >>> > >>>> >>> > >>>> The initial scope seems reasonable; very interested in the long >>> term >>> > >>>> aspect in accelerator integrations. >>> > >>>> >>> > >>>> On Tue, Jan 13, 2026 at 19:18 Rishab Joshi <[email protected] >>> > >>> > >>>> wrote: >>> > >>>> >>> > >>>>> +1 (non-binding). >>> > >>>>> Interesting problem. Looking forward to seeing the project >>> mature and >>> > >>>>> evolve. >>> > >>>>> >>> > >>>>> On Tue, Jan 13, 2026 at 7:05 PM Jungtaek Lim < >>> > >>>>> [email protected]> wrote: >>> > >>>>> >>> > >>>>>> +1 (non-binding) >>> > >>>>>> >>> > >>>>>> I can't imagine the coverage of UDF for this SPIP at this >>> point, but >>> > >>>>>> I think this is the way to go if things work out, so definitely >>> good to >>> > >>>>>> explore. >>> > >>>>>> >>> > >>>>>> On Wed, Jan 14, 2026 at 12:00 PM Felix Cheung < >>> > >>>>>> [email protected]> wrote: >>> > >>>>>> >>> > >>>>>>> +1 worth a shot, it would be great to have some improvement in >>> this >>> > >>>>>>> space >>> > >>>>>>> >>> > >>>>>>> >>> > >>>>>>> ------------------------------ >>> > >>>>>>> *From:* Hyukjin Kwon <[email protected]> >>> > >>>>>>> *Sent:* Tuesday, January 13, 2026 6:50:19 PM >>> > >>>>>>> *To:* [email protected] <[email protected]> >>> > >>>>>>> *Subject:* Re: [VOTE] SPIP: Improving Spark SQL UDFs with >>> > >>>>>>> Transpilation >>> > >>>>>>> >>> > >>>>>>> I'm +1 on this as long as we can isolate the implementation >>> and turn >>> > >>>>>>> it off by default till it's proven that it works and it's >>> ready. >>> > >>>>>>> >>> > >>>>>>> I am not fully confident that it will work but I think it >>> should be >>> > >>>>>>> worth trying. >>> > >>>>>>> >>> > >>>>>>> On Tue, Jan 13, 2026 at 6:29 AM huaxin gao < >>> [email protected]> >>> > >>>>>>> wrote: >>> > >>>>>>> >>> > >>>>>>> +1 >>> > >>>>>>> >>> > >>>>>>> On Mon, Jan 12, 2026 at 12:24 PM Holden Karau < >>> > >>>>>>> [email protected]> wrote: >>> > >>>>>>> >>> > >>>>>>> Hi Folks, >>> > >>>>>>> >>> > >>>>>>> Discussion on the SPIP Spark SQL UDF transpilation >>> > >>>>>>> < >>> https://lists.apache.org/thread/xj8qfqvo5f9o188984mwh2kcg0fnqs9c> >>> > >>>>>>> seems to have settled, so I'm now bringing it for a vote. The >>> normal >>> > >>>>>>> requirements is that an SPIP vote is open for at least 72 >>> hours, but given >>> > >>>>>>> that some are just returning from the winter holidays, I plan >>> to leave this >>> > >>>>>>> vote open until Sunday Jan 18th. >>> > >>>>>>> >>> > >>>>>>> From the original discussion: >>> > >>>>>>> >>> > >>>>>>> It's been a few years since we last looked at transpilation, >>> and >>> > >>>>>>> with the growth of Pandas on Spark I think it's time we >>> revisit it. I've >>> > >>>>>>> got a JIRA filed < >>> https://issues.apache.org/jira/browse/SPARK-54783> >>> > >>>>>>> some rough proof of concept code >>> > >>>>>>> <https://github.com/apache/spark/pull/53547> (I think doing >>> the >>> > >>>>>>> transpilation Python side instead of Scala side makes more >>> sense, but was >>> > >>>>>>> interesting to play with), and of course everyones favourite a >>> > >>>>>>> design doc. >>> > >>>>>>> < >>> https://docs.google.com/document/d/1cHc6tiR4yO3hppTzrK1F1w9RwyEPMvaeEuL2ub2LURg/edit?usp=sharing> >>> (I >>> > >>>>>>> also have a collection of YouTube streams playing with the idea >>> > >>>>>>> <https://www.youtube.com/@HoldenKarau/streams> if anyone >>> wants to >>> > >>>>>>> follow along on that journey). >>> > >>>>>>> >>> > >>>>>>> Cheers, >>> > >>>>>>> >>> > >>>>>>> Holden :) >>> > >>>>>>> >>> > >>>>>>> -- >>> > >>>>>>> Twitter: https://twitter.com/holdenkarau >>> > >>>>>>> Fight Health Insurance: https://www.fighthealthinsurance.com/ >>> > >>>>>>> <https://www.fighthealthinsurance.com/?q=hk_email> >>> > >>>>>>> Books (Learning Spark, High Performance Spark, etc.): >>> > >>>>>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >>> > >>>>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >>> > >>>>>>> Pronouns: she/her >>> > >>>>>>> >>> > >>>>>>> >>> > >>>>> >>> > >>>>> -- >>> > >>>>> Regards >>> > >>>>> Rishab Joshi >>> > >>>>> >>> > >>>> -- >>> > >>>> >>> > >>>> Liu Cao >>> > >>>> >>> > >>>> >>> > >>>> >>> > >>> >>> --------------------------------------------------------------------- >>> To unsubscribe e-mail: [email protected] >>> >>> >> >> -- >> John Zhuge >> > > > -- > Twitter: https://twitter.com/holdenkarau > Fight Health Insurance: https://www.fighthealthinsurance.com/ > <https://www.fighthealthinsurance.com/?q=hk_email> > Books (Learning Spark, High Performance Spark, etc.): > https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> > YouTube Live Streams: https://www.youtube.com/user/holdenkarau > Pronouns: she/her >
