+1 worth a shot, it would be great to have some improvement in this space
________________________________ From: Hyukjin Kwon <[email protected]> Sent: Tuesday, January 13, 2026 6:50:19 PM To: [email protected] <[email protected]> Subject: Re: [VOTE] SPIP: Improving Spark SQL UDFs with Transpilation I'm +1 on this as long as we can isolate the implementation and turn it off by default till it's proven that it works and it's ready. I am not fully confident that it will work but I think it should be worth trying. On Tue, Jan 13, 2026 at 6:29 AM huaxin gao <[email protected]<mailto:[email protected]>> wrote: +1 On Mon, Jan 12, 2026 at 12:24 PM Holden Karau <[email protected]<mailto:[email protected]>> wrote: Hi Folks, Discussion on the SPIP Spark SQL UDF transpilation<https://lists.apache.org/thread/xj8qfqvo5f9o188984mwh2kcg0fnqs9c> seems to have settled, so I'm now bringing it for a vote. The normal requirements is that an SPIP vote is open for at least 72 hours, but given that some are just returning from the winter holidays, I plan to leave this vote open until Sunday Jan 18th. From the original discussion: It's been a few years since we last looked at transpilation, and with the growth of Pandas on Spark I think it's time we revisit it. I've got a JIRA filed<https://issues.apache.org/jira/browse/SPARK-54783> some rough proof of concept code<https://github.com/apache/spark/pull/53547> (I think doing the transpilation Python side instead of Scala side makes more sense, but was interesting to play with), and of course everyones favourite a design doc.<https://docs.google.com/document/d/1cHc6tiR4yO3hppTzrK1F1w9RwyEPMvaeEuL2ub2LURg/edit?usp=sharing> (I also have a collection of YouTube streams playing with the idea<https://www.youtube.com/@HoldenKarau/streams> if anyone wants to follow along on that journey). Cheers, Holden :) -- Twitter: https://twitter.com/holdenkarau Fight Health Insurance: https://www.fighthealthinsurance.com/<https://www.fighthealthinsurance.com/?q=hk_email> Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> YouTube Live Streams: https://www.youtube.com/user/holdenkarau Pronouns: she/her
