holdenk opened a new pull request, #56327: URL: https://github.com/apache/spark/pull/56327
### What changes were proposed in this pull request? This is the first PR adding initial transpilation from Python to Catalyst and the framework for others to plug in different transpilers which can target other targets. ### Why are the changes needed? Python UDF performance has been a perinial challenge in Spark, and while PyArrow makes the data copy slightly less expensive it's still much slower than native code. Additionally, Python UDF usage has increased substantailly with Pandas on Spark and it is especially hard for folks to write catalyst expressions here. ### Does this PR introduce _any_ user-facing change? New configuration flags are user visible (default to off). ### How was this patch tested? New unit tests that trigger always and hypothesis tests that trigger only selectively. ### Was this patch authored or co-authored using generative AI tooling? Hypothesis teests were generated by Claude 4.7, eq semantics null and truthiness w/claude 4.6, GH code review (unknown backing model) on itiial draft). snake camel swap with sonnet. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
