xinrong-meng opened a new pull request, #41125: URL: https://github.com/apache/spark/pull/41125
### What changes were proposed in this pull request? The PR proposes to provide support for the registration of an Arrow-optimized Python UDF in both vanilla PySpark and Spark Connect. ### Why are the changes needed? Currently, when users register an Arrow-optimized Python UDF, it will be registered as a pickled Python UDF and thus, executed without Arrow optimization. We should support Arrow-optimized Python UDFs registration and execute them with Arrow optimization. ### Does this PR introduce _any_ user-facing change? Yes. Previously, a registered Arrow-optimized Python UDF will be executed without Arrow optimization. Now, it will be executed with Arrow optimization. Enabling or disabling Arrow optimization can produce result differences in some cases. ### How was this patch tested? Unit test. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
