viirya opened a new pull request, #59676: URL: https://github.com/apache/spark/pull/59676
### What changes were proposed in this pull request? Backport of #58978 to branch-4.x. The cherry-pick conflicted only in the root `pom.xml`, where the new `jep.version` property sits next to `ammonite.version`, which differs on this branch (3.0.8 instead of 3.0.9). The branch's version is kept; otherwise the change is the same as on master. SPARK-59275 and SPARK-60071, which this change relies on, are already on this branch. ### Why are the changes needed? Same as #58978. Worker-based Arrow UDFs exchange data between executor JVMs and separate Python processes, which can dominate execution time for large inputs or outputs with relatively inexpensive computation. In-process UDFs run CPython inside the executor JVM through JEP and exchange Arrow buffers through the C Data Interface. ### Does this PR introduce _any_ user-facing change? Yes, as described in #58978: a new opt-in `pyspark.inprocess.inprocess_udf` API, the `spark.python.inProcess.sitePackages` and `spark.sql.execution.pythonUDF.inProcess.fullValidation.enabled` configurations, and the `InProcessPythonPlugin` executor plugin. JEP and `arrow-c-data` are `provided` dependencies that users install separately. Existing Python UDF execution is unchanged. ### How was this patch tested? The tests added in #58978 pass on this branch with JDK 17: `InProcessPythonRuntimeSuite`, `InProcessPythonUDFSuite`, `InProcessArrowBridgeSuite`, `ConvertToCatalystSuite`, `CollapseProjectSuite` and `InvalidInputErrorsSuite`, as do `PythonUDFSuite`, `ArrowEvalPythonExecSuite`, `BatchEvalPythonExecSuite`, `ExtractPythonUDFsSuite`, `RowQueueSuite`, `CachedTableSuite` and `DatasetCacheSuite`, except `SPARK-27052` in `PythonUDFSuite`, which needs PyArrow in the system Python used by workers. The PySpark suites `test_inprocess_runtime` (50 tests) and `test_inprocess_udf` (83 tests) pass on macOS/Python 3.13 (PyArrow 23, JEP 4.3.1). ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (review, implementation fixes, tests, CI integration, benchmarks, and documentation; model/version not recorded) Generated-by: Claude Code (Claude Opus 5.5; performance fixes, review fixes, tests, benchmarks and this backport) This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
