lisancao opened a new pull request, #59778: URL: https://github.com/apache/spark/pull/59778
### What changes were proposed in this pull request? This PR makes Spark Connect fall back from Arrow-optimized regular Python UDFs to regular Python UDF execution when the server-side Python environment cannot run the Arrow path. Today the Connect client validates its own Python environment before sending an Arrow-optimized UDF, but the UDF executes in Python workers launched by the server; if the server lacks the required dependency, the query fails instead of following Spark Classic's fallback behavior. The server now checks the Python executable and worker environment that will run the UDF before planning `SQL_ARROW_BATCHED_UDF`. It downgrades to `SQL_BATCHED_UDF` only when the check proves the required package is unavailable; if the check itself cannot run, the UDF stays Arrow-optimized so unrelated worker-launch errors are not hidden. The result is cached per Connect session by Python executable, worker environment, and whether legacy pandas conversion is enabled. The dependency check is intentionally conditional: * PyArrow is always required for Arrow-optimized Python UDFs. * pandas is required only when `spark.sql.legacy.execution.pythonUDF.pandas.conversion.enabled=true`, because the non-legacy `SQL_ARROW_BATCHED_UDF` worker path no longer imports pandas, while the legacy conversion path still does. ### Why are the changes needed? Spark Connect currently decides to create an Arrow-optimized regular Python UDF on the client, based on the client's Python environment. The UDF runs on the server side, so a client with PyArrow installed can send an Arrow UDF to a server whose Python workers cannot import PyArrow. That causes execution failure even though regular Python UDF execution would work. This is a Connect-specific client/server environment skew. Rechecking on the server makes the fallback decision match the Python environment that actually runs the UDF, while preserving Arrow optimization when the server has the required packages. ### Does this PR introduce _any_ user-facing change? Yes. In Spark Connect, when regular Python UDF Arrow optimization is enabled but the server-side Python cannot run the Arrow-optimized path because PyArrow is unavailable, or because pandas is unavailable while legacy pandas conversion is enabled, Spark falls back to regular Python UDF execution and logs a server-side warning instead of failing the query. This also means that, as in Spark Classic, if only executors have the required packages but the driver/server Python does not, the UDF runs as a regular Python UDF. To keep the UDF Arrow-optimized, install the required packages in the driver/server Python environment. ### How was this patch tested? Added Scala tests for the Python availability probe and Connect planner fallback, including PyArrow missing, pandas missing with and without legacy conversion, session cache behavior, worker executable selection, and registered UDFs. Added a Spark Connect Python integration test for server-side fallback when PyArrow is hidden from server workers, and for the pandas legacy-conversion case. Locally ran: ``` ./build/sbt "sql/testOnly org.apache.spark.sql.execution.python.PythonUDFSuite -- -z SPARK-60241" "connect/testOnly org.apache.spark.sql.connect.planner.SparkConnectPlannerSuite -- -z SPARK-60241" ``` ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (GPT-5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
