Lisa Cao created SPARK-60241:
--------------------------------
Summary: Fall back to regular Python UDFs when a Spark Connect
server doesn't have PyArrow
Key: SPARK-60241
URL: https://issues.apache.org/jira/browse/SPARK-60241
Project: Spark
Issue Type: Improvement
Components: PySpark, Connect
Affects Versions: 4.2.0
Reporter: Lisa Cao
Since 4.2.0, regular Python UDFs are Arrow-optimized by default (SPARK-54555).
SPARK-51393 falls back to a regular UDF when PyArrow or pandas is missing, but
the check runs where the UDF is created. In classic that's the driver, so it
falls back with a warning. Under Spark Connect it's the client, and
pyspark-client always installs PyArrow, so the UDF is sent as an Arrow UDF and
fails in the server's Python worker if the server doesn't have PyArrow:
{noformat}
ModuleNotFoundError: No module named 'pyarrow'
{noformat}
Reproduced with pyspark-client 4.2.0 against
apache/spark:4.2.0-scala2.13-java21-python3-ubuntu, which ships without PyArrow
(SPARK-60117 adds it). Workarounds today: {{useArrow=False}} on the UDF, or
starting the server with {{spark.sql.execution.pythonUDF.arrow.enabled=false}}.
The fallback probably has to happen when the server plans the UDF, since the
executor already streams Arrow batches to the worker for Arrow UDFs. If a
server-side fallback isn't wanted, a clear error that points at the workarounds
would still help.
>From the discussion on https://github.com/apache/spark/pull/59280.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
