[
https://issues.apache.org/jira/browse/SPARK-60241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60241:
-----------------------------------
Labels: pull-request-available (was: )
> Fall back to regular Python UDFs when a Spark Connect server doesn't have
> PyArrow
> ---------------------------------------------------------------------------------
>
> Key: SPARK-60241
> URL: https://issues.apache.org/jira/browse/SPARK-60241
> Project: Spark
> Issue Type: Improvement
> Components: Connect, PySpark
> Affects Versions: 4.2.0
> Reporter: Lisa Cao
> Priority: Major
> Labels: pull-request-available
>
> Since 4.2.0, regular Python UDFs are Arrow-optimized by default
> (SPARK-54555). SPARK-51393 falls back to a regular UDF when PyArrow or pandas
> is missing, but the check runs where the UDF is created. In classic that's
> the driver, so it falls back with a warning. Under Spark Connect it's the
> client, and pyspark-client always installs PyArrow, so the UDF is sent as an
> Arrow UDF and fails in the server's Python worker if the server doesn't have
> PyArrow:
> {noformat}
> ModuleNotFoundError: No module named 'pyarrow'
> {noformat}
> Reproduced with pyspark-client 4.2.0 against
> apache/spark:4.2.0-scala2.13-java21-python3-ubuntu, which ships without
> PyArrow (SPARK-60117 adds it). Workarounds today: {{useArrow=False}} on the
> UDF, or starting the server with
> {{spark.sql.execution.pythonUDF.arrow.enabled=false}}.
> The fallback probably has to happen when the server plans the UDF, since the
> executor already streams Arrow batches to the worker for Arrow UDFs. If a
> server-side fallback isn't wanted, a clear error that points at the
> workarounds would still help.
> From the discussion on https://github.com/apache/spark/pull/59280.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
