Lisa Cao created SPARK-60828:
--------------------------------
Summary: Fall back to regular Python UDTFs when a Spark Connect
server doesn't have PyArrow
Key: SPARK-60828
URL: https://issues.apache.org/jira/browse/SPARK-60828
Project: Spark
Issue Type: Improvement
Components: PySpark, Connect
Affects Versions: 4.2.0, 4.2.1, 4.3.0
Environment: Docker (Ubuntu 24.04), Java 17.0.20.1 (Temurin), Python
3.11.15, pandas 3.1.0, pyarrow 26.0.0; Spark binary distributions from the
Apache mirrors, and master built from source; Connect clients from
pyspark-client; classic PySpark in local mode; for 3.5.9 (2026-07-17), the
Python packages current at the release; the Python that runs Spark's Python
workers has no pandas or PyArrow
Reporter: Lisa Cao
Since 4.2.0 (SPARK-54555), Python UDTFs are Arrow-optimized by default. Classic
PySpark falls back to a regular UDTF, with a warning, when its Python doesn't
have PyArrow or pandas. Spark Connect makes that check in the client's Python
({{_create_py_udtf}} in {{pyspark/sql/connect/udtf.py}}), which always has
both, so the server runs an Arrow-optimized UDTF, and a default {{@udtf}} fails
when the server's Python doesn't have PyArrow. SPARK-60241 is the same gap for
Python UDFs.
h3. Reproduce
{code:python}
from pyspark.sql import SparkSession
from pyspark.sql.functions import lit, udtf
spark = SparkSession.builder.getOrCreate()
class Count:
def eval(self, n: int):
for i in range(n):
yield (i,)
count = udtf(Count, returnType="i: int")
rows = [r.i for r in count(lit(3)).collect()]
assert rows == [0, 1, 2], rows
{code}
Run it with classic PySpark ({{python repro.py}}), and through Spark Connect
with {{SPARK_REMOTE=sc://<host>:15002 python repro.py}}. The Python that runs
Spark's Python workers has no PyArrow or pandas (in classic, neither does the
driver's Python); the Connect client has both.
h3. Actual
{noformat}
pyspark.errors.exceptions.connect.PythonException:
An exception was thrown from the Python worker. Please see the stack trace
below.
[PYTHON_EXCEPTION] An exception was thrown from the Python worker: Traceback
(most recent call last):
File "/opt/spark/python/lib/pyspark.zip/pyspark/worker.py", line 202, in
invoke_udf
handler = handler_cls(
File "/opt/spark/python/lib/pyspark.zip/pyspark/eval_handlers/_udtf.py", line
1134, in __init__
import pyarrow # noqa: F401
ModuleNotFoundError: No module named 'pyarrow'
SQLSTATE: 38000
{noformat}
h3. Expected
The UDTF returns rows 0, 1 and 2, as in classic PySpark: Spark Connect falls
back to a regular Python UDTF when the server's Python doesn't have PyArrow,
and logs a warning, as SPARK-60241 proposes for Python UDFs.
h3. Where it reproduces
||Spark||Classic||Spark Connect||
|3.5.9|PASS|PASS|
|4.0.5|PASS|PASS|
|4.1.4|PASS|PASS|
|4.2.0|PASS|FAIL|
|4.2.1|PASS|FAIL|
|4.3.0|PASS|FAIL|
|master (5.0.0-SNAPSHOT, 9e771a7bef9)|PASS|FAIL|
Docker (Ubuntu 24.04), Java 17.0.20.1 (Temurin), Python 3.11.15, pandas 3.1.0,
pyarrow 26.0.0.
h3. Notes
* Introduced by SPARK-54555, which made Arrow-optimized Python UDFs and UDTFs
the default in 4.2.0: 4.1.4 passes and 4.2.0 fails.
* Still fails on master at 9e771a7bef9 (2026-11-02).
* The official images ship without PyArrow: a Spark Connect server started from
{{apache/spark:4.2.0}} as shipped (Python 3.10, no pandas or PyArrow) fails the
same way.
* Older clients fail too. The client reads
{{spark.sql.execution.pythonUDTF.arrow.enabled}} from the server, so
pyspark-client 4.1.4 fails the same way against 4.2.0, 4.2.1, 4.3.0 and master
servers, and passes against a 4.1.4 server. The fallback belongs on the server,
as in SPARK-60241.
* The open SPARK-60241 PR (https://github.com/apache/spark/pull/59778) handles
Python UDFs only; with it applied, this repro still fails through Spark Connect.
* Workarounds, both passing on 4.2.0, 4.2.1, 4.3.0 and master, in classic and
through Spark Connect: {{useArrow=False}} on the UDTF, or
{{spark.conf.set("spark.sql.execution.pythonUDTF.arrow.enabled", "false")}}
before creating it.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
