This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new bdc08dd [SPARK-61125][CONNECT][PYTHON] Pickle the handler of a UDTF
when it is used, and raise UDTF_SERIALIZATION_ERROR, in the Python client as
PySpark does
bdc08dd is described below
commit bdc08dd08f565121988f010a10fc3d61eb8c2849
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 06:41:26 2026 +0900
[SPARK-61125][CONNECT][PYTHON] Pickle the handler of a UDTF when it is
used, and raise UDTF_SERIALIZATION_ERROR, in the Python client as PySpark does
### What changes were proposed in this pull request?
[SPARK-61125](https://issues.apache.org/jira/browse/SPARK-61125): The
handler class of a UDTF is pickled when the UDTF is used or registered, and a
handler that does not pickle raises `UDTF_SERIALIZATION_ERROR`, as PySpark's.
PySpark Connect pickles the handler of a UDTF as it builds the `PythonUDTF`
of the plan. For a `pickle.PicklingError`, it raises a `PySparkPicklingError`
of `UDTF_SERIALIZATION_ERROR`, of the name of the UDTF and "Please check the
stack trace and make sure the function is serializable."
`UserDefinedTableFunction.command` now pickles the handler when it is read, and
raises that error.
### Why are the changes needed?
The client pickled the handler when `udtf` decorated the class, and raised
the `PicklingError` as it was. So a handler that does not pickle raised where
the UDTF was defined, rather than where it was used, and with another type.
Apache Spark's `test_udtf_pickle_error` failed on this client.
### Does this PR introduce _any_ user-facing change?
Yes. A UDTF of a handler that does not pickle raises
`UDTF_SERIALIZATION_ERROR` when it is used, as with PySpark 4.3.0.
### How was this patch tested?
- A new offline test checks the error of a handler that refers to a file
opened for writing, against a mock Connect server, when the UDTF is called and
when it is registered.
- Apache Spark's `test_udtf_pickle_error` of `test_parity_udtf.py` passes
on this client against a local server, in its three variants.
- The lib tests of both crates (590 passed), the build of all tests, also
with `wasm-udf`, the offline Python suites (2179 passed, 8 skipped), and the
live drop-in and Arrow suites (174 passed) and live cases (10 passed) against a
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #323 from HyukjinKwon/SPARK-61125-udtf-pickling.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
python/pyspark/sql/udtf.py | 24 ++++++++++++++++++++----
python/tests/test_dropin_offline.py | 30 ++++++++++++++++++++++++++++++
2 files changed, 50 insertions(+), 4 deletions(-)
diff --git a/python/pyspark/sql/udtf.py b/python/pyspark/sql/udtf.py
index b13f96e..ab6b1f5 100644
--- a/python/pyspark/sql/udtf.py
+++ b/python/pyspark/sql/udtf.py
@@ -26,10 +26,11 @@ from pyspark._pyspark import (
)
import inspect
+import pickle
import sys
from typing import Any, Optional, Type
-from pyspark.errors import PySparkAttributeError, PySparkTypeError
+from pyspark.errors import PySparkAttributeError, PySparkPicklingError,
PySparkTypeError
from pyspark.serializers import CloudPickleSerializer
__all__ = [
@@ -125,9 +126,24 @@ class UserDefinedTableFunction:
)
self.deterministic = deterministic
self.python_ver = f"{sys.version_info.major}.{sys.version_info.minor}"
- # Connect sends the schema separately in PythonUDTF.return_type. The
- # worker command contains only the handler class, unlike scalar UDFs.
- self.command = CloudPickleSerializer().dumps(func)
+
+ @property
+ def command(self) -> bytes:
+ """The pickled handler class, as PySpark Connect's
``PythonUDTF.command``: pickled when
+ the UDTF is used, and ``UDTF_SERIALIZATION_ERROR`` of a handler that
does not pickle.
+ Connect sends the schema separately, in ``PythonUDTF.return_type``, so
the command is
+ only the handler class, unlike that of a scalar UDF."""
+ try:
+ return CloudPickleSerializer().dumps(self.func)
+ except pickle.PicklingError:
+ raise PySparkPicklingError(
+ errorClass="UDTF_SERIALIZATION_ERROR",
+ messageParameters={
+ "name": self.name,
+ "message": "Please check the stack trace and "
+ "make sure the function is serializable.",
+ },
+ )
def __call__(self, *args: Any, **kwargs: Any) -> Any:
from pyspark import _pyspark
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index 0a53e75..3bc8f45 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -1550,6 +1550,36 @@ def
test_subqueries_are_sent_with_the_plans_they_refer_to_as_pyspark(mock_connec
assert len(options[1]) == 1 and len(options[2]) == 1 and 3 not in options
+def test_udtf_handlers_are_pickled_when_used_as_pyspark(mock_connect,
tmp_path):
+ # As PySpark Connect's PythonUDTF: the handler is pickled when the UDTF is
used, so a
+ # handler that does not pickle raises there, a PySparkPicklingError of
+ # UDTF_SERIALIZATION_ERROR of its PicklingError.
+ from pyspark.errors import PySparkPicklingError
+ from pyspark.sql.functions import udtf
+
+ spark, _ = mock_connect
+ file_obj = open(tmp_path / "file.txt", "w")
+ try:
+
+ @udtf(returnType="x: int")
+ class Unpicklable:
+ def eval(self):
+ file_obj
+ yield (1,)
+
+ for use in (lambda: Unpicklable(), lambda:
spark.udtf.register("unpicklable", Unpicklable)):
+ with pytest.raises(PySparkPicklingError) as error:
+ use()
+ assert error.value.getCondition() == "UDTF_SERIALIZATION_ERROR"
+ assert error.value.getMessageParameters() == {
+ "name": "Unpicklable",
+ "message": "Please check the stack trace and make sure the
function is "
+ "serializable.",
+ }
+ finally:
+ file_obj.close()
+
+
def test_udtfs_are_created_as_pyspark(mock_connect):
# As PySpark's _create_py_udtf and _create_pyarrow_udtf: a UDTF is not
deterministic
# unless asDeterministic() marks it so, in place; arrow_udtf() makes a
PyArrow-native
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]