This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new bdc08dd  [SPARK-61125][CONNECT][PYTHON] Pickle the handler of a UDTF 
when it is used, and raise UDTF_SERIALIZATION_ERROR, in the Python client as 
PySpark does
bdc08dd is described below

commit bdc08dd08f565121988f010a10fc3d61eb8c2849
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 06:41:26 2026 +0900

    [SPARK-61125][CONNECT][PYTHON] Pickle the handler of a UDTF when it is 
used, and raise UDTF_SERIALIZATION_ERROR, in the Python client as PySpark does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-61125](https://issues.apache.org/jira/browse/SPARK-61125): The 
handler class of a UDTF is pickled when the UDTF is used or registered, and a 
handler that does not pickle raises `UDTF_SERIALIZATION_ERROR`, as PySpark's.
    
    PySpark Connect pickles the handler of a UDTF as it builds the `PythonUDTF` 
of the plan. For a `pickle.PicklingError`, it raises a `PySparkPicklingError` 
of `UDTF_SERIALIZATION_ERROR`, of the name of the UDTF and "Please check the 
stack trace and make sure the function is serializable." 
`UserDefinedTableFunction.command` now pickles the handler when it is read, and 
raises that error.
    
    ### Why are the changes needed?
    
    The client pickled the handler when `udtf` decorated the class, and raised 
the `PicklingError` as it was. So a handler that does not pickle raised where 
the UDTF was defined, rather than where it was used, and with another type. 
Apache Spark's `test_udtf_pickle_error` failed on this client.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. A UDTF of a handler that does not pickle raises 
`UDTF_SERIALIZATION_ERROR` when it is used, as with PySpark 4.3.0.
    
    ### How was this patch tested?
    
    - A new offline test checks the error of a handler that refers to a file 
opened for writing, against a mock Connect server, when the UDTF is called and 
when it is registered.
    - Apache Spark's `test_udtf_pickle_error` of `test_parity_udtf.py` passes 
on this client against a local server, in its three variants.
    - The lib tests of both crates (590 passed), the build of all tests, also 
with `wasm-udf`, the offline Python suites (2179 passed, 8 skipped), and the 
live drop-in and Arrow suites (174 passed) and live cases (10 passed) against a 
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #323 from HyukjinKwon/SPARK-61125-udtf-pickling.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 python/pyspark/sql/udtf.py          | 24 ++++++++++++++++++++----
 python/tests/test_dropin_offline.py | 30 ++++++++++++++++++++++++++++++
 2 files changed, 50 insertions(+), 4 deletions(-)

diff --git a/python/pyspark/sql/udtf.py b/python/pyspark/sql/udtf.py
index b13f96e..ab6b1f5 100644
--- a/python/pyspark/sql/udtf.py
+++ b/python/pyspark/sql/udtf.py
@@ -26,10 +26,11 @@ from pyspark._pyspark import (
 )
 
 import inspect
+import pickle
 import sys
 from typing import Any, Optional, Type
 
-from pyspark.errors import PySparkAttributeError, PySparkTypeError
+from pyspark.errors import PySparkAttributeError, PySparkPicklingError, 
PySparkTypeError
 from pyspark.serializers import CloudPickleSerializer
 
 __all__ = [
@@ -125,9 +126,24 @@ class UserDefinedTableFunction:
         )
         self.deterministic = deterministic
         self.python_ver = f"{sys.version_info.major}.{sys.version_info.minor}"
-        # Connect sends the schema separately in PythonUDTF.return_type. The
-        # worker command contains only the handler class, unlike scalar UDFs.
-        self.command = CloudPickleSerializer().dumps(func)
+
+    @property
+    def command(self) -> bytes:
+        """The pickled handler class, as PySpark Connect's 
``PythonUDTF.command``: pickled when
+        the UDTF is used, and ``UDTF_SERIALIZATION_ERROR`` of a handler that 
does not pickle.
+        Connect sends the schema separately, in ``PythonUDTF.return_type``, so 
the command is
+        only the handler class, unlike that of a scalar UDF."""
+        try:
+            return CloudPickleSerializer().dumps(self.func)
+        except pickle.PicklingError:
+            raise PySparkPicklingError(
+                errorClass="UDTF_SERIALIZATION_ERROR",
+                messageParameters={
+                    "name": self.name,
+                    "message": "Please check the stack trace and "
+                    "make sure the function is serializable.",
+                },
+            )
 
     def __call__(self, *args: Any, **kwargs: Any) -> Any:
         from pyspark import _pyspark
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index 0a53e75..3bc8f45 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -1550,6 +1550,36 @@ def 
test_subqueries_are_sent_with_the_plans_they_refer_to_as_pyspark(mock_connec
     assert len(options[1]) == 1 and len(options[2]) == 1 and 3 not in options
 
 
+def test_udtf_handlers_are_pickled_when_used_as_pyspark(mock_connect, 
tmp_path):
+    # As PySpark Connect's PythonUDTF: the handler is pickled when the UDTF is 
used, so a
+    # handler that does not pickle raises there, a PySparkPicklingError of
+    # UDTF_SERIALIZATION_ERROR of its PicklingError.
+    from pyspark.errors import PySparkPicklingError
+    from pyspark.sql.functions import udtf
+
+    spark, _ = mock_connect
+    file_obj = open(tmp_path / "file.txt", "w")
+    try:
+
+        @udtf(returnType="x: int")
+        class Unpicklable:
+            def eval(self):
+                file_obj
+                yield (1,)
+
+        for use in (lambda: Unpicklable(), lambda: 
spark.udtf.register("unpicklable", Unpicklable)):
+            with pytest.raises(PySparkPicklingError) as error:
+                use()
+            assert error.value.getCondition() == "UDTF_SERIALIZATION_ERROR"
+            assert error.value.getMessageParameters() == {
+                "name": "Unpicklable",
+                "message": "Please check the stack trace and make sure the 
function is "
+                "serializable.",
+            }
+    finally:
+        file_obj.close()
+
+
 def test_udtfs_are_created_as_pyspark(mock_connect):
     # As PySpark's _create_py_udtf and _create_pyarrow_udtf: a UDTF is not 
deterministic
     # unless asDeterministic() marks it so, in place; arrow_udtf() makes a 
PyArrow-native


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to