This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new f9438ab  [SPARK-60962][CONNECT][PYTHON] Pickle user-defined types as 
Python objects, and make them of arguments, as PySpark does
f9438ab is described below

commit f9438abb721267237753a84ffbb04e453eafc47d
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Fri Nov 6 23:00:43 2026 +0900

    [SPARK-60962][CONNECT][PYTHON] Pickle user-defined types as Python objects, 
and make them of arguments, as PySpark does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-60962](https://issues.apache.org/jira/browse/SPARK-60962): 
user-defined types now pickle as Python objects, and are made with the 
arguments of their `__init__`, as PySpark's do.
    
    - As PySpark's `UserDefinedType`, a Python object, a UDT pickles as its 
class and its state. cloudpickle sends a class of `__main__` by value, and the 
state keeps the parameters of the instance. It pickled as its JSON, which names 
the class by its module, and is rebuilt without arguments.
    - As PySpark's `DataType`, whose `__new__` is `object.__new__`, a subclass 
of `DataType`, such as a UDT, is made with the arguments of its `__init__`.
    
    ### Why are the changes needed?
    
    A UDT that `__main__` defines, as in a notebook or a script, failed as the 
return type of a UDF: the worker rebuilt it from its JSON, by importing its 
module, which `__main__` on the worker does not define. A UDT whose `__init__` 
keeps parameters was rebuilt without them. Even making one failed: 
`PointUDT(10.0)` raised `TypeError: DataType.__new__() takes 0 positional 
arguments but 1 was given`.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. A UDT of `__main__`, and of parameters, is a UDF return type, as in 
PySpark 4.3.0.
    
    ### How was this patch tested?
    
    - A new live test runs, in a new interpreter, a UDF whose return type is a 
UDT that `__main__` defines with a parameter, and collects its values. It also 
passes with official PySpark 4.3.0, and fails without this change.
    - A new offline test checks that a UDT of parameters pickles and copies 
with them. It also loads, here, a UDT that another interpreter's `__main__` 
defined and pickled. It fails without this change.
    - The lib tests and every integration test, e2e included, against a local 
Spark 4.3.0 server (664 passed, and 125 of the transport crate), the build of 
all tests with `wasm-udf`, the offline Python suites (1328 passed, 8 skipped), 
and the live drop-in and Arrow suites (87 passed) and live cases (10 passed). 
Clippy reports no diagnostics in the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #212 from HyukjinKwon/SPARK-60962-udt-pickle.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/types.rs      |  8 ++++++-
 python/pyspark/sql/types.py         |  9 +++++++
 python/tests/test_dropin_e2e.py     | 48 +++++++++++++++++++++++++++++++++++++
 python/tests/test_dropin_offline.py | 45 ++++++++++++++++++++++++++++++++++
 4 files changed, 109 insertions(+), 1 deletion(-)

diff --git a/crates/pyspark-rs/src/types.rs b/crates/pyspark-rs/src/types.rs
index 0c44a6c..78125cf 100644
--- a/crates/pyspark-rs/src/types.rs
+++ b/crates/pyspark-rs/src/types.rs
@@ -150,8 +150,14 @@ impl PyDataType {
     /// (e.g. a user's `UserDefinedType`). The sentinel `inner` is never 
consulted for
     /// such subclasses — they override the object-model methods in Python, and
     /// `py_to_data_type` special-cases `UserDefinedType` before reading 
`inner`.
+    /// As `object.__new__` of PySpark's pure-Python `DataType`: the arguments 
are those of the
+    /// `__init__` of a subclass, such as a `UserDefinedType` of parameters.
     #[new]
-    fn __new__() -> Self {
+    #[pyo3(signature = (*_args, **_kwargs))]
+    fn __new__(
+        _args: &Bound<'_, pyo3::types::PyTuple>,
+        _kwargs: Option<&Bound<'_, pyo3::types::PyDict>>,
+    ) -> Self {
         PyDataType {
             inner: DataType::Null,
         }
diff --git a/python/pyspark/sql/types.py b/python/pyspark/sql/types.py
index 43abd15..71b1c3f 100644
--- a/python/pyspark/sql/types.py
+++ b/python/pyspark/sql/types.py
@@ -120,6 +120,15 @@ class UserDefinedType(DataType):
     def typeName(cls) -> str:
         return cls.__name__.lower()
 
+    def __reduce__(self):
+        # As PySpark's UDT, a Python object, which pickles as its class and 
its state:
+        # cloudpickle sends a class of __main__ by value, and the state keeps 
the parameters
+        # of the instance. DataType.__reduce__ would send its JSON, which 
names the class by
+        # its module and rebuilds the instance without arguments.
+        import copyreg
+
+        return copyreg.__newobj__, (type(self),), self.__dict__ or None
+
     @classmethod
     def sqlType(cls) -> "DataType":
         """Underlying SQL storage type for this UDT."""
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index ed08be5..5eeee7c 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -1036,6 +1036,54 @@ def test_ml_vector_functions_as_pyspark(session):
     assert tuple(row.first()) == ([1.5, 2.0], [1.5, 2.0])
 
 
+def test_udfs_return_user_defined_types_of_main_as_pyspark():
+    # As PySpark's: a UDT that __main__ defines, of parameters, is a UDF 
return type, as
+    # cloudpickle sends its class by value and its state to the worker. In a 
new
+    # interpreter, whose __main__ defines the classes.
+    import subprocess
+    import sys
+
+    remote = os.environ.get("SPARK_REMOTE")
+    if not remote:
+        pytest.skip("SPARK_REMOTE is required for live-server tests")
+    subprocess.run([sys.executable, "-c", """
+import sys
+from pyspark.sql import Row, SparkSession, functions as F
+from pyspark.sql.types import ArrayType, DoubleType, UserDefinedType
+
+class Point:
+    def __init__(self, x, y):
+        self.x, self.y = x, y
+
+    def __eq__(self, other):
+        return isinstance(other, Point) and (self.x, self.y) == (other.x, 
other.y)
+
+class PointUDT(UserDefinedType):
+    def __init__(self, scale=1.0):
+        self.scale = scale
+
+    @classmethod
+    def sqlType(cls):
+        return ArrayType(DoubleType(), False)
+
+    @classmethod
+    def module(cls):
+        return "__main__"
+
+    def serialize(self, obj):
+        return [obj.x * self.scale, obj.y * self.scale]
+
+    def deserialize(self, datum):
+        return Point(datum[0], datum[1])
+
+Point.__UDT__ = PointUDT()
+spark = SparkSession.builder.remote(sys.argv[1]).getOrCreate()
+df = spark.range(2).select(F.udf(lambda i: Point(i, i + 1), 
PointUDT(10.0))("id").alias("p"))
+assert df.collect() == [Row(p=Point(0.0, 10.0)), Row(p=Point(10.0, 20.0))]
+assert df.select(F.to_json(F.struct("p"))).collect()[1][0] == 
'{"p":[10.0,20.0]}'
+""", remote], check=True)
+
+
 def 
test_collected_values_of_user_defined_types_are_deserialized_as_pyspark(session):
     # As official PySpark 4.3.0: a collected value of a UDT is its 
deserialized object, and
     # dtypes names the UDT by its simpleString(), in every way that rows are 
collected.
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index 3417e6a..ce58183 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -986,6 +986,51 @@ class _DayPointUDT(T.UserDefinedType):
         return tuple(datum)
 
 
+class _ScaledPointUDT(_DayPointUDT):
+    def __init__(self, scale=1.0):
+        self.scale = scale
+
+
+def test_user_defined_types_pickle_as_python_objects_as_pyspark():
+    # As PySpark's UDT, a Python object: a UDT of parameters is made with 
them, and pickles
+    # as its class and its state, which cloudpickle sends by value for a class 
of __main__.
+    # In a new interpreter, whose __main__ defines the class, so that this one 
does not.
+    import copy
+    import pickle
+    import subprocess
+    import sys
+
+    from pyspark.serializers import CloudPickleSerializer
+
+    udt = _ScaledPointUDT(3.0)
+    for copied in (pickle.loads(pickle.dumps(udt)), copy.deepcopy(udt)):
+        assert type(copied) is _ScaledPointUDT and copied.scale == 3.0
+    assert pickle.loads(pickle.dumps(_DayPointUDT())).__dict__ == {}
+
+    pickled = subprocess.run([sys.executable, "-c", """
+import sys
+from pyspark.serializers import CloudPickleSerializer
+from pyspark.sql.types import ArrayType, DoubleType, UserDefinedType
+
+class MainPointUDT(UserDefinedType):
+    def __init__(self, scale):
+        self.scale = scale
+
+    @classmethod
+    def sqlType(cls):
+        return ArrayType(DoubleType())
+
+    @classmethod
+    def module(cls):
+        return "__main__"
+
+sys.stdout.buffer.write(CloudPickleSerializer().dumps(MainPointUDT(2.5)))
+"""], check=True, capture_output=True).stdout
+    main_udt = CloudPickleSerializer().loads(pickled)
+    assert type(main_udt).__name__ == "MainPointUDT" and main_udt.scale == 2.5
+    assert main_udt.sqlType() == T.ArrayType(T.DoubleType())
+
+
 def test_collected_values_of_user_defined_types_are_deserialized_as_pyspark():
     # The converters that the client applies to collected rows: the values of 
user-defined
     # types, in arrays, maps and structs too, are deserialized by their UDTs, 
as PySpark's


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to