This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 1ae429f  [SPARK-61128][CONNECT][PYTHON] Make ArrayType, MapType, 
StructField and StructType give back and pickle the user-defined types they are 
made of, and StructField picklable, in the Python client as PySpark does
1ae429f is described below

commit 1ae429f96bda9a5a232bf3355a1b72cc76897ec8
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 06:24:49 2026 +0900

    [SPARK-61128][CONNECT][PYTHON] Make ArrayType, MapType, StructField and 
StructType give back and pickle the user-defined types they are made of, and 
StructField picklable, in the Python client as PySpark does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-61128](https://issues.apache.org/jira/browse/SPARK-61128): PySpark's 
`ArrayType`, `MapType`, `StructField` and `StructType` hold the type objects 
they are made of. So `ArrayType(udt).elementType is udt`, for a user-defined 
type `udt`, and a type pickles of those objects. The client's types now do the 
same:
    
    - **Giving back the user-defined types.** The client lowers its types into 
the Rust core. It now keeps each user-defined type it lowers, keyed by the JSON 
of its core type, and gives that object back wherever the type is turned back 
into a Python object:
      - the children of `ArrayType`, `MapType`, `StructField` and `StructType`, 
also nested
      - their `repr` and `simpleString`, and the conversions of their values
    
      A user-defined type that the client did not lower, such as one of the 
schema of a result, is still rebuilt from its JSON, as PySpark's 
`UserDefinedType.fromJson` does.
    - **Pickling.** A type with a user-defined type pickles as its class of 
`pyspark.sql.types` with its type objects, as PySpark's types do, and the 
server's Python workers rebuild it. A type without one still pickles by its 
JSON.
    - **`StructField` pickles**, of its name, type, nullability and metadata. 
Before, it could not be pickled at all.
    
    ### Why are the changes needed?
    
    The client rebuilt a user-defined type from its JSON, by importing its 
`pyClass`. That name need not be the module of the class: for Apache Spark's 
`ExamplePointUDT` it is `pyspark.sql.tests.ExamplePointUDT`, while the class is 
in `pyspark.testing.objects`. So on the client, these raised 
`PySparkValueError: [UNSUPPORTED_OPERATION] unpickling user defined types is 
not supported`:
    - `ArrayType(ExamplePointUDT()).elementType`, and the type of 
`StructField("p", ExamplePointUDT())`
    - their `repr`
    - `udf(returnType=ArrayType(ExamplePointUDT()))`, so Apache Spark's 
`test_udf_with_udt` failed on this client
    
    Pickled by its JSON, such a type would fail the same way on the server's 
Python worker.
    
    `pickle.dumps(StructField("a", IntegerType()))` raised `TypeError: cannot 
pickle 'pyspark.sql.types.StructField' object`.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes:
    - The types give back the user-defined types they are made of, and pickle 
with them, as with PySpark 4.3.0.
    - A `StructField` can be pickled.
    
    ### How was this patch tested?
    
    - A new offline test checks:
      - the user-defined types that `ArrayType`, `MapType` (key and value), 
`StructField` and `StructType` give back, also nested, and their `repr` and 
`simpleString`
      - the round trips of these types with `pickle` and cloudpickle, also of a 
`StructField` with metadata and one without a user-defined type
      - what a type with a user-defined type pickles of
    
      The cases give the same results with official PySpark 4.3.0.
    - A new live test runs UDFs of an array of `ExamplePointUDT` on the server. 
It fails without this change.
    - Apache Spark's `test_parity_udf.py`, `test_parity_serde.py`, 
`test_parity_creation.py` and `test_parity_udf_combinations.py` run on this 
client against a local server.
      - `test_udf_with_udt` now passes, and so do the tests of user-defined 
types in `arrow/test_parity_arrow.py`, `arrow/test_parity_arrow_python_udf.py` 
and `test_parity_dataframe.py`.
      - The one other failure, `test_udf_with_string_return_type`, fails 
without this change too, because the client's `Row` is not a tuple.
    - The lib tests of both crates (590 passed), the build of all tests, also 
with `wasm-udf`, the offline Python suites (2179 passed, 8 skipped), and the 
live drop-in and Arrow suites (175 passed) and live cases (10 passed) against a 
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #321 from HyukjinKwon/SPARK-61128-udt-children.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/types.rs      | 116 +++++++++++++++++++++++++++++-------
 python/tests/test_dropin_e2e.py     |  22 +++++++
 python/tests/test_dropin_offline.py |  46 ++++++++++++++
 3 files changed, 164 insertions(+), 20 deletions(-)

diff --git a/crates/pyspark-rs/src/types.rs b/crates/pyspark-rs/src/types.rs
index c0c198f..6ed29d0 100644
--- a/crates/pyspark-rs/src/types.rs
+++ b/crates/pyspark-rs/src/types.rs
@@ -28,6 +28,36 @@ fn type_reduce(py: Python<'_>, json: &str) -> 
PyResult<(Py<PyAny>, (String,))> {
     Ok((f.unbind(), (json.to_string(),)))
 }
 
+/// The `__reduce__` of a type of `data_type`, whose class is `cls`: by its 
JSON, as
+/// [`type_reduce`], unless it has a user-defined type, whose JSON need not 
import its class.
+/// Such a type pickles of the type objects it is made of, `args()`, as 
PySpark's types do; the
+/// class is that of `pyspark.sql.types`, also on the Python workers of the 
server.
+fn reduce_with_udts<'py>(
+    cls: Bound<'py, pyo3::types::PyType>,
+    data_type: &DataType,
+    args: impl FnOnce() -> PyResult<Bound<'py, pyo3::types::PyTuple>>,
+) -> PyResult<(Py<PyAny>, Py<PyAny>)> {
+    let py = cls.py();
+    if has_udt(data_type) {
+        return Ok((cls.into_any().unbind(), args()?.into_any().unbind()));
+    }
+    let (f, args) = type_reduce(py, &data_type.json())?;
+    Ok((f, args.into_pyobject(py)?.into_any().unbind()))
+}
+
+/// The user-defined types that the client lowered into core types, by the 
JSON of their core
+/// types. PySpark's `ArrayType`, `MapType`, `StructField` and `StructType` 
hold the type
+/// objects they are made of; so a core type gives back the user-defined type 
it was made of,
+/// rather than one of its JSON. The `pyClass` of the JSON need not name the 
module of the class,
+/// as for Apache Spark's `ExamplePointUDT`, of 
`pyspark.sql.tests.ExamplePointUDT`.
+static LOWERED_UDTS: PyOnceLock<Py<pyo3::types::PyDict>> = PyOnceLock::new();
+
+fn lowered_udts(py: Python<'_>) -> &Bound<'_, pyo3::types::PyDict> {
+    LOWERED_UDTS
+        .get_or_init(py, || pyo3::types::PyDict::new(py).unbind())
+        .bind(py)
+}
+
 /// Python wrapper for any DataType. This is the base of the type-class 
hierarchy
 /// (mirroring pyspark's `DataType`); concrete type classes extend it 
(directly or via
 /// the intermediate abstract bases below), so `isinstance(dt, DataType)` and 
the object
@@ -1434,12 +1464,17 @@ impl PyArrayType {
     ) -> PyResult<Bound<'py, PyAny>> {
         shim_conversion(slf.as_any(), "_to_internal", obj)
     }
-    fn __reduce__(&self, py: Python<'_>) -> PyResult<(Py<PyAny>, (String,))> {
+    fn __reduce__(slf: &Bound<'_, Self>) -> PyResult<(Py<PyAny>, Py<PyAny>)> {
+        let py = slf.py();
+        let this = slf.borrow();
         let dt = DataType::Array {
-            element_type: Box::new(self.element_type.clone()),
-            contains_null: self.contains_null,
+            element_type: Box::new(this.element_type.clone()),
+            contains_null: this.contains_null,
         };
-        type_reduce(py, &dt.json())
+        reduce_with_udts(slf.get_type(), &dt, || {
+            let element_type = data_type_to_py(py, &this.element_type)?;
+            (element_type, this.contains_null).into_pyobject(py)
+        })
     }
     // Kwargs mirror the reference: ArrayType(elementType, containsNull=True).
     #[new]
@@ -1565,13 +1600,19 @@ impl PyMapType {
     ) -> PyResult<Bound<'py, PyAny>> {
         shim_conversion(slf.as_any(), "_to_internal", obj)
     }
-    fn __reduce__(&self, py: Python<'_>) -> PyResult<(Py<PyAny>, (String,))> {
+    fn __reduce__(slf: &Bound<'_, Self>) -> PyResult<(Py<PyAny>, Py<PyAny>)> {
+        let py = slf.py();
+        let this = slf.borrow();
         let dt = DataType::Map {
-            key_type: Box::new(self.key_type.clone()),
-            value_type: Box::new(self.value_type.clone()),
-            value_contains_null: self.value_contains_null,
+            key_type: Box::new(this.key_type.clone()),
+            value_type: Box::new(this.value_type.clone()),
+            value_contains_null: this.value_contains_null,
         };
-        type_reduce(py, &dt.json())
+        reduce_with_udts(slf.get_type(), &dt, || {
+            let key_type = data_type_to_py(py, &this.key_type)?;
+            let value_type = data_type_to_py(py, &this.value_type)?;
+            (key_type, value_type, this.value_contains_null).into_pyobject(py)
+        })
     }
     // Kwargs mirror the reference: MapType(keyType, valueType, 
valueContainsNull=True).
     #[new]
@@ -1868,6 +1909,25 @@ impl PyStructField {
     fn __hash__(slf: &Bound<'_, Self>) -> PyResult<isize> {
         slf.str()?.hash()
     }
+    /// A field pickles, as PySpark's does: of its name, type, nullability and 
metadata, with
+    /// the class of `pyspark.sql.types`, also on the Python workers of the 
server.
+    fn __reduce__<'py>(
+        slf: &Bound<'py, Self>,
+    ) -> PyResult<(
+        Bound<'py, pyo3::types::PyType>,
+        Bound<'py, pyo3::types::PyTuple>,
+    )> {
+        let py = slf.py();
+        let this = slf.borrow();
+        let args = (
+            this.field.name.clone(),
+            data_type_to_py(py, &this.field.data_type)?,
+            this.field.nullable,
+            metadata_map_to_py(py, &this.field.metadata)?,
+        )
+            .into_pyobject(py)?;
+        Ok((slf.get_type(), args))
+    }
 }
 
 #[pyclass(name = "StructType", module = "pyspark.sql.types", extends = 
PyDataType)]
@@ -1908,11 +1968,20 @@ impl PyStructType {
     ) -> PyResult<Bound<'py, PyAny>> {
         shim_conversion(slf.as_any(), "_to_internal", obj)
     }
-    fn __reduce__(&self, py: Python<'_>) -> PyResult<(Py<PyAny>, (String,))> {
+    fn __reduce__(slf: &Bound<'_, Self>) -> PyResult<(Py<PyAny>, Py<PyAny>)> {
+        let py = slf.py();
+        let this = slf.borrow();
         let dt = DataType::Struct {
-            fields: self.fields.clone(),
+            fields: this.fields.clone(),
         };
-        type_reduce(py, &dt.json())
+        reduce_with_udts(slf.get_type(), &dt, || {
+            let fields = this
+                .fields
+                .iter()
+                .map(|f| Py::new(py, PyStructField { field: f.clone() }))
+                .collect::<PyResult<Vec<_>>>()?;
+            (fields,).into_pyobject(py)
+        })
     }
     /// `StructType(fields=None)` where each field is a `StructField` (matching
     /// pyspark). A list of DDL field strings is also accepted for convenience.
@@ -3292,12 +3361,17 @@ pub(crate) fn data_type_to_py(py: Python<'_>, dt: 
&DataType) -> PyResult<Py<PyAn
             .call1((data_type_string,))?
             .unbind(),
         DataType::Udt { .. } => {
-            // A UDT reconstructs by re-importing its Python class; delegate 
to the Python
-            // parser.
-            py.import("pyspark.sql.types")?
-                .getattr("_parse_datatype_json_string")?
-                .call1((dt.json(),))?
-                .unbind()
+            // The user-defined type that the client lowered into `dt`; or 
else, as of the
+            // schema of the server, one that re-imports its Python class, of 
the Python parser.
+            let json = dt.json();
+            match lowered_udts(py).get_item(&json)? {
+                Some(udt) => udt.unbind(),
+                None => py
+                    .import("pyspark.sql.types")?
+                    .getattr("_parse_datatype_json_string")?
+                    .call1((json,))?
+                    .unbind(),
+            }
         }
     };
     Ok(o)
@@ -3450,13 +3524,15 @@ pub(crate) fn py_to_data_type(obj: &Bound<'_, PyAny>) 
-> PyResult<DataType> {
                 }
             };
             let sql_type = py_to_data_type(&obj.call_method0("sqlType")?)?;
-            return Ok(DataType::Udt {
+            let udt = DataType::Udt {
                 type_str: "udt".to_string(),
                 jvm_class: get("class")?,
                 python_class: get("pyClass")?,
                 serialized_python_class: get("serializedClass")?,
                 sql_type: Some(Box::new(sql_type)),
-            });
+            };
+            lowered_udts(obj.py()).set_item(udt.json(), obj)?;
+            return Ok(udt);
         }
     }
     // `UnparsedDataType` of `pyspark.sql.connect.types`, a pure-Python 
subclass of the base, is
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 1c3e4b4..bea3e3e 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -2455,6 +2455,28 @@ def test_ml_vector_functions_as_pyspark(session):
     assert tuple(row.first()) == ([1.5, 2.0], [1.5, 2.0])
 
 
+def 
test_udfs_of_types_with_user_defined_types_run_on_the_server_as_pyspark(session):
+    # As PySpark's: a UDF of a return type of an array of ExamplePointUDT, 
whose JSON's pyClass
+    # imports no class, runs on the Python worker of the server, to which the 
type pickles with
+    # the user-defined type it is made of.
+    from pyspark.testing.objects import ExamplePoint, ExamplePointUDT
+
+    @F.udf(returnType=ArrayType(ExamplePointUDT()))
+    def points(d):
+        return [ExamplePoint(d + i, 10 * d + i) for i in range(2)]
+
+    @F.udf(returnType=DoubleType())
+    def last_y(es):
+        return es[-1].y
+
+    df = session.range(1).select(F.lit(1.0).alias("d"))
+    expected = [Row(p=[ExamplePoint(1.0, 10.0), ExamplePoint(2.0, 11.0)])]
+    assert df.select(points("d").alias("p")).collect() == expected
+    assert df.select(last_y(points("d")).alias("y")).collect() == [Row(y=11.0)]
+
+
+
+
 def test_udfs_return_user_defined_types_of_main_as_pyspark():
     # As PySpark's: a UDT that __main__ defines, of parameters, is a UDF 
return type, as
     # cloudpickle sends its class by value and its state to the worker. In a 
new
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index 9ce3d67..0a53e75 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -1747,6 +1747,52 @@ 
sys.stdout.buffer.write(CloudPickleSerializer().dumps(MainPointUDT(2.5)))
     assert main_udt.sqlType() == T.ArrayType(T.DoubleType())
 
 
+def test_types_give_back_and_pickle_their_user_defined_types_as_pyspark():
+    # As PySpark's ArrayType, MapType, StructField and StructType, which hold 
the type objects
+    # they are made of: each gives back its user-defined type, also nested, 
though the pyClass
+    # of the JSON of ExamplePointUDT, pyspark.sql.tests.ExamplePointUDT, 
imports no class. They
+    # pickle with it, and so does a StructField of any type.
+    import pickle
+
+    from pyspark.serializers import CloudPickleSerializer
+    from pyspark.testing.objects import ExamplePointUDT
+
+    udt = ExamplePointUDT()
+    assert udt.jsonValue()["pyClass"] == "pyspark.sql.tests.ExamplePointUDT"
+    nested = T.MapType(T.StringType(), T.StructType([T.StructField("p", udt)]))
+    assert T.ArrayType(udt).elementType is udt
+    assert T.MapType(udt, T.StringType()).keyType is udt
+    assert T.MapType(T.StringType(), udt).valueType is udt
+    assert T.StructField("p", udt).dataType is udt
+    assert T.StructType([T.StructField("p", udt)]).fields[0].dataType is udt
+    assert T.StructType().add("p", udt)["p"].dataType is udt
+    assert T.ArrayType(T.ArrayType(udt)).elementType.elementType is udt
+    assert nested.valueType["p"].dataType is udt
+    assert repr(T.ArrayType(udt)) == "ArrayType(ExamplePointUDT(), True)"
+    assert T.StructType([T.StructField("p", udt)]).simpleString() == 
"struct<p:udt>"
+
+    serializer = CloudPickleSerializer()
+    for t in [
+        T.ArrayType(udt, False),
+        T.MapType(T.StringType(), udt, False),
+        T.StructType([T.StructField("p", udt, False, {"k": "v"})]),
+        nested,
+        T.StructField("p", udt, True, {"k": 1}),
+        T.StructField("i", T.IntegerType(), False, {"k": "v"}),
+        T.StructType([T.StructField("i", T.IntegerType())]),
+    ]:
+        for copied in (pickle.loads(pickle.dumps(t)), 
serializer.loads(serializer.dumps(t))):
+            assert type(copied) is type(t) and copied == t and repr(copied) == 
repr(t)
+    field = pickle.loads(pickle.dumps(T.StructField("p", udt, True, {"k": 1})))
+    assert type(field.dataType) is ExamplePointUDT and field.metadata == {"k": 
1}
+    # A type with a user-defined type pickles of the class of 
pyspark.sql.types and the type
+    # objects, which the Python workers of the server rebuild; one without, by 
its JSON.
+    assert T.ArrayType(udt).__reduce__() == (T.ArrayType, (udt, True))
+    assert T.ArrayType(T.IntegerType()).__reduce__()[0] is 
T._parse_datatype_json_string
+
+
+
+
 def test_collected_values_of_user_defined_types_are_deserialized_as_pyspark():
     # The converters that the client applies to collected rows: the values of 
user-defined
     # types, in arrays, maps and structs too, are deserialized by their UDTs, 
as PySpark's


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to