This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 1ae429f [SPARK-61128][CONNECT][PYTHON] Make ArrayType, MapType,
StructField and StructType give back and pickle the user-defined types they are
made of, and StructField picklable, in the Python client as PySpark does
1ae429f is described below
commit 1ae429f96bda9a5a232bf3355a1b72cc76897ec8
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 06:24:49 2026 +0900
[SPARK-61128][CONNECT][PYTHON] Make ArrayType, MapType, StructField and
StructType give back and pickle the user-defined types they are made of, and
StructField picklable, in the Python client as PySpark does
### What changes were proposed in this pull request?
[SPARK-61128](https://issues.apache.org/jira/browse/SPARK-61128): PySpark's
`ArrayType`, `MapType`, `StructField` and `StructType` hold the type objects
they are made of. So `ArrayType(udt).elementType is udt`, for a user-defined
type `udt`, and a type pickles of those objects. The client's types now do the
same:
- **Giving back the user-defined types.** The client lowers its types into
the Rust core. It now keeps each user-defined type it lowers, keyed by the JSON
of its core type, and gives that object back wherever the type is turned back
into a Python object:
- the children of `ArrayType`, `MapType`, `StructField` and `StructType`,
also nested
- their `repr` and `simpleString`, and the conversions of their values
A user-defined type that the client did not lower, such as one of the
schema of a result, is still rebuilt from its JSON, as PySpark's
`UserDefinedType.fromJson` does.
- **Pickling.** A type with a user-defined type pickles as its class of
`pyspark.sql.types` with its type objects, as PySpark's types do, and the
server's Python workers rebuild it. A type without one still pickles by its
JSON.
- **`StructField` pickles**, of its name, type, nullability and metadata.
Before, it could not be pickled at all.
### Why are the changes needed?
The client rebuilt a user-defined type from its JSON, by importing its
`pyClass`. That name need not be the module of the class: for Apache Spark's
`ExamplePointUDT` it is `pyspark.sql.tests.ExamplePointUDT`, while the class is
in `pyspark.testing.objects`. So on the client, these raised
`PySparkValueError: [UNSUPPORTED_OPERATION] unpickling user defined types is
not supported`:
- `ArrayType(ExamplePointUDT()).elementType`, and the type of
`StructField("p", ExamplePointUDT())`
- their `repr`
- `udf(returnType=ArrayType(ExamplePointUDT()))`, so Apache Spark's
`test_udf_with_udt` failed on this client
Pickled by its JSON, such a type would fail the same way on the server's
Python worker.
`pickle.dumps(StructField("a", IntegerType()))` raised `TypeError: cannot
pickle 'pyspark.sql.types.StructField' object`.
### Does this PR introduce _any_ user-facing change?
Yes:
- The types give back the user-defined types they are made of, and pickle
with them, as with PySpark 4.3.0.
- A `StructField` can be pickled.
### How was this patch tested?
- A new offline test checks:
- the user-defined types that `ArrayType`, `MapType` (key and value),
`StructField` and `StructType` give back, also nested, and their `repr` and
`simpleString`
- the round trips of these types with `pickle` and cloudpickle, also of a
`StructField` with metadata and one without a user-defined type
- what a type with a user-defined type pickles of
The cases give the same results with official PySpark 4.3.0.
- A new live test runs UDFs of an array of `ExamplePointUDT` on the server.
It fails without this change.
- Apache Spark's `test_parity_udf.py`, `test_parity_serde.py`,
`test_parity_creation.py` and `test_parity_udf_combinations.py` run on this
client against a local server.
- `test_udf_with_udt` now passes, and so do the tests of user-defined
types in `arrow/test_parity_arrow.py`, `arrow/test_parity_arrow_python_udf.py`
and `test_parity_dataframe.py`.
- The one other failure, `test_udf_with_string_return_type`, fails
without this change too, because the client's `Row` is not a tuple.
- The lib tests of both crates (590 passed), the build of all tests, also
with `wasm-udf`, the offline Python suites (2179 passed, 8 skipped), and the
live drop-in and Arrow suites (175 passed) and live cases (10 passed) against a
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #321 from HyukjinKwon/SPARK-61128-udt-children.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/types.rs | 116 +++++++++++++++++++++++++++++-------
python/tests/test_dropin_e2e.py | 22 +++++++
python/tests/test_dropin_offline.py | 46 ++++++++++++++
3 files changed, 164 insertions(+), 20 deletions(-)
diff --git a/crates/pyspark-rs/src/types.rs b/crates/pyspark-rs/src/types.rs
index c0c198f..6ed29d0 100644
--- a/crates/pyspark-rs/src/types.rs
+++ b/crates/pyspark-rs/src/types.rs
@@ -28,6 +28,36 @@ fn type_reduce(py: Python<'_>, json: &str) ->
PyResult<(Py<PyAny>, (String,))> {
Ok((f.unbind(), (json.to_string(),)))
}
+/// The `__reduce__` of a type of `data_type`, whose class is `cls`: by its
JSON, as
+/// [`type_reduce`], unless it has a user-defined type, whose JSON need not
import its class.
+/// Such a type pickles of the type objects it is made of, `args()`, as
PySpark's types do; the
+/// class is that of `pyspark.sql.types`, also on the Python workers of the
server.
+fn reduce_with_udts<'py>(
+ cls: Bound<'py, pyo3::types::PyType>,
+ data_type: &DataType,
+ args: impl FnOnce() -> PyResult<Bound<'py, pyo3::types::PyTuple>>,
+) -> PyResult<(Py<PyAny>, Py<PyAny>)> {
+ let py = cls.py();
+ if has_udt(data_type) {
+ return Ok((cls.into_any().unbind(), args()?.into_any().unbind()));
+ }
+ let (f, args) = type_reduce(py, &data_type.json())?;
+ Ok((f, args.into_pyobject(py)?.into_any().unbind()))
+}
+
+/// The user-defined types that the client lowered into core types, by the
JSON of their core
+/// types. PySpark's `ArrayType`, `MapType`, `StructField` and `StructType`
hold the type
+/// objects they are made of; so a core type gives back the user-defined type
it was made of,
+/// rather than one of its JSON. The `pyClass` of the JSON need not name the
module of the class,
+/// as for Apache Spark's `ExamplePointUDT`, of
`pyspark.sql.tests.ExamplePointUDT`.
+static LOWERED_UDTS: PyOnceLock<Py<pyo3::types::PyDict>> = PyOnceLock::new();
+
+fn lowered_udts(py: Python<'_>) -> &Bound<'_, pyo3::types::PyDict> {
+ LOWERED_UDTS
+ .get_or_init(py, || pyo3::types::PyDict::new(py).unbind())
+ .bind(py)
+}
+
/// Python wrapper for any DataType. This is the base of the type-class
hierarchy
/// (mirroring pyspark's `DataType`); concrete type classes extend it
(directly or via
/// the intermediate abstract bases below), so `isinstance(dt, DataType)` and
the object
@@ -1434,12 +1464,17 @@ impl PyArrayType {
) -> PyResult<Bound<'py, PyAny>> {
shim_conversion(slf.as_any(), "_to_internal", obj)
}
- fn __reduce__(&self, py: Python<'_>) -> PyResult<(Py<PyAny>, (String,))> {
+ fn __reduce__(slf: &Bound<'_, Self>) -> PyResult<(Py<PyAny>, Py<PyAny>)> {
+ let py = slf.py();
+ let this = slf.borrow();
let dt = DataType::Array {
- element_type: Box::new(self.element_type.clone()),
- contains_null: self.contains_null,
+ element_type: Box::new(this.element_type.clone()),
+ contains_null: this.contains_null,
};
- type_reduce(py, &dt.json())
+ reduce_with_udts(slf.get_type(), &dt, || {
+ let element_type = data_type_to_py(py, &this.element_type)?;
+ (element_type, this.contains_null).into_pyobject(py)
+ })
}
// Kwargs mirror the reference: ArrayType(elementType, containsNull=True).
#[new]
@@ -1565,13 +1600,19 @@ impl PyMapType {
) -> PyResult<Bound<'py, PyAny>> {
shim_conversion(slf.as_any(), "_to_internal", obj)
}
- fn __reduce__(&self, py: Python<'_>) -> PyResult<(Py<PyAny>, (String,))> {
+ fn __reduce__(slf: &Bound<'_, Self>) -> PyResult<(Py<PyAny>, Py<PyAny>)> {
+ let py = slf.py();
+ let this = slf.borrow();
let dt = DataType::Map {
- key_type: Box::new(self.key_type.clone()),
- value_type: Box::new(self.value_type.clone()),
- value_contains_null: self.value_contains_null,
+ key_type: Box::new(this.key_type.clone()),
+ value_type: Box::new(this.value_type.clone()),
+ value_contains_null: this.value_contains_null,
};
- type_reduce(py, &dt.json())
+ reduce_with_udts(slf.get_type(), &dt, || {
+ let key_type = data_type_to_py(py, &this.key_type)?;
+ let value_type = data_type_to_py(py, &this.value_type)?;
+ (key_type, value_type, this.value_contains_null).into_pyobject(py)
+ })
}
// Kwargs mirror the reference: MapType(keyType, valueType,
valueContainsNull=True).
#[new]
@@ -1868,6 +1909,25 @@ impl PyStructField {
fn __hash__(slf: &Bound<'_, Self>) -> PyResult<isize> {
slf.str()?.hash()
}
+ /// A field pickles, as PySpark's does: of its name, type, nullability and
metadata, with
+ /// the class of `pyspark.sql.types`, also on the Python workers of the
server.
+ fn __reduce__<'py>(
+ slf: &Bound<'py, Self>,
+ ) -> PyResult<(
+ Bound<'py, pyo3::types::PyType>,
+ Bound<'py, pyo3::types::PyTuple>,
+ )> {
+ let py = slf.py();
+ let this = slf.borrow();
+ let args = (
+ this.field.name.clone(),
+ data_type_to_py(py, &this.field.data_type)?,
+ this.field.nullable,
+ metadata_map_to_py(py, &this.field.metadata)?,
+ )
+ .into_pyobject(py)?;
+ Ok((slf.get_type(), args))
+ }
}
#[pyclass(name = "StructType", module = "pyspark.sql.types", extends =
PyDataType)]
@@ -1908,11 +1968,20 @@ impl PyStructType {
) -> PyResult<Bound<'py, PyAny>> {
shim_conversion(slf.as_any(), "_to_internal", obj)
}
- fn __reduce__(&self, py: Python<'_>) -> PyResult<(Py<PyAny>, (String,))> {
+ fn __reduce__(slf: &Bound<'_, Self>) -> PyResult<(Py<PyAny>, Py<PyAny>)> {
+ let py = slf.py();
+ let this = slf.borrow();
let dt = DataType::Struct {
- fields: self.fields.clone(),
+ fields: this.fields.clone(),
};
- type_reduce(py, &dt.json())
+ reduce_with_udts(slf.get_type(), &dt, || {
+ let fields = this
+ .fields
+ .iter()
+ .map(|f| Py::new(py, PyStructField { field: f.clone() }))
+ .collect::<PyResult<Vec<_>>>()?;
+ (fields,).into_pyobject(py)
+ })
}
/// `StructType(fields=None)` where each field is a `StructField` (matching
/// pyspark). A list of DDL field strings is also accepted for convenience.
@@ -3292,12 +3361,17 @@ pub(crate) fn data_type_to_py(py: Python<'_>, dt:
&DataType) -> PyResult<Py<PyAn
.call1((data_type_string,))?
.unbind(),
DataType::Udt { .. } => {
- // A UDT reconstructs by re-importing its Python class; delegate
to the Python
- // parser.
- py.import("pyspark.sql.types")?
- .getattr("_parse_datatype_json_string")?
- .call1((dt.json(),))?
- .unbind()
+ // The user-defined type that the client lowered into `dt`; or
else, as of the
+ // schema of the server, one that re-imports its Python class, of
the Python parser.
+ let json = dt.json();
+ match lowered_udts(py).get_item(&json)? {
+ Some(udt) => udt.unbind(),
+ None => py
+ .import("pyspark.sql.types")?
+ .getattr("_parse_datatype_json_string")?
+ .call1((json,))?
+ .unbind(),
+ }
}
};
Ok(o)
@@ -3450,13 +3524,15 @@ pub(crate) fn py_to_data_type(obj: &Bound<'_, PyAny>)
-> PyResult<DataType> {
}
};
let sql_type = py_to_data_type(&obj.call_method0("sqlType")?)?;
- return Ok(DataType::Udt {
+ let udt = DataType::Udt {
type_str: "udt".to_string(),
jvm_class: get("class")?,
python_class: get("pyClass")?,
serialized_python_class: get("serializedClass")?,
sql_type: Some(Box::new(sql_type)),
- });
+ };
+ lowered_udts(obj.py()).set_item(udt.json(), obj)?;
+ return Ok(udt);
}
}
// `UnparsedDataType` of `pyspark.sql.connect.types`, a pure-Python
subclass of the base, is
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 1c3e4b4..bea3e3e 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -2455,6 +2455,28 @@ def test_ml_vector_functions_as_pyspark(session):
assert tuple(row.first()) == ([1.5, 2.0], [1.5, 2.0])
+def
test_udfs_of_types_with_user_defined_types_run_on_the_server_as_pyspark(session):
+ # As PySpark's: a UDF of a return type of an array of ExamplePointUDT,
whose JSON's pyClass
+ # imports no class, runs on the Python worker of the server, to which the
type pickles with
+ # the user-defined type it is made of.
+ from pyspark.testing.objects import ExamplePoint, ExamplePointUDT
+
+ @F.udf(returnType=ArrayType(ExamplePointUDT()))
+ def points(d):
+ return [ExamplePoint(d + i, 10 * d + i) for i in range(2)]
+
+ @F.udf(returnType=DoubleType())
+ def last_y(es):
+ return es[-1].y
+
+ df = session.range(1).select(F.lit(1.0).alias("d"))
+ expected = [Row(p=[ExamplePoint(1.0, 10.0), ExamplePoint(2.0, 11.0)])]
+ assert df.select(points("d").alias("p")).collect() == expected
+ assert df.select(last_y(points("d")).alias("y")).collect() == [Row(y=11.0)]
+
+
+
+
def test_udfs_return_user_defined_types_of_main_as_pyspark():
# As PySpark's: a UDT that __main__ defines, of parameters, is a UDF
return type, as
# cloudpickle sends its class by value and its state to the worker. In a
new
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index 9ce3d67..0a53e75 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -1747,6 +1747,52 @@
sys.stdout.buffer.write(CloudPickleSerializer().dumps(MainPointUDT(2.5)))
assert main_udt.sqlType() == T.ArrayType(T.DoubleType())
+def test_types_give_back_and_pickle_their_user_defined_types_as_pyspark():
+ # As PySpark's ArrayType, MapType, StructField and StructType, which hold
the type objects
+ # they are made of: each gives back its user-defined type, also nested,
though the pyClass
+ # of the JSON of ExamplePointUDT, pyspark.sql.tests.ExamplePointUDT,
imports no class. They
+ # pickle with it, and so does a StructField of any type.
+ import pickle
+
+ from pyspark.serializers import CloudPickleSerializer
+ from pyspark.testing.objects import ExamplePointUDT
+
+ udt = ExamplePointUDT()
+ assert udt.jsonValue()["pyClass"] == "pyspark.sql.tests.ExamplePointUDT"
+ nested = T.MapType(T.StringType(), T.StructType([T.StructField("p", udt)]))
+ assert T.ArrayType(udt).elementType is udt
+ assert T.MapType(udt, T.StringType()).keyType is udt
+ assert T.MapType(T.StringType(), udt).valueType is udt
+ assert T.StructField("p", udt).dataType is udt
+ assert T.StructType([T.StructField("p", udt)]).fields[0].dataType is udt
+ assert T.StructType().add("p", udt)["p"].dataType is udt
+ assert T.ArrayType(T.ArrayType(udt)).elementType.elementType is udt
+ assert nested.valueType["p"].dataType is udt
+ assert repr(T.ArrayType(udt)) == "ArrayType(ExamplePointUDT(), True)"
+ assert T.StructType([T.StructField("p", udt)]).simpleString() ==
"struct<p:udt>"
+
+ serializer = CloudPickleSerializer()
+ for t in [
+ T.ArrayType(udt, False),
+ T.MapType(T.StringType(), udt, False),
+ T.StructType([T.StructField("p", udt, False, {"k": "v"})]),
+ nested,
+ T.StructField("p", udt, True, {"k": 1}),
+ T.StructField("i", T.IntegerType(), False, {"k": "v"}),
+ T.StructType([T.StructField("i", T.IntegerType())]),
+ ]:
+ for copied in (pickle.loads(pickle.dumps(t)),
serializer.loads(serializer.dumps(t))):
+ assert type(copied) is type(t) and copied == t and repr(copied) ==
repr(t)
+ field = pickle.loads(pickle.dumps(T.StructField("p", udt, True, {"k": 1})))
+ assert type(field.dataType) is ExamplePointUDT and field.metadata == {"k":
1}
+ # A type with a user-defined type pickles of the class of
pyspark.sql.types and the type
+ # objects, which the Python workers of the server rebuild; one without, by
its JSON.
+ assert T.ArrayType(udt).__reduce__() == (T.ArrayType, (udt, True))
+ assert T.ArrayType(T.IntegerType()).__reduce__()[0] is
T._parse_datatype_json_string
+
+
+
+
def test_collected_values_of_user_defined_types_are_deserialized_as_pyspark():
# The converters that the client applies to collected rows: the values of
user-defined
# types, in arrays, maps and structs too, are deserialized by their UDTs,
as PySpark's
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]