This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new f9438ab [SPARK-60962][CONNECT][PYTHON] Pickle user-defined types as
Python objects, and make them of arguments, as PySpark does
f9438ab is described below
commit f9438abb721267237753a84ffbb04e453eafc47d
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Fri Nov 6 23:00:43 2026 +0900
[SPARK-60962][CONNECT][PYTHON] Pickle user-defined types as Python objects,
and make them of arguments, as PySpark does
### What changes were proposed in this pull request?
[SPARK-60962](https://issues.apache.org/jira/browse/SPARK-60962):
user-defined types now pickle as Python objects, and are made with the
arguments of their `__init__`, as PySpark's do.
- As PySpark's `UserDefinedType`, a Python object, a UDT pickles as its
class and its state. cloudpickle sends a class of `__main__` by value, and the
state keeps the parameters of the instance. It pickled as its JSON, which names
the class by its module, and is rebuilt without arguments.
- As PySpark's `DataType`, whose `__new__` is `object.__new__`, a subclass
of `DataType`, such as a UDT, is made with the arguments of its `__init__`.
### Why are the changes needed?
A UDT that `__main__` defines, as in a notebook or a script, failed as the
return type of a UDF: the worker rebuilt it from its JSON, by importing its
module, which `__main__` on the worker does not define. A UDT whose `__init__`
keeps parameters was rebuilt without them. Even making one failed:
`PointUDT(10.0)` raised `TypeError: DataType.__new__() takes 0 positional
arguments but 1 was given`.
### Does this PR introduce _any_ user-facing change?
Yes. A UDT of `__main__`, and of parameters, is a UDF return type, as in
PySpark 4.3.0.
### How was this patch tested?
- A new live test runs, in a new interpreter, a UDF whose return type is a
UDT that `__main__` defines with a parameter, and collects its values. It also
passes with official PySpark 4.3.0, and fails without this change.
- A new offline test checks that a UDT of parameters pickles and copies
with them. It also loads, here, a UDT that another interpreter's `__main__`
defined and pickled. It fails without this change.
- The lib tests and every integration test, e2e included, against a local
Spark 4.3.0 server (664 passed, and 125 of the transport crate), the build of
all tests with `wasm-udf`, the offline Python suites (1328 passed, 8 skipped),
and the live drop-in and Arrow suites (87 passed) and live cases (10 passed).
Clippy reports no diagnostics in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #212 from HyukjinKwon/SPARK-60962-udt-pickle.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/types.rs | 8 ++++++-
python/pyspark/sql/types.py | 9 +++++++
python/tests/test_dropin_e2e.py | 48 +++++++++++++++++++++++++++++++++++++
python/tests/test_dropin_offline.py | 45 ++++++++++++++++++++++++++++++++++
4 files changed, 109 insertions(+), 1 deletion(-)
diff --git a/crates/pyspark-rs/src/types.rs b/crates/pyspark-rs/src/types.rs
index 0c44a6c..78125cf 100644
--- a/crates/pyspark-rs/src/types.rs
+++ b/crates/pyspark-rs/src/types.rs
@@ -150,8 +150,14 @@ impl PyDataType {
/// (e.g. a user's `UserDefinedType`). The sentinel `inner` is never
consulted for
/// such subclasses — they override the object-model methods in Python, and
/// `py_to_data_type` special-cases `UserDefinedType` before reading
`inner`.
+ /// As `object.__new__` of PySpark's pure-Python `DataType`: the arguments
are those of the
+ /// `__init__` of a subclass, such as a `UserDefinedType` of parameters.
#[new]
- fn __new__() -> Self {
+ #[pyo3(signature = (*_args, **_kwargs))]
+ fn __new__(
+ _args: &Bound<'_, pyo3::types::PyTuple>,
+ _kwargs: Option<&Bound<'_, pyo3::types::PyDict>>,
+ ) -> Self {
PyDataType {
inner: DataType::Null,
}
diff --git a/python/pyspark/sql/types.py b/python/pyspark/sql/types.py
index 43abd15..71b1c3f 100644
--- a/python/pyspark/sql/types.py
+++ b/python/pyspark/sql/types.py
@@ -120,6 +120,15 @@ class UserDefinedType(DataType):
def typeName(cls) -> str:
return cls.__name__.lower()
+ def __reduce__(self):
+ # As PySpark's UDT, a Python object, which pickles as its class and
its state:
+ # cloudpickle sends a class of __main__ by value, and the state keeps
the parameters
+ # of the instance. DataType.__reduce__ would send its JSON, which
names the class by
+ # its module and rebuilds the instance without arguments.
+ import copyreg
+
+ return copyreg.__newobj__, (type(self),), self.__dict__ or None
+
@classmethod
def sqlType(cls) -> "DataType":
"""Underlying SQL storage type for this UDT."""
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index ed08be5..5eeee7c 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -1036,6 +1036,54 @@ def test_ml_vector_functions_as_pyspark(session):
assert tuple(row.first()) == ([1.5, 2.0], [1.5, 2.0])
+def test_udfs_return_user_defined_types_of_main_as_pyspark():
+ # As PySpark's: a UDT that __main__ defines, of parameters, is a UDF
return type, as
+ # cloudpickle sends its class by value and its state to the worker. In a
new
+ # interpreter, whose __main__ defines the classes.
+ import subprocess
+ import sys
+
+ remote = os.environ.get("SPARK_REMOTE")
+ if not remote:
+ pytest.skip("SPARK_REMOTE is required for live-server tests")
+ subprocess.run([sys.executable, "-c", """
+import sys
+from pyspark.sql import Row, SparkSession, functions as F
+from pyspark.sql.types import ArrayType, DoubleType, UserDefinedType
+
+class Point:
+ def __init__(self, x, y):
+ self.x, self.y = x, y
+
+ def __eq__(self, other):
+ return isinstance(other, Point) and (self.x, self.y) == (other.x,
other.y)
+
+class PointUDT(UserDefinedType):
+ def __init__(self, scale=1.0):
+ self.scale = scale
+
+ @classmethod
+ def sqlType(cls):
+ return ArrayType(DoubleType(), False)
+
+ @classmethod
+ def module(cls):
+ return "__main__"
+
+ def serialize(self, obj):
+ return [obj.x * self.scale, obj.y * self.scale]
+
+ def deserialize(self, datum):
+ return Point(datum[0], datum[1])
+
+Point.__UDT__ = PointUDT()
+spark = SparkSession.builder.remote(sys.argv[1]).getOrCreate()
+df = spark.range(2).select(F.udf(lambda i: Point(i, i + 1),
PointUDT(10.0))("id").alias("p"))
+assert df.collect() == [Row(p=Point(0.0, 10.0)), Row(p=Point(10.0, 20.0))]
+assert df.select(F.to_json(F.struct("p"))).collect()[1][0] ==
'{"p":[10.0,20.0]}'
+""", remote], check=True)
+
+
def
test_collected_values_of_user_defined_types_are_deserialized_as_pyspark(session):
# As official PySpark 4.3.0: a collected value of a UDT is its
deserialized object, and
# dtypes names the UDT by its simpleString(), in every way that rows are
collected.
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index 3417e6a..ce58183 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -986,6 +986,51 @@ class _DayPointUDT(T.UserDefinedType):
return tuple(datum)
+class _ScaledPointUDT(_DayPointUDT):
+ def __init__(self, scale=1.0):
+ self.scale = scale
+
+
+def test_user_defined_types_pickle_as_python_objects_as_pyspark():
+ # As PySpark's UDT, a Python object: a UDT of parameters is made with
them, and pickles
+ # as its class and its state, which cloudpickle sends by value for a class
of __main__.
+ # In a new interpreter, whose __main__ defines the class, so that this one
does not.
+ import copy
+ import pickle
+ import subprocess
+ import sys
+
+ from pyspark.serializers import CloudPickleSerializer
+
+ udt = _ScaledPointUDT(3.0)
+ for copied in (pickle.loads(pickle.dumps(udt)), copy.deepcopy(udt)):
+ assert type(copied) is _ScaledPointUDT and copied.scale == 3.0
+ assert pickle.loads(pickle.dumps(_DayPointUDT())).__dict__ == {}
+
+ pickled = subprocess.run([sys.executable, "-c", """
+import sys
+from pyspark.serializers import CloudPickleSerializer
+from pyspark.sql.types import ArrayType, DoubleType, UserDefinedType
+
+class MainPointUDT(UserDefinedType):
+ def __init__(self, scale):
+ self.scale = scale
+
+ @classmethod
+ def sqlType(cls):
+ return ArrayType(DoubleType())
+
+ @classmethod
+ def module(cls):
+ return "__main__"
+
+sys.stdout.buffer.write(CloudPickleSerializer().dumps(MainPointUDT(2.5)))
+"""], check=True, capture_output=True).stdout
+ main_udt = CloudPickleSerializer().loads(pickled)
+ assert type(main_udt).__name__ == "MainPointUDT" and main_udt.scale == 2.5
+ assert main_udt.sqlType() == T.ArrayType(T.DoubleType())
+
+
def test_collected_values_of_user_defined_types_are_deserialized_as_pyspark():
# The converters that the client applies to collected rows: the values of
user-defined
# types, in arrays, maps and structs too, are deserialized by their UDTs,
as PySpark's
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]