This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 32cfcb7 [SPARK-60918][CONNECT] Make == and hash() of Spark Connect
Rust data types match PySpark
32cfcb7 is described below
commit 32cfcb701cc62616d2f143e9ae135aff5d9d162c
Author: Liang-Chi Hsieh <[email protected]>
AuthorDate: Thu Nov 5 18:37:09 2026 +0900
[SPARK-60918][CONNECT] Make == and hash() of Spark Connect Rust data types
match PySpark
### What changes were proposed in this pull request?
In the Rust-backed Python drop-in, the data type classes (`DataType` and
every concrete type such as `LongType`, `StructType`) defined no
`__eq__`/`__hash__`, so `==` fell back to object identity. This PR defines
`__eq__`, `__ne__` and `__hash__` once on the `DataType` base in
`crates/pyspark-rs/src/types.rs`, and every concrete type inherits them:
- `==` compares the core `DataType` built from each side's live fields via
`py_to_data_type`. A `StructType` mutated with `add()` compares by its current
fields, and the bare `DataType` returned by `fromDDL` equals the matching
concrete class.
- `UserDefinedType` is a pure-Python subclass whose state lives in
`__dict__`, so it keeps PySpark's rule verbatim: `isinstance(other,
self.__class__) and self.__dict__ == other.__dict__`.
- `hash()` hashes the simple string of the same live core type that `==`
compares, so equal types always hash equal. PySpark uses `hash(str(self))`, but
here `str()` of a mutated `StructType` still renders the fields it was
constructed with (`StructType` has no `__str__` and falls back to the base's
snapshot), which would break the hash/eq contract. A UDT keeps
`hash(str(self))`.
- `StructField` defined `__eq__` without `__hash__`, which made it
unhashable. It now has a `__hash__` like PySpark's.
### Why are the changes needed?
To match PySpark. Before this change:
```python
>>> from pyspark.sql.types import *
>>> LongType() == LongType()
False
>>> IntegerType() == DataType.fromDDL("int")
False
>>> hash(StructField("a", LongType()))
TypeError: unhashable type: 'builtins.StructField'
```
PySpark returns `True`, `True` and a hash value.
### Does this PR introduce _any_ user-facing change?
Yes. Data types now compare and hash like PySpark's, so they work as set
members and dict keys, and schema comparisons such as `df.schema == expected`
compare by value.
One known difference remains: in PySpark, a bare `DataType()` equals any
concrete type (`DataType() == LongType()` is `True`, because both `__dict__`s
are empty). Here it is `False`, because the bare `DataType` also stands for the
concrete types that `fromDDL` returns.
### How was this patch tested?
- New offline tests in `python/tests/test_dropin_offline.py`: pairwise
`==`/`!=`/`hash` across 42 types (atomic and parameterized types, intervals,
spatial, nested array/map/struct, field nullability and metadata), equality
across construction paths (`fromDDL`, `_parse_datatype_json_string`, pickle,
`add()`), non-DataType operands, a mutated `StructType`, set/dict membership,
`StructField`, and UDTs.
- The same tests pass against the official PySpark v4.3.0
`pyspark.sql.types`, except the `fromDDL` assertions, which need a live session
in PySpark.
- A probe script compared 4,481 eq/ne/hash results between the drop-in and
official PySpark v4.3.0. All match after this change; 161 differed before it.
- `cargo fmt --all --check`, `cargo test -p apache-spark-connect --release
--lib`, and `cargo test -p apache-spark-connect --release [--features wasm-udf]
--no-run` pass, and clippy reports nothing in the changed file.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
Closes #156 from viirya/SPARK-60918.
Authored-by: Liang-Chi Hsieh <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/types.rs | 63 ++++++++++++++++++++
python/tests/test_dropin_offline.py | 114 ++++++++++++++++++++++++++++++++++++
2 files changed, 177 insertions(+)
diff --git a/crates/pyspark-rs/src/types.rs b/crates/pyspark-rs/src/types.rs
index 935fc72..983c141 100644
--- a/crates/pyspark-rs/src/types.rs
+++ b/crates/pyspark-rs/src/types.rs
@@ -200,6 +200,64 @@ impl PyDataType {
fn simpleString(&self) -> String {
self.inner.simple_string()
}
+
+ // Equality/hashing are defined once here and inherited by every concrete
type class
+ // (and by pure-Python subclasses such as `UserDefinedType`). Mirrors
pyspark's
+ // `DataType.__eq__`/`__ne__`/`__hash__`.
+ fn __eq__(slf: &Bound<'_, Self>, other: &Bound<'_, PyAny>) ->
PyResult<bool> {
+ data_type_eq(slf.as_any(), other)
+ }
+
+ fn __ne__(slf: &Bound<'_, Self>, other: &Bound<'_, PyAny>) ->
PyResult<bool> {
+ Ok(!data_type_eq(slf.as_any(), other)?)
+ }
+
+ /// pyspark hashes `str(self)`. Hash the simple string of the same live
core type that
+ /// `__eq__` compares, so equal types always hash equal (`str()` can lag
behind the
+ /// live fields of a mutated `StructType`). A `UserDefinedType` is
compared by class
+ /// and `__dict__` rather than by core type, so it keeps pyspark's
`hash(str(self))`.
+ fn __hash__(slf: &Bound<'_, Self>) -> PyResult<isize> {
+ let any = slf.as_any();
+ if is_udt(any)? {
+ return any.str()?.hash();
+ }
+ let s = py_to_data_type(any)?.simple_string();
+ pyo3::types::PyString::new(any.py(), &s).hash()
+ }
+}
+
+/// Whether `obj` is an instance of the pure-Python
`pyspark.sql.types.UserDefinedType`.
+fn is_udt(obj: &Bound<'_, PyAny>) -> PyResult<bool> {
+ match obj
+ .py()
+ .import("pyspark.sql.types")
+ .and_then(|m| m.getattr("UserDefinedType"))
+ {
+ Ok(cls) => obj.is_instance(&cls),
+ Err(_) => Ok(false),
+ }
+}
+
+/// `DataType.__eq__` with pyspark semantics. pyspark compares
+/// `isinstance(other, self.__class__) and self.__dict__ == other.__dict__`;
the Rust-backed
+/// classes keep their state in Rust rather than `__dict__`, so compare the
core `DataType`
+/// built from each side's live fields instead (a mutated `StructType`
compares by its
+/// current fields, and a bare `DataType` returned by `fromDDL` equals the
concrete class).
+/// `UserDefinedType` is a pure-Python subclass whose state does live in
`__dict__`, so it
+/// keeps pyspark's formula verbatim.
+fn data_type_eq(a: &Bound<'_, PyAny>, b: &Bound<'_, PyAny>) -> PyResult<bool> {
+ if !b.is_instance_of::<PyDataType>() {
+ return Ok(false);
+ }
+ let a_udt = is_udt(a)?;
+ if a_udt || is_udt(b)? {
+ // A non-UDT `self` is never a superclass of a UDT other than the bare
base,
+ // whose (empty) state pyspark would compare; treat that as unequal
too.
+ return Ok(a_udt
+ && b.is_instance(&a.get_type())?
+ && a.getattr("__dict__")?.eq(b.getattr("__dict__")?)?);
+ }
+ Ok(py_to_data_type(a)? == py_to_data_type(b)?)
}
// Define concrete type classes
@@ -1586,6 +1644,11 @@ impl PyStructField {
false
}
}
+ /// pyspark: `hash(str(self))` (inherited from `DataType`). Defining
`__eq__` alone
+ /// would leave StructField unhashable.
+ fn __hash__(slf: &Bound<'_, Self>) -> PyResult<isize> {
+ slf.str()?.hash()
+ }
}
#[pyclass(name = "StructType", module = "pyspark.sql.types", extends =
PyDataType)]
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index e2a2627..350e012 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -106,6 +106,120 @@ def test_structfield_construction():
assert "n" in st.simpleString()
+# ---------------------------------------------------------------------------
type equality
+# Expected results are what PySpark 4.3.0 returns (`DataType.__eq__` is
+# `isinstance(other, self.__class__) and self.__dict__ == other.__dict__`,
`__hash__` is
+# `hash(str(self))`).
+
+def _eq_cases():
+ F = T.StructField
+ return [
+ T.NullType, T.BooleanType, T.ByteType, T.ShortType, T.IntegerType,
T.LongType,
+ T.FloatType, T.DoubleType, T.StringType, lambda:
T.StringType("UTF8_LCASE"),
+ T.BinaryType, T.DateType, T.TimestampType, T.TimestampNTZType,
T.VariantType,
+ T.CalendarIntervalType, T.DecimalType, lambda: T.DecimalType(10, 2),
+ lambda: T.DecimalType(12, 2), lambda: T.CharType(5), lambda:
T.CharType(6),
+ lambda: T.VarcharType(5), lambda: T.TimeType(3), lambda: T.TimeType(6),
+ lambda: T.YearMonthIntervalType(0, 1), lambda:
T.YearMonthIntervalType(0, 0),
+ lambda: T.DayTimeIntervalType(0, 3), lambda: T.DayTimeIntervalType(1,
3),
+ lambda: T.GeometryType(0), lambda: T.GeometryType(4326),
+ lambda: T.GeographyType(4326),
+ lambda: T.ArrayType(T.IntegerType()), lambda:
T.ArrayType(T.IntegerType(), False),
+ lambda: T.ArrayType(T.LongType()),
+ lambda: T.MapType(T.StringType(), T.IntegerType()),
+ lambda: T.MapType(T.StringType(), T.IntegerType(), False),
+ lambda: T.StructType([F("a", T.IntegerType())]),
+ lambda: T.StructType([F("a", T.IntegerType(), False)]),
+ lambda: T.StructType([F("b", T.IntegerType())]),
+ lambda: T.StructType([F("a", T.IntegerType(), True, {"k": "v"})]),
+ lambda: T.StructType(),
+ lambda: T.StructType([
+ F("x", T.ArrayType(T.MapType(T.StringType(), T.DecimalType(10,
2))))
+ ]),
+ ]
+
+
+def test_datatype_eq_and_hash_match_pyspark():
+ cases = _eq_cases()
+ left = [f() for f in cases]
+ right = [f() for f in cases]
+ for i, a in enumerate(left):
+ for j, b in enumerate(right):
+ # Every case is a distinct type, so equal exactly on the diagonal.
+ assert (a == b) is (i == j), (a, b)
+ assert (a != b) is (i != j), (a, b)
+ assert hash(a) == hash(right[i]), a
+ assert type(T.LongType()).__eq__ is not object.__eq__
+
+
+def test_datatype_eq_across_construction_paths():
+ import json
+
+ st = T.StructType([T.StructField("a", T.IntegerType())])
+ assert T.IntegerType() == T.DataType.fromDDL("int")
+ assert T.DataType.fromDDL("int") == T.IntegerType()
+ assert hash(T.DataType.fromDDL("int")) == hash(T.IntegerType())
+ assert T.DataType.fromDDL("bigint") != T.IntegerType()
+ assert T._parse_datatype_json_string(json.dumps("integer")) ==
T.IntegerType()
+ assert T._parse_datatype_json_string(st.json()) == st
+ assert pickle.loads(pickle.dumps(st)) == st
+ assert T.StructType().add("a", T.IntegerType()) == st
+
+
+def test_datatype_eq_with_non_datatypes():
+ for t in (T.IntegerType(), T.StructType([T.StructField("a",
T.IntegerType())])):
+ assert (t == "int") is False
+ assert (t == None) is False # noqa: E711
+ assert t != "int"
+ assert (T.StructField("a", T.LongType()) == T.LongType()) is False
+
+
+def test_mutated_structtype_compares_by_live_fields():
+ target = T.StructType([T.StructField("a", T.IntegerType())])
+ s = T.StructType()
+ assert s != target
+ s.add("a", T.IntegerType())
+ assert s == target
+ assert hash(s) == hash(target)
+
+
+def test_datatypes_usable_in_sets_and_dicts():
+ assert len({T.LongType(), T.LongType(), T.IntegerType()}) == 2
+ assert {T.ArrayType(T.LongType()): 1}.get(T.ArrayType(T.LongType())) == 1
+ fa, fa2, fb = (
+ T.StructField("a", T.LongType()),
+ T.StructField("a", T.LongType()),
+ T.StructField("a", T.IntegerType()),
+ )
+ assert fa == fa2 and fa != fb
+ assert hash(fa) == hash(fa2)
+ assert len({fa, fa2, fb}) == 2
+
+
+def test_user_defined_type_eq_and_hash():
+ class PointUDT(T.UserDefinedType):
+ @classmethod
+ def sqlType(cls):
+ return T.ArrayType(T.DoubleType(), False)
+
+ class OtherUDT(T.UserDefinedType):
+ @classmethod
+ def sqlType(cls):
+ return T.ArrayType(T.DoubleType(), False)
+
+ # pyspark: same class and equal __dict__ -> equal; a different UDT class
(even with
+ # the same sqlType) or a non-UDT type -> unequal.
+ assert PointUDT() == PointUDT()
+ assert hash(PointUDT()) == hash(PointUDT())
+ assert PointUDT() != OtherUDT()
+ assert PointUDT() != T.ArrayType(T.DoubleType(), False)
+ assert T.ArrayType(T.DoubleType(), False) != PointUDT()
+ assert T.LongType() != PointUDT()
+ u = PointUDT()
+ u.extra = 1
+ assert u != PointUDT()
+
+
# ---------------------------------------------------------------------------
StorageLevel
def test_storage_level():
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]