This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 32cfcb7  [SPARK-60918][CONNECT] Make == and hash() of Spark Connect 
Rust data types match PySpark
32cfcb7 is described below

commit 32cfcb701cc62616d2f143e9ae135aff5d9d162c
Author: Liang-Chi Hsieh <[email protected]>
AuthorDate: Thu Nov 5 18:37:09 2026 +0900

    [SPARK-60918][CONNECT] Make == and hash() of Spark Connect Rust data types 
match PySpark
    
    ### What changes were proposed in this pull request?
    
    In the Rust-backed Python drop-in, the data type classes (`DataType` and 
every concrete type such as `LongType`, `StructType`) defined no 
`__eq__`/`__hash__`, so `==` fell back to object identity. This PR defines 
`__eq__`, `__ne__` and `__hash__` once on the `DataType` base in 
`crates/pyspark-rs/src/types.rs`, and every concrete type inherits them:
    
    - `==` compares the core `DataType` built from each side's live fields via 
`py_to_data_type`. A `StructType` mutated with `add()` compares by its current 
fields, and the bare `DataType` returned by `fromDDL` equals the matching 
concrete class.
    - `UserDefinedType` is a pure-Python subclass whose state lives in 
`__dict__`, so it keeps PySpark's rule verbatim: `isinstance(other, 
self.__class__) and self.__dict__ == other.__dict__`.
    - `hash()` hashes the simple string of the same live core type that `==` 
compares, so equal types always hash equal. PySpark uses `hash(str(self))`, but 
here `str()` of a mutated `StructType` still renders the fields it was 
constructed with (`StructType` has no `__str__` and falls back to the base's 
snapshot), which would break the hash/eq contract. A UDT keeps 
`hash(str(self))`.
    - `StructField` defined `__eq__` without `__hash__`, which made it 
unhashable. It now has a `__hash__` like PySpark's.
    
    ### Why are the changes needed?
    
    To match PySpark. Before this change:
    
    ```python
    >>> from pyspark.sql.types import *
    >>> LongType() == LongType()
    False
    >>> IntegerType() == DataType.fromDDL("int")
    False
    >>> hash(StructField("a", LongType()))
    TypeError: unhashable type: 'builtins.StructField'
    ```
    
    PySpark returns `True`, `True` and a hash value.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. Data types now compare and hash like PySpark's, so they work as set 
members and dict keys, and schema comparisons such as `df.schema == expected` 
compare by value.
    
    One known difference remains: in PySpark, a bare `DataType()` equals any 
concrete type (`DataType() == LongType()` is `True`, because both `__dict__`s 
are empty). Here it is `False`, because the bare `DataType` also stands for the 
concrete types that `fromDDL` returns.
    
    ### How was this patch tested?
    
    - New offline tests in `python/tests/test_dropin_offline.py`: pairwise 
`==`/`!=`/`hash` across 42 types (atomic and parameterized types, intervals, 
spatial, nested array/map/struct, field nullability and metadata), equality 
across construction paths (`fromDDL`, `_parse_datatype_json_string`, pickle, 
`add()`), non-DataType operands, a mutated `StructType`, set/dict membership, 
`StructField`, and UDTs.
    - The same tests pass against the official PySpark v4.3.0 
`pyspark.sql.types`, except the `fromDDL` assertions, which need a live session 
in PySpark.
    - A probe script compared 4,481 eq/ne/hash results between the drop-in and 
official PySpark v4.3.0. All match after this change; 161 differed before it.
    - `cargo fmt --all --check`, `cargo test -p apache-spark-connect --release 
--lib`, and `cargo test -p apache-spark-connect --release [--features wasm-udf] 
--no-run` pass, and clippy reports nothing in the changed file.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Code (Claude Opus 5.5)
    
    Closes #156 from viirya/SPARK-60918.
    
    Authored-by: Liang-Chi Hsieh <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/types.rs      |  63 ++++++++++++++++++++
 python/tests/test_dropin_offline.py | 114 ++++++++++++++++++++++++++++++++++++
 2 files changed, 177 insertions(+)

diff --git a/crates/pyspark-rs/src/types.rs b/crates/pyspark-rs/src/types.rs
index 935fc72..983c141 100644
--- a/crates/pyspark-rs/src/types.rs
+++ b/crates/pyspark-rs/src/types.rs
@@ -200,6 +200,64 @@ impl PyDataType {
     fn simpleString(&self) -> String {
         self.inner.simple_string()
     }
+
+    // Equality/hashing are defined once here and inherited by every concrete 
type class
+    // (and by pure-Python subclasses such as `UserDefinedType`). Mirrors 
pyspark's
+    // `DataType.__eq__`/`__ne__`/`__hash__`.
+    fn __eq__(slf: &Bound<'_, Self>, other: &Bound<'_, PyAny>) -> 
PyResult<bool> {
+        data_type_eq(slf.as_any(), other)
+    }
+
+    fn __ne__(slf: &Bound<'_, Self>, other: &Bound<'_, PyAny>) -> 
PyResult<bool> {
+        Ok(!data_type_eq(slf.as_any(), other)?)
+    }
+
+    /// pyspark hashes `str(self)`. Hash the simple string of the same live 
core type that
+    /// `__eq__` compares, so equal types always hash equal (`str()` can lag 
behind the
+    /// live fields of a mutated `StructType`). A `UserDefinedType` is 
compared by class
+    /// and `__dict__` rather than by core type, so it keeps pyspark's 
`hash(str(self))`.
+    fn __hash__(slf: &Bound<'_, Self>) -> PyResult<isize> {
+        let any = slf.as_any();
+        if is_udt(any)? {
+            return any.str()?.hash();
+        }
+        let s = py_to_data_type(any)?.simple_string();
+        pyo3::types::PyString::new(any.py(), &s).hash()
+    }
+}
+
+/// Whether `obj` is an instance of the pure-Python 
`pyspark.sql.types.UserDefinedType`.
+fn is_udt(obj: &Bound<'_, PyAny>) -> PyResult<bool> {
+    match obj
+        .py()
+        .import("pyspark.sql.types")
+        .and_then(|m| m.getattr("UserDefinedType"))
+    {
+        Ok(cls) => obj.is_instance(&cls),
+        Err(_) => Ok(false),
+    }
+}
+
+/// `DataType.__eq__` with pyspark semantics. pyspark compares
+/// `isinstance(other, self.__class__) and self.__dict__ == other.__dict__`; 
the Rust-backed
+/// classes keep their state in Rust rather than `__dict__`, so compare the 
core `DataType`
+/// built from each side's live fields instead (a mutated `StructType` 
compares by its
+/// current fields, and a bare `DataType` returned by `fromDDL` equals the 
concrete class).
+/// `UserDefinedType` is a pure-Python subclass whose state does live in 
`__dict__`, so it
+/// keeps pyspark's formula verbatim.
+fn data_type_eq(a: &Bound<'_, PyAny>, b: &Bound<'_, PyAny>) -> PyResult<bool> {
+    if !b.is_instance_of::<PyDataType>() {
+        return Ok(false);
+    }
+    let a_udt = is_udt(a)?;
+    if a_udt || is_udt(b)? {
+        // A non-UDT `self` is never a superclass of a UDT other than the bare 
base,
+        // whose (empty) state pyspark would compare; treat that as unequal 
too.
+        return Ok(a_udt
+            && b.is_instance(&a.get_type())?
+            && a.getattr("__dict__")?.eq(b.getattr("__dict__")?)?);
+    }
+    Ok(py_to_data_type(a)? == py_to_data_type(b)?)
 }
 
 // Define concrete type classes
@@ -1586,6 +1644,11 @@ impl PyStructField {
             false
         }
     }
+    /// pyspark: `hash(str(self))` (inherited from `DataType`). Defining 
`__eq__` alone
+    /// would leave StructField unhashable.
+    fn __hash__(slf: &Bound<'_, Self>) -> PyResult<isize> {
+        slf.str()?.hash()
+    }
 }
 
 #[pyclass(name = "StructType", module = "pyspark.sql.types", extends = 
PyDataType)]
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index e2a2627..350e012 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -106,6 +106,120 @@ def test_structfield_construction():
     assert "n" in st.simpleString()
 
 
+# --------------------------------------------------------------------------- 
type equality
+# Expected results are what PySpark 4.3.0 returns (`DataType.__eq__` is
+# `isinstance(other, self.__class__) and self.__dict__ == other.__dict__`, 
`__hash__` is
+# `hash(str(self))`).
+
+def _eq_cases():
+    F = T.StructField
+    return [
+        T.NullType, T.BooleanType, T.ByteType, T.ShortType, T.IntegerType, 
T.LongType,
+        T.FloatType, T.DoubleType, T.StringType, lambda: 
T.StringType("UTF8_LCASE"),
+        T.BinaryType, T.DateType, T.TimestampType, T.TimestampNTZType, 
T.VariantType,
+        T.CalendarIntervalType, T.DecimalType, lambda: T.DecimalType(10, 2),
+        lambda: T.DecimalType(12, 2), lambda: T.CharType(5), lambda: 
T.CharType(6),
+        lambda: T.VarcharType(5), lambda: T.TimeType(3), lambda: T.TimeType(6),
+        lambda: T.YearMonthIntervalType(0, 1), lambda: 
T.YearMonthIntervalType(0, 0),
+        lambda: T.DayTimeIntervalType(0, 3), lambda: T.DayTimeIntervalType(1, 
3),
+        lambda: T.GeometryType(0), lambda: T.GeometryType(4326),
+        lambda: T.GeographyType(4326),
+        lambda: T.ArrayType(T.IntegerType()), lambda: 
T.ArrayType(T.IntegerType(), False),
+        lambda: T.ArrayType(T.LongType()),
+        lambda: T.MapType(T.StringType(), T.IntegerType()),
+        lambda: T.MapType(T.StringType(), T.IntegerType(), False),
+        lambda: T.StructType([F("a", T.IntegerType())]),
+        lambda: T.StructType([F("a", T.IntegerType(), False)]),
+        lambda: T.StructType([F("b", T.IntegerType())]),
+        lambda: T.StructType([F("a", T.IntegerType(), True, {"k": "v"})]),
+        lambda: T.StructType(),
+        lambda: T.StructType([
+            F("x", T.ArrayType(T.MapType(T.StringType(), T.DecimalType(10, 
2))))
+        ]),
+    ]
+
+
+def test_datatype_eq_and_hash_match_pyspark():
+    cases = _eq_cases()
+    left = [f() for f in cases]
+    right = [f() for f in cases]
+    for i, a in enumerate(left):
+        for j, b in enumerate(right):
+            # Every case is a distinct type, so equal exactly on the diagonal.
+            assert (a == b) is (i == j), (a, b)
+            assert (a != b) is (i != j), (a, b)
+        assert hash(a) == hash(right[i]), a
+    assert type(T.LongType()).__eq__ is not object.__eq__
+
+
+def test_datatype_eq_across_construction_paths():
+    import json
+
+    st = T.StructType([T.StructField("a", T.IntegerType())])
+    assert T.IntegerType() == T.DataType.fromDDL("int")
+    assert T.DataType.fromDDL("int") == T.IntegerType()
+    assert hash(T.DataType.fromDDL("int")) == hash(T.IntegerType())
+    assert T.DataType.fromDDL("bigint") != T.IntegerType()
+    assert T._parse_datatype_json_string(json.dumps("integer")) == 
T.IntegerType()
+    assert T._parse_datatype_json_string(st.json()) == st
+    assert pickle.loads(pickle.dumps(st)) == st
+    assert T.StructType().add("a", T.IntegerType()) == st
+
+
+def test_datatype_eq_with_non_datatypes():
+    for t in (T.IntegerType(), T.StructType([T.StructField("a", 
T.IntegerType())])):
+        assert (t == "int") is False
+        assert (t == None) is False  # noqa: E711
+        assert t != "int"
+    assert (T.StructField("a", T.LongType()) == T.LongType()) is False
+
+
+def test_mutated_structtype_compares_by_live_fields():
+    target = T.StructType([T.StructField("a", T.IntegerType())])
+    s = T.StructType()
+    assert s != target
+    s.add("a", T.IntegerType())
+    assert s == target
+    assert hash(s) == hash(target)
+
+
+def test_datatypes_usable_in_sets_and_dicts():
+    assert len({T.LongType(), T.LongType(), T.IntegerType()}) == 2
+    assert {T.ArrayType(T.LongType()): 1}.get(T.ArrayType(T.LongType())) == 1
+    fa, fa2, fb = (
+        T.StructField("a", T.LongType()),
+        T.StructField("a", T.LongType()),
+        T.StructField("a", T.IntegerType()),
+    )
+    assert fa == fa2 and fa != fb
+    assert hash(fa) == hash(fa2)
+    assert len({fa, fa2, fb}) == 2
+
+
+def test_user_defined_type_eq_and_hash():
+    class PointUDT(T.UserDefinedType):
+        @classmethod
+        def sqlType(cls):
+            return T.ArrayType(T.DoubleType(), False)
+
+    class OtherUDT(T.UserDefinedType):
+        @classmethod
+        def sqlType(cls):
+            return T.ArrayType(T.DoubleType(), False)
+
+    # pyspark: same class and equal __dict__ -> equal; a different UDT class 
(even with
+    # the same sqlType) or a non-UDT type -> unequal.
+    assert PointUDT() == PointUDT()
+    assert hash(PointUDT()) == hash(PointUDT())
+    assert PointUDT() != OtherUDT()
+    assert PointUDT() != T.ArrayType(T.DoubleType(), False)
+    assert T.ArrayType(T.DoubleType(), False) != PointUDT()
+    assert T.LongType() != PointUDT()
+    u = PointUDT()
+    u.extra = 1
+    assert u != PointUDT()
+
+
 # --------------------------------------------------------------------------- 
StorageLevel
 
 def test_storage_level():


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to