viirya opened a new pull request, #156:
URL: https://github.com/apache/spark-connect-rust/pull/156
### What changes were proposed in this pull request?
In the Rust-backed Python drop-in, the data type classes (`DataType` and
every concrete type such as `LongType`, `StructType`) defined no
`__eq__`/`__hash__`, so `==` fell back to object identity. This PR defines
`__eq__`, `__ne__` and `__hash__` once on the `DataType` base in
`crates/pyspark-rs/src/types.rs`, and every concrete type inherits them:
- `==` compares the core `DataType` built from each side's live fields via
`py_to_data_type`. A `StructType` mutated with `add()` compares by its current
fields, and the bare `DataType` returned by `fromDDL` equals the matching
concrete class.
- `UserDefinedType` is a pure-Python subclass whose state lives in
`__dict__`, so it keeps PySpark's rule verbatim: `isinstance(other,
self.__class__) and self.__dict__ == other.__dict__`.
- `hash()` hashes the simple string of the same live core type that `==`
compares, so equal types always hash equal. PySpark uses `hash(str(self))`, but
here `str()` of a mutated `StructType` still renders the fields it was
constructed with (`StructType` has no `__str__` and falls back to the base's
snapshot), which would break the hash/eq contract. A UDT keeps
`hash(str(self))`.
- `StructField` defined `__eq__` without `__hash__`, which made it
unhashable. It now has a `__hash__` like PySpark's.
### Why are the changes needed?
To match PySpark. Before this change:
```python
>>> from pyspark.sql.types import *
>>> LongType() == LongType()
False
>>> IntegerType() == DataType.fromDDL("int")
False
>>> hash(StructField("a", LongType()))
TypeError: unhashable type: 'builtins.StructField'
```
PySpark returns `True`, `True` and a hash value.
### Does this PR introduce _any_ user-facing change?
Yes. Data types now compare and hash like PySpark's, so they work as set
members and dict keys, and schema comparisons such as `df.schema == expected`
compare by value.
One known difference remains: in PySpark, a bare `DataType()` equals any
concrete type (`DataType() == LongType()` is `True`, because both `__dict__`s
are empty). Here it is `False`, because the bare `DataType` also stands for the
concrete types that `fromDDL` returns.
### How was this patch tested?
- New offline tests in `python/tests/test_dropin_offline.py`: pairwise
`==`/`!=`/`hash` across 42 types (atomic and parameterized types, intervals,
spatial, nested array/map/struct, field nullability and metadata), equality
across construction paths (`fromDDL`, `_parse_datatype_json_string`, pickle,
`add()`), non-DataType operands, a mutated `StructType`, set/dict membership,
`StructField`, and UDTs.
- The same tests pass against the official PySpark v4.3.0
`pyspark.sql.types`, except the `fromDDL` assertions, which need a live session
in PySpark.
- A probe script compared 4,481 eq/ne/hash results between the drop-in and
official PySpark v4.3.0. All match after this change; 161 differed before it.
- `cargo fmt --all --check`, `cargo test -p apache-spark-connect --release
--lib`, and `cargo test -p apache-spark-connect --release [--features wasm-udf]
--no-run` pass, and clippy reports nothing in the changed file.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
