This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 05aac92 [SPARK-60919][CONNECT][PYTHON] Make Row hashable like a tuple
in the Rust drop-in
05aac92 is described below
commit 05aac92c79b96cf98e5ab85dc1eb87cf5288162f
Author: Liang-Chi Hsieh <[email protected]>
AuthorDate: Thu Nov 5 18:37:52 2026 +0900
[SPARK-60919][CONNECT][PYTHON] Make Row hashable like a tuple in the Rust
drop-in
### What changes were proposed in this pull request?
JIRA: [SPARK-60919](https://issues.apache.org/jira/browse/SPARK-60919)
Add `__hash__` to the drop-in's `Row` (`crates/pyspark-rs/src/row.rs`). It
hashes the tuple of the row's values.
### Why are the changes needed?
`Row` defines `__eq__` but not `__hash__`, so `hash(Row(a=1))` raises
`TypeError: unhashable type: 'builtins.Row'`. In PySpark, `Row` subclasses
`tuple` and overrides neither method, so it hashes like a tuple of its values:
`hash(Row(a=1)) == hash((1,))`, and Rows work in sets and as dict keys.
This stays consistent with `__eq__`, since equal rows have equal values. A
nested struct value comes back as a `Row`, so it hashes too. An array value (a
`list`) or a map value (a `dict`) still raises `TypeError`, as it does inside a
tuple.
### Does this PR introduce _any_ user-facing change?
Yes. `Row` objects can now be hashed, so they can go in sets and be used as
dict keys, matching PySpark.
### How was this patch tested?
New offline tests in `python/tests/test_dropin_offline.py`:
- the hash equals the hash of the values tuple, including nested rows and
an empty row
- equal rows collapse in a set, and a row works as a dict key
- rows holding list or dict values raise `TypeError`
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Anthropic)
Closes #157 from viirya/SPARK-60919.
Authored-by: Liang-Chi Hsieh <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/row.rs | 14 ++++++++++++++
python/tests/test_dropin_offline.py | 35 +++++++++++++++++++++++++++++++++++
2 files changed, 49 insertions(+)
diff --git a/crates/pyspark-rs/src/row.rs b/crates/pyspark-rs/src/row.rs
index d3bb100..d90f426 100644
--- a/crates/pyspark-rs/src/row.rs
+++ b/crates/pyspark-rs/src/row.rs
@@ -213,6 +213,20 @@ impl PyRow {
self.row == other.row
}
+ /// Hash like a tuple of the row's values, matching pyspark (`Row`
subclasses
+ /// `tuple`, so `hash(Row(a=1)) == hash((1,))`). Consistent with `__eq__`:
equal
+ /// rows have equal values, hence equal hashes. Unhashable values (array
-> list,
+ /// map -> dict) raise TypeError, as they do inside a tuple.
+ fn __hash__(&self, py: Python<'_>) -> PyResult<isize> {
+ let vals = self
+ .row
+ .values()
+ .iter()
+ .map(|v| value_to_py(py, v))
+ .collect::<PyResult<Vec<_>>>()?;
+ pyo3::types::PyTuple::new(py, vals)?.hash()
+ }
+
/// Number of occurrences of `value` among the row's values (tuple.count).
fn count(&self, py: Python<'_>, value: &Bound<'_, PyAny>) ->
PyResult<usize> {
let mut n = 0;
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index 350e012..5da4658 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -2495,6 +2495,41 @@ def test_row_asdict_ordering():
assert keys == ["a", "b", "c"]
+def test_row_hash_matches_tuple():
+ """Row hashes like a tuple of its values (pyspark Row subclasses tuple)."""
+ from pyspark.sql import Row
+
+ assert hash(Row(a=1)) == hash(Row(a=1))
+ assert hash(Row(a=1, b="x")) == hash((1, "x"))
+ assert hash(Row(1, 2.5, None)) == hash((1, 2.5, None))
+ assert hash(Row()) == hash(())
+ # Nested struct values materialize as Rows, which are hashable too.
+ assert hash(Row(a=Row(b=1))) == hash(((1,),))
+
+
+def test_row_usable_in_set_and_dict():
+ """Equal Rows collapse in a set and work as dict keys."""
+ from pyspark.sql import Row
+
+ s = {Row(a=1, b=2), Row(a=1, b=2), Row(a=3, b=4)}
+ assert len(s) == 2
+ assert Row(a=1, b=2) in s
+ d = {Row(k="x"): 1}
+ assert d[Row(k="x")] == 1
+
+
+def test_row_hash_unhashable_values_raise():
+ """Rows holding list/dict values are unhashable, as a tuple holding them
is."""
+ import pytest
+
+ from pyspark.sql import Row
+
+ with pytest.raises(TypeError):
+ hash(Row(a=[1, 2]))
+ with pytest.raises(TypeError):
+ hash(Row(a={"k": 1}))
+
+
# --- Connect import-path shims + offline utility helpers ---------------------
# The pyspark.sql.connect.* modules are thin re-exports that pandas-on-Spark
and the
# official connect tests import from; verify they import and re-export the
Rust-backed
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]