This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 1fac7f8  [SPARK-60912][CONNECT] Make Row behave as PySpark's Row
1fac7f8 is described below

commit 1fac7f82912c49c4293724d1b8cb76d43810e264
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Thu Nov 5 18:54:47 2026 +0900

    [SPARK-60912][CONNECT] Make Row behave as PySpark's Row
    
    ### What changes were proposed in this pull request?
    
    [SPARK-60912](https://issues.apache.org/jira/browse/SPARK-60912): the 
Python client's `Row` now behaves as PySpark's `Row`, a `tuple` subclass, does.
    
    - `repr()` and `str()` give `Row(x=1, y='a')`, with the `repr()` of each 
value, and `<Row(1, 2)>` for a row without field names, so 
`print(df.collect())` prints `[Row(id=0)]` rather than `[[0]]`.
    - A row made in Python holds the values that it was made with, as they are, 
as PySpark's does. A row from the server converts its values on access, as 
before. The rest of the client converts the values of a row made in Python when 
it needs them, as it converts those of a tuple.
    - As the tuple of its values does, a row hashes, compares with rows and 
tuples regardless of field names, orders, slices, concatenates and repeats into 
tuples, reverses, and pickles. It pickles through `_create_row`, now PySpark's, 
so that field names survive, duplicate ones too, and the server's PySpark can 
unpickle it.
    - `Row("name", "age")` is a row class: calling it makes a row named by its 
values, and too many values raise `TOO_MANY_VALUES`.
    - A row without field names, such as `Row(1, 2)`, has no `__fields__`, its 
`asDict()` raises `CANNOT_CONVERT_TYPE`, and `in` tests its values. For a row 
with names, `in` tests the names. `asDict(recursive=True)` turns nested rows 
into dicts, also in lists and dicts.
    - The errors are PySpark's: a missing field raises `PySparkAttributeError` 
(`ATTRIBUTE_NOT_SUPPORTED`) as an attribute and `PySparkValueError` as an item, 
setting an attribute other than `__fields__`, which renames the fields, raises 
`READ_ONLY`, and mixing positional and keyword arguments raises 
`CANNOT_SET_TOGETHER`.
    
    `isinstance(row, tuple)` stays false, as a PyO3 class cannot subclass 
`tuple`.
    
    ### Why are the changes needed?
    
    A row printed as a list of its values in a Rust format: 
`print(spark.range(1).collect())` printed `[[0]]`, and a date printed as its 
day number and `None` as `NULL`. `Row("name", "age")("Alice", 11)` raised 
`TypeError`, as did `hash()`, `sorted()`, `pickle.dumps()`, slicing and 
concatenation, `Row(x=1) == (1,)` was false, `"x" in row` tested the values, 
`Row(1, 2).asDict()` returned a dict keyed by `_1` and `_2`, and missing fields 
raised a plain `AttributeError` and a `KeyError`. A r [...]
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes, `Row` now behaves as in PySpark Connect 4.3.0, except that it is not 
an instance of `tuple`. Two existing tests that expected the previous behavior, 
`in` over the values of a row with names, and `_create_row` reading the names 
of a row with names, now expect PySpark's.
    
    ### How was this patch tested?
    
    - Probes of 59 cases, covering making rows, the repr of made and collected 
rows, field access and its errors, the tuple operations, pickling, `asDict`, 
row classes, and the row of a join with duplicate names, give output identical 
to official PySpark 4.3.0, except for `isinstance(row, tuple)`.
    - New offline tests check the repr of ten kinds of rows, the tuple 
behaviors, the field names and their errors, and `asDict`. They also pass with 
official PySpark 4.3.0.
    - A new live test checks collected rows: their repr, equality and hash with 
a made row, pickling, nested values, and duplicate names. It also passes with 
official PySpark 4.3.0.
    - The offline Python suites (433 passed, 8 live cases skipped), and the 
live drop-in and Arrow suites (22 passed) and live cases (9 passed). Clippy 
reports no diagnostics in the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #154 from HyukjinKwon/SPARK-60912-row.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/row.rs        | 438 ++++++++++++++++++++++++++++--------
 crates/pyspark-rs/src/session.rs    |   5 +-
 python/pyspark/sql/types.py         |  14 +-
 python/tests/test_dropin_e2e.py     |  25 ++
 python/tests/test_dropin_offline.py | 135 ++++++++++-
 5 files changed, 501 insertions(+), 116 deletions(-)

diff --git a/crates/pyspark-rs/src/row.rs b/crates/pyspark-rs/src/row.rs
index d90f426..480fdc9 100644
--- a/crates/pyspark-rs/src/row.rs
+++ b/crates/pyspark-rs/src/row.rs
@@ -1,19 +1,128 @@
 //! PyO3 wrapper for spark_connect::row::Row, mirroring `pyspark.sql.Row`.
 
-use pyo3::exceptions::{PyAttributeError, PyIndexError, PyKeyError};
+use pyo3::basic::CompareOp;
+use pyo3::exceptions::PyIndexError;
 use pyo3::prelude::*;
-use pyo3::types::{PyBytes, PyDict, PyList};
+use pyo3::types::{PyBytes, PyDict, PyList, PySlice, PyString, PyTuple};
 use spark_connect::row::{Row, Value};
 
-/// Python wrapper for a Spark Row.
+use crate::column::raise_pyspark;
+
+/// Python wrapper for a Spark Row. Behaves as `pyspark.sql.Row`, a `tuple` 
subclass, does,
+/// though it cannot subclass `tuple` itself.
 #[pyclass(name = "Row", module = "pyspark.sql.types")]
 pub struct PyRow {
-    pub(crate) row: Row,
+    data: RowData,
+    /// Whether the row has field names (`__fields__`). As in PySpark, a row 
made of values
+    /// alone, such as `Row(1, 2)` or the row class `Row("name", "age")`, has 
none; the rest
+    /// of the client knows its fields as `_1`, `_2`, and so on.
+    named: bool,
+}
+
+enum RowData {
+    /// A row that the server returned, whose Spark values become Python 
values on access.
+    Spark(Row),
+    /// A row made in Python, which holds its Python values as they are, as 
PySpark's does.
+    Python {
+        fields: Vec<String>,
+        values: Py<PyTuple>,
+    },
 }
 
 impl PyRow {
     pub fn new(row: Row) -> Self {
-        PyRow { row }
+        PyRow {
+            data: RowData::Spark(row),
+            named: true,
+        }
+    }
+
+    /// A row made in Python of `values`, named by `fields`.
+    fn of_python(fields: Vec<String>, values: Bound<'_, PyTuple>, named: bool) 
-> Self {
+        PyRow {
+            data: RowData::Python {
+                fields,
+                values: values.unbind(),
+            },
+            named,
+        }
+    }
+
+    fn fields(&self) -> &[String] {
+        match &self.data {
+            RowData::Spark(row) => row.fields(),
+            RowData::Python { fields, .. } => fields,
+        }
+    }
+
+    /// The values, as the tuple that a PySpark row is.
+    fn values<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyTuple>> {
+        match &self.data {
+            RowData::Spark(row) => {
+                let values = row
+                    .values()
+                    .iter()
+                    .map(|v| value_to_py(py, v))
+                    .collect::<PyResult<Vec<_>>>()?;
+                PyTuple::new(py, values)
+            }
+            RowData::Python { values, .. } => Ok(values.bind(py).clone()),
+        }
+    }
+
+    /// The value of the field at `index`, which is in range.
+    fn value<'py>(&self, py: Python<'py>, index: usize) -> PyResult<Bound<'py, 
PyAny>> {
+        match &self.data {
+            RowData::Spark(row) => value_to_py(py, row.get_unchecked(index)),
+            RowData::Python { values, .. } => values.bind(py).get_item(index),
+        }
+    }
+
+    /// The value of the first field named `name`, if any.
+    fn value_by_name<'py>(
+        &self,
+        py: Python<'py>,
+        name: &str,
+    ) -> PyResult<Option<Bound<'py, PyAny>>> {
+        match self.fields().iter().position(|field| field == name) {
+            Some(index) => self.value(py, index).map(Some),
+            None => Ok(None),
+        }
+    }
+
+    /// The row with Spark values, for the rest of the client, which converts 
the values of a
+    /// row made in Python as it converts those of a tuple.
+    pub(crate) fn to_row(&self, py: Python<'_>) -> PyResult<Row> {
+        match &self.data {
+            RowData::Spark(row) => Ok(row.clone()),
+            RowData::Python { fields, values } => {
+                let values = values
+                    .bind(py)
+                    .iter()
+                    .map(|v| crate::session::py_to_value(&v))
+                    .collect::<PyResult<Vec<_>>>()?;
+                Ok(Row::new(fields.clone(), values))
+            }
+        }
+    }
+
+    /// What PySpark raises for an attribute that a row does not have.
+    fn no_attribute(py: Python<'_>, name: &str) -> PyErr {
+        raise_pyspark(
+            py,
+            "PySparkAttributeError",
+            "ATTRIBUTE_NOT_SUPPORTED",
+            &[("attr_name", name)],
+        )
+    }
+
+    /// The values of `other` as a tuple, if it is a row or a tuple, as a 
tuple operation
+    /// takes them.
+    fn tuple_of<'py>(other: &Bound<'py, PyAny>) -> PyResult<Option<Bound<'py, 
PyTuple>>> {
+        if let Ok(row) = other.cast::<PyRow>() {
+            return row.borrow().values(other.py()).map(Some);
+        }
+        Ok(other.cast::<PyTuple>().ok().cloned())
     }
 }
 
@@ -104,169 +213,304 @@ pub fn value_to_py<'py>(py: Python<'py>, v: &Value) -> 
PyResult<Bound<'py, PyAny
 
 #[pymethods]
 impl PyRow {
-    /// Construct a Row, mirroring `pyspark.sql.Row`:
-    /// `Row(name=value, ...)` yields named fields; `Row(v1, v2, ...)` yields
-    /// positional fields named `_1`, `_2`, ....
+    /// Construct a Row, mirroring `pyspark.sql.Row`: `Row(name=value, ...)` 
makes a row with
+    /// field names, and `Row(v1, v2, ...)` one of values alone, which can 
also serve as a row
+    /// class: `Row("name", "age")("Alice", 11)`.
     #[new]
     #[pyo3(signature = (*args, **kwargs))]
     fn __new__(
-        args: &Bound<'_, pyo3::types::PyTuple>,
+        py: Python<'_>,
+        args: &Bound<'_, PyTuple>,
         kwargs: Option<&Bound<'_, PyDict>>,
     ) -> PyResult<Self> {
+        let kwargs = kwargs.filter(|kw| !kw.is_empty());
         if let Some(kw) = kwargs {
             if !args.is_empty() {
-                return Err(pyo3::exceptions::PyValueError::new_err(
-                    "Row cannot mix positional and keyword arguments",
+                return Err(raise_pyspark(
+                    py,
+                    "PySparkValueError",
+                    "CANNOT_SET_TOGETHER",
+                    &[("arg_list", "args and kwargs")],
                 ));
             }
             let mut fields = Vec::with_capacity(kw.len());
             let mut values = Vec::with_capacity(kw.len());
             for (k, v) in kw.iter() {
                 fields.push(k.extract::<String>()?);
-                values.push(crate::session::py_to_value(&v)?);
+                values.push(v);
             }
-            Ok(PyRow::new(Row::new(fields, values)))
+            Ok(PyRow::of_python(fields, PyTuple::new(py, values)?, true))
         } else {
-            let mut values = Vec::with_capacity(args.len());
-            for v in args.iter() {
-                values.push(crate::session::py_to_value(&v)?);
-            }
-            let fields = (0..values.len()).map(|i| format!("_{}", i + 
1)).collect();
-            Ok(PyRow::new(Row::new(fields, values)))
+            let fields = (0..args.len()).map(|i| format!("_{}", i + 
1)).collect();
+            Ok(PyRow::of_python(fields, args.clone(), false))
         }
     }
 
     /// Number of fields.
     fn __len__(&self) -> usize {
-        self.row.len()
+        self.fields().len()
+    }
+
+    fn __iter__<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, 
pyo3::types::PyIterator>> {
+        self.values(py)?.try_iter()
+    }
+
+    fn __reversed__<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, 
PyAny>> {
+        py.import("builtins")?
+            .getattr("reversed")?
+            .call1((self.values(py)?,))
     }
 
-    /// `x in row` — membership over the row's values (pyspark `Row` 
subclasses `tuple`,
-    /// so `value in row` tests the values). Compares with Python equality.
+    /// `x in row` tests the field names of a row that has them, and the 
values otherwise.
     fn __contains__(&self, py: Python<'_>, item: &Bound<'_, PyAny>) -> 
PyResult<bool> {
-        for v in self.row.values() {
-            if value_to_py(py, v)?.eq(item)? {
-                return Ok(true);
-            }
+        if self.named {
+            return PyList::new(py, self.fields())?.contains(item);
         }
-        Ok(false)
+        self.values(py)?.contains(item)
     }
 
-    /// Index (int, supports negative) or field-name (str) access. Raises
-    /// IndexError / KeyError like pyspark so the Row is a proper sequence.
+    /// An index or a slice, as of a tuple, or a field name.
     fn __getitem__<'py>(
         &self,
         py: Python<'py>,
-        key: &Bound<'_, PyAny>,
+        key: &Bound<'py, PyAny>,
     ) -> PyResult<Bound<'py, PyAny>> {
+        if key.cast::<PySlice>().is_ok() {
+            return self.values(py)?.as_any().get_item(key);
+        }
         if let Ok(mut i) = key.extract::<isize>() {
-            let n = self.row.len() as isize;
+            let n = self.fields().len() as isize;
             if i < 0 {
                 i += n;
             }
             if i < 0 || i >= n {
-                return Err(PyIndexError::new_err("Row index out of range"));
+                return Err(PyIndexError::new_err("tuple index out of range"));
             }
-            return value_to_py(py, self.row.get(i as usize).unwrap());
+            return self.value(py, i as usize);
         }
-        if let Ok(name) = key.extract::<String>() {
-            return match self.row.get_by_name(&name) {
-                Some(v) => value_to_py(py, v),
-                None => Err(PyKeyError::new_err(name)),
-            };
+        // As PySpark looks the key up in `__fields__`, which a row without 
names lacks.
+        if !self.named {
+            return Err(Self::no_attribute(py, "__fields__"));
         }
-        Err(PyKeyError::new_err("Row key must be int or str"))
+        if let Ok(name) = key.cast::<PyString>() {
+            if let Some(value) = self.value_by_name(py, &name.to_cow()?)? {
+                return Ok(value);
+            }
+        }
+        let error = py.import("pyspark.errors")?.getattr("PySparkValueError")?;
+        Err(PyErr::from_value(error.call1((key,))?))
     }
 
     /// Attribute access by field name (`row.id`).
     fn __getattr__<'py>(&self, py: Python<'py>, name: &str) -> 
PyResult<Bound<'py, PyAny>> {
-        match self.row.get_by_name(name) {
-            Some(v) => value_to_py(py, v),
-            None => Err(PyAttributeError::new_err(name.to_string())),
+        if !name.starts_with("__") && self.named {
+            if let Some(value) = self.value_by_name(py, name)? {
+                return Ok(value);
+            }
+        }
+        Err(Self::no_attribute(py, name))
+    }
+
+    /// Only `__fields__` can be set, which names the fields, as in PySpark.
+    fn __setattr__(
+        &mut self,
+        py: Python<'_>,
+        name: &str,
+        value: &Bound<'_, PyAny>,
+    ) -> PyResult<()> {
+        if name != "__fields__" {
+            return Err(raise_pyspark(
+                py,
+                "PySparkRuntimeError",
+                "READ_ONLY",
+                &[("object", "Row")],
+            ));
         }
+        let mut names = value
+            .try_iter()?
+            .map(|name| name?.str().map(|name| name.to_string()))
+            .collect::<PyResult<Vec<_>>>()?;
+        // PySpark pairs names with values as `zip` does; fields beyond the 
names keep theirs.
+        let fields = self.fields();
+        names.truncate(fields.len());
+        names.extend(fields[names.len()..].iter().cloned());
+        match &mut self.data {
+            RowData::Spark(row) => *row = Row::new(names, 
row.values().to_vec()),
+            RowData::Python { fields, .. } => *fields = names,
+        }
+        self.named = true;
+        Ok(())
     }
 
-    /// Field names, in order. A list ATTRIBUTE (pyspark's `Row.__fields__` is 
a list, not a
-    /// method), so `list(row.__fields__)` and `row.__fields__` work as in the 
reference.
+    /// Field names, in order: a list attribute, as PySpark's `Row.__fields__` 
is. A row
+    /// without names has none.
     #[getter]
-    fn __fields__(&self) -> Vec<String> {
-        self.row.fields().to_vec()
+    fn __fields__(&self, py: Python<'_>) -> PyResult<Vec<String>> {
+        if !self.named {
+            return Err(Self::no_attribute(py, "__fields__"));
+        }
+        Ok(self.fields().to_vec())
     }
 
-    /// As a dict of field name -> value (mirrors `Row.asDict`).
+    /// As a dict of field name -> value (mirrors `Row.asDict`), with nested 
rows, also in
+    /// lists and dicts, as dicts if `recursive`.
     #[pyo3(name = "asDict")]
     #[pyo3(signature = (recursive=false))]
     fn as_dict<'py>(&self, py: Python<'py>, recursive: bool) -> 
PyResult<Bound<'py, PyDict>> {
-        let _ = recursive; // nested structs are already converted to dicts by 
value_to_py
+        fn conv<'py>(obj: Bound<'py, PyAny>) -> PyResult<Bound<'py, PyAny>> {
+            let py = obj.py();
+            if let Ok(row) = obj.cast::<PyRow>() {
+                return Ok(row.borrow().as_dict(py, true)?.into_any());
+            }
+            if let Ok(list) = obj.cast::<PyList>() {
+                let items = 
list.iter().map(conv).collect::<PyResult<Vec<_>>>()?;
+                return Ok(PyList::new(py, items)?.into_any());
+            }
+            if let Ok(dict) = obj.cast::<PyDict>() {
+                let converted = PyDict::new(py);
+                for (k, v) in dict.iter() {
+                    converted.set_item(k, conv(v)?)?;
+                }
+                return Ok(converted.into_any());
+            }
+            Ok(obj)
+        }
+        if !self.named {
+            return Err(raise_pyspark(
+                py,
+                "PySparkTypeError",
+                "CANNOT_CONVERT_TYPE",
+                &[("from_type", "Row"), ("to_type", "dict")],
+            ));
+        }
         let dict = PyDict::new(py);
-        for (name, val) in self.row.fields().iter().zip(self.row.values()) {
-            dict.set_item(name, value_to_py(py, val)?)?;
+        for (name, value) in self.fields().iter().zip(self.values(py)?.iter()) 
{
+            dict.set_item(name, if recursive { conv(value)? } else { value })?;
         }
         Ok(dict)
     }
 
-    fn __repr__(&self) -> String {
-        format!("{}", self.row)
+    /// A row class, such as `Row("name", "age")`, makes a row of `args` named 
by its values.
+    #[pyo3(signature = (*args))]
+    fn __call__(&self, py: Python<'_>, args: &Bound<'_, PyTuple>) -> 
PyResult<PyRow> {
+        if args.len() > self.fields().len() {
+            let (expected, actual) = (self.fields().len().to_string(), 
args.len().to_string());
+            return Err(raise_pyspark(
+                py,
+                "PySparkValueError",
+                "TOO_MANY_VALUES",
+                &[
+                    ("expected", expected.as_str()),
+                    ("item", "fields"),
+                    ("actual", actual.as_str()),
+                ],
+            ));
+        }
+        let fields = self
+            .values(py)?
+            .iter()
+            .take(args.len())
+            .map(|name| name.str().map(|name| name.to_string()))
+            .collect::<PyResult<Vec<_>>>()?;
+        Ok(PyRow::of_python(fields, args.clone(), true))
     }
 
-    fn __eq__(&self, other: &PyRow) -> bool {
-        self.row == other.row
+    /// Pickles as PySpark's row does: with `_create_row` from the field names 
and values, or
+    /// as `Row(*values)` for a row without names.
+    fn __reduce__<'py>(slf: &Bound<'py, Self>) -> PyResult<Bound<'py, 
PyTuple>> {
+        let py = slf.py();
+        let row = slf.borrow();
+        let values = row.values(py)?;
+        if row.named {
+            let create_row = 
py.import("pyspark.sql.types")?.getattr("_create_row")?;
+            let args = (PyList::new(py, row.fields())?, 
values).into_pyobject(py)?;
+            return (create_row, args).into_pyobject(py);
+        }
+        (slf.get_type(), values).into_pyobject(py)
     }
 
-    /// Hash like a tuple of the row's values, matching pyspark (`Row` 
subclasses
-    /// `tuple`, so `hash(Row(a=1)) == hash((1,))`). Consistent with `__eq__`: 
equal
-    /// rows have equal values, hence equal hashes. Unhashable values (array 
-> list,
-    /// map -> dict) raise TypeError, as they do inside a tuple.
+    /// `Row(name='Alice', age=11)`, with the `repr()` of each value, or 
`<Row('a', 'b')>` for
+    /// a row without names, as PySpark prints a row.
+    fn __repr__(&self, py: Python<'_>) -> PyResult<String> {
+        let values = self.values(py)?;
+        let mut parts = Vec::with_capacity(values.len());
+        if self.named {
+            for (name, value) in self.fields().iter().zip(values.iter()) {
+                parts.push(format!("{}={}", name, value.repr()?));
+            }
+            return Ok(format!("Row({})", parts.join(", ")));
+        }
+        for value in values.iter() {
+            parts.push(value.repr()?.to_string());
+        }
+        Ok(format!("<Row({})>", parts.join(", ")))
+    }
+
+    /// Hashes as the tuple of its values does.
     fn __hash__(&self, py: Python<'_>) -> PyResult<isize> {
-        let vals = self
-            .row
-            .values()
-            .iter()
-            .map(|v| value_to_py(py, v))
-            .collect::<PyResult<Vec<_>>>()?;
-        pyo3::types::PyTuple::new(py, vals)?.hash()
+        self.values(py)?.hash()
+    }
+
+    /// Compares as the tuple of its values does, with another row or a tuple.
+    fn __richcmp__<'py>(
+        &self,
+        other: &Bound<'py, PyAny>,
+        op: CompareOp,
+    ) -> PyResult<Bound<'py, PyAny>> {
+        let py = other.py();
+        match Self::tuple_of(other)? {
+            Some(other) => self.values(py)?.rich_compare(other, op),
+            None => Ok(py.NotImplemented().into_bound(py)),
+        }
+    }
+
+    /// Concatenates as a tuple does, into a tuple.
+    fn __add__<'py>(&self, other: &Bound<'py, PyAny>) -> PyResult<Bound<'py, 
PyAny>> {
+        let py = other.py();
+        match Self::tuple_of(other)? {
+            Some(other) => self.values(py)?.as_any().add(other),
+            None => Ok(py.NotImplemented().into_bound(py)),
+        }
+    }
+
+    fn __radd__<'py>(&self, other: &Bound<'py, PyAny>) -> PyResult<Bound<'py, 
PyAny>> {
+        let py = other.py();
+        match Self::tuple_of(other)? {
+            Some(other) => other.as_any().add(self.values(py)?),
+            None => Ok(py.NotImplemented().into_bound(py)),
+        }
+    }
+
+    /// Repeats as a tuple does, into a tuple.
+    fn __mul__<'py>(&self, times: &Bound<'py, PyAny>) -> PyResult<Bound<'py, 
PyAny>> {
+        self.values(times.py())?.as_any().mul(times)
+    }
+
+    fn __rmul__<'py>(&self, times: &Bound<'py, PyAny>) -> PyResult<Bound<'py, 
PyAny>> {
+        self.__mul__(times)
     }
 
     /// Number of occurrences of `value` among the row's values (tuple.count).
     fn count(&self, py: Python<'_>, value: &Bound<'_, PyAny>) -> 
PyResult<usize> {
-        let mut n = 0;
-        for v in self.row.values() {
-            if value_to_py(py, v)?.eq(value)? {
-                n += 1;
-            }
-        }
-        Ok(n)
+        self.values(py)?
+            .as_any()
+            .call_method1("count", (value,))?
+            .extract()
     }
 
-    /// Index of the first occurrence of `value` (tuple.index); raises 
ValueError if
-    /// not found. `start`/`stop` bound the search, with Python's 
negative-index rules.
-    #[pyo3(signature = (value, start=0, stop=None))]
+    /// Index of the first occurrence of `value` among the row's values 
(tuple.index).
+    #[pyo3(signature = (value, *args))]
     fn index(
         &self,
         py: Python<'_>,
         value: &Bound<'_, PyAny>,
-        start: isize,
-        stop: Option<isize>,
+        args: &Bound<'_, PyTuple>,
     ) -> PyResult<usize> {
-        let len = self.row.len() as isize;
-        let norm = |i: isize| -> isize {
-            if i < 0 {
-                (len + i).max(0)
-            } else {
-                i.min(len)
-            }
-        };
-        let lo = norm(start);
-        let hi = stop.map(norm).unwrap_or(len);
-        let mut i = lo;
-        while i < hi {
-            if value_to_py(py, self.row.get(i as usize).unwrap())?.eq(value)? {
-                return Ok(i as usize);
-            }
-            i += 1;
-        }
-        Err(PyErr::new::<pyo3::exceptions::PyValueError, _>(
-            "tuple.index(x): x not in tuple",
-        ))
+        let mut call = vec![value.clone()];
+        call.extend(args.iter());
+        self.values(py)?
+            .as_any()
+            .call_method1("index", PyTuple::new(py, call)?)?
+            .extract()
     }
 }
diff --git a/crates/pyspark-rs/src/session.rs b/crates/pyspark-rs/src/session.rs
index f302757..cd40ee8 100644
--- a/crates/pyspark-rs/src/session.rs
+++ b/crates/pyspark-rs/src/session.rs
@@ -527,7 +527,8 @@ impl PySparkSession {
             items.push(if let Ok(dict) = item.cast::<PyDict>() {
                 record(dict)?
             } else if let Ok(pyrow) = item.extract::<PyRef<PyRow>>() {
-                Item::Named(pyrow.row.fields().to_vec(), 
pyrow.row.values().to_vec())
+                let row = pyrow.to_row(py)?;
+                Item::Named(row.fields().to_vec(), row.values().to_vec())
             } else if let Ok(lst) = item.cast::<PyList>() {
                 Item::Plain(
                     lst.iter()
@@ -1259,7 +1260,7 @@ pub(crate) fn py_to_value(obj: &Bound<'_, PyAny>) -> 
PyResult<Value> {
     // pyspark Row -> struct: its named fields (values already core Values), 
so a nested
     // Row inside createDataFrame data (e.g. Row(s=Row(x=1))) converts 
recursively.
     if let Ok(pyrow) = obj.extract::<PyRef<crate::row::PyRow>>() {
-        return Ok(Value::Struct(pyrow.row.clone().into()));
+        return Ok(Value::Struct(pyrow.to_row(obj.py())?.into()));
     }
 
     // dict -> map (keys coerced to their string form, values recursively 
converted)
diff --git a/python/pyspark/sql/types.py b/python/pyspark/sql/types.py
index 3e8a051..6fb1d17 100644
--- a/python/pyspark/sql/types.py
+++ b/python/pyspark/sql/types.py
@@ -377,13 +377,7 @@ def _drop_metadata(d):
     return d
 
 def _create_row(fields, values):
-    """Build a Row with the given field names (mirrors 
pyspark.sql.types._create_row).
-
-    Upstream builds ``Row(*values)`` then sets ``__fields__``; our Rust 
``Row`` carries its
-    field names from construction instead, so build it from the (name, value) 
pairs.
-    """
-    try:
-        names = list(fields.__fields__)  # `fields` may itself be a Row
-    except AttributeError:
-        names = list(fields)
-    return Row(**{name: value for name, value in zip(names, values)})
+    """A Row of ``values`` named by ``fields``, as in PySpark, whose pickled 
rows call it."""
+    row = Row(*values)
+    row.__fields__ = fields
+    return row
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 060552a..682afbd 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -297,3 +297,28 @@ def 
test_server_errors_carry_their_condition_and_details(session):
     assert context.fragment() == "1/0"
     # spark.sql.connect.serverStacktrace.enabled is true by default.
     assert "\n\nJVM stacktrace:\norg.apache.spark.SparkArithmeticException" in 
str(e)
+
+
+def test_collected_rows_behave_as_pyspark_rows(session):
+    import pickle
+
+    from pyspark.sql import Row
+
+    rows = session.range(1).collect()
+    assert repr(rows) == "[Row(id=0)]"
+    assert rows[0] == Row(id=0) and hash(rows[0]) == hash(Row(id=0))
+    assert repr(pickle.loads(pickle.dumps(rows[0]))) == "Row(id=0)"
+    row = session.range(1).selectExpr(
+        "named_struct('b', 1, 'c', array(1, 2)) AS a",
+        "date'2020-01-02' AS d",
+        "1.50BD AS n",
+        "CAST(NULL AS INT) AS z",
+    ).first()
+    assert repr(row) == (
+        "Row(a=Row(b=1, c=[1, 2]), d=datetime.date(2020, 1, 2), 
n=Decimal('1.50'), z=None)"
+    )
+    assert row.asDict(recursive=True)["a"] == {"b": 1, "c": [1, 2]}
+    # Columns of the same name, as of a join, keep both values through 
pickling.
+    duplicate = session.range(1).selectExpr("1 AS a", "2 AS a").first()
+    assert repr(duplicate) == "Row(a=1, a=2)" and duplicate.asDict() == {"a": 
2}
+    assert repr(pickle.loads(pickle.dumps(duplicate))) == "Row(a=1, a=2)"
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index 413fcef..0fb6533 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -6,6 +6,8 @@ Python side has real, measurable coverage in CI without a 
running server. Tests
 that require a live server live in test_dropin_e2e.py and are gated on 
SPARK_REMOTE.
 """
 
+import datetime
+import decimal
 import json
 import pickle
 import time
@@ -1205,10 +1207,11 @@ def test_row_construction_and_access():
     assert r2["a"] == 10
     assert r2.b == 20
 
-    # contains check
-    assert 1 in r
-    assert 10 in r2
-    assert 99 not in r
+    # `in` tests the field names of a row that has them, as in PySpark, and 
the values of
+    # one that does not.
+    assert "x" in r and "b" in r2
+    assert 1 not in r
+    assert 1 in Row(1, 2) and 99 not in Row(1, 2)
 
 
 def test_row_iteration():
@@ -1220,6 +1223,124 @@ def test_row_iteration():
     assert values == [1, 2, 3]
 
 
+class _Thing:
+    def __repr__(self):
+        return "Thing()"
+
+
[email protected](
+    "make, expected",
+    [
+        (lambda R: R(x=1, y="a"), "Row(x=1, y='a')"),
+        (lambda R: R(1, 2), "<Row(1, 2)>"),
+        (lambda R: R("a", "b"), "<Row('a', 'b')>"),
+        (lambda R: R(), "<Row()>"),
+        (lambda R: R(a=R(b=1, c=[1, 2]), d={"k": None}), "Row(a=Row(b=1, c=[1, 
2]), d={'k': None})"),
+        (
+            lambda R: R(d=datetime.date(2020, 1, 2), 
n=decimal.Decimal("1.50"), b=b"\x00"),
+            "Row(d=datetime.date(2020, 1, 2), n=Decimal('1.50'), b=b'\\x00')",
+        ),
+        (lambda R: R(a=None, f=1.5, t=True), "Row(a=None, f=1.5, t=True)"),
+        # A row keeps the values it was made with, as they are.
+        (lambda R: R(a=(1, 2), s={1}, o=_Thing()), "Row(a=(1, 2), s={1}, 
o=Thing())"),
+        (lambda R: R("name", "age")("Alice", 11), "Row(name='Alice', age=11)"),
+        (lambda R: R("name", "age")("Alice"), "Row(name='Alice')"),
+    ],
+)
+def test_row_repr(make, expected):
+    from pyspark.sql import Row
+
+    row = make(Row)
+    assert repr(row) == expected
+    assert str(row) == expected
+
+
+def test_row_behaves_as_a_tuple_of_its_values():
+    import pickle
+
+    from pyspark.sql import Row
+
+    r = Row(a=1, b="x")
+    # Equality, order and hash are those of the tuple of values; the names do 
not count.
+    assert r == (1, "x") and (1, "x") == r and r == Row(c=1, d="x")
+    assert r != [1, "x"] and r != (1,)
+    assert Row(a=1) < Row(a=2) and Row(a=1) < (2,) and not Row(a=2) <= (1,)
+    assert sorted([Row(x=2), Row(x=1)]) == [Row(x=1), Row(x=2)]
+    assert hash(r) == hash((1, "x"))
+    assert len({Row(a=1), Row(a=1)}) == 1 and {Row(a=1): 5}[Row(a=1)] == 5
+    # Slices, concatenation and repetition are tuples.
+    assert r[0:1] == (1,) and type(r[0:1]) is tuple and r[::-1] == ("x", 1)
+    assert r + (3,) == (1, "x", 3) and (0,) + r == (0, 1, "x")
+    assert r * 2 == (1, "x", 1, "x") and 2 * r == (1, "x", 1, "x")
+    assert list(reversed(r)) == ["x", 1] and r[-1] == "x" and list(r) == [1, 
"x"]
+    with pytest.raises(IndexError, match="tuple index out of range"):
+        r[5]
+    assert (r.count(1), r.index("x")) == (1, 1)
+    with pytest.raises(ValueError, match="x not in tuple"):
+        Row(a=1, b=2).index(2, 0, 1)
+    # Pickled rows keep their names, also when two fields share one.
+    for row in [r, Row(1, 2), Row(a=Row(b=[Row(c=1)])), Row("a", "b")(1, 2)]:
+        restored = pickle.loads(pickle.dumps(row))
+        assert repr(restored) == repr(row)
+    assert pickle.loads(pickle.dumps(Row(1, 2))) == (1, 2)
+
+
+def test_row_fields():
+    from pyspark.errors import (
+        PySparkAttributeError,
+        PySparkRuntimeError,
+        PySparkTypeError,
+        PySparkValueError,
+    )
+    from pyspark.sql import Row
+
+    r = Row(a=1, b="x")
+    assert r.__fields__ == ["a", "b"] and hasattr(r, "__fields__")
+    assert not hasattr(Row(1, 2), "__fields__")
+    # `in` tests the names of a row with names, and the values of one without.
+    assert "a" in r and 1 not in r and 1 in Row(1, 2)
+    with pytest.raises(PySparkAttributeError) as error:
+        r.missing
+    assert error.value.getCondition() == "ATTRIBUTE_NOT_SUPPORTED"
+    assert error.value.getMessageParameters() == {"attr_name": "missing"}
+    with pytest.raises(PySparkValueError):
+        r["missing"]
+    with pytest.raises(PySparkAttributeError) as error:
+        Row(1, 2)["a"]
+    assert error.value.getMessageParameters() == {"attr_name": "__fields__"}
+    with pytest.raises(PySparkRuntimeError) as error:
+        r.a = 2
+    assert error.value.getCondition() == "READ_ONLY"
+    renamed = Row(a=1, b=2)
+    renamed.__fields__ = ["p", "q"]
+    assert repr(renamed) == "Row(p=1, q=2)" and renamed.p == 1
+    with pytest.raises(PySparkValueError) as error:
+        Row(1, a=2)
+    assert error.value.getCondition() == "CANNOT_SET_TOGETHER"
+    with pytest.raises(PySparkValueError) as error:
+        Row("name", "age")(1, 2, 3)
+    assert error.value.getCondition() == "TOO_MANY_VALUES"
+    assert error.value.getMessageParameters() == {
+        "expected": "2",
+        "item": "fields",
+        "actual": "3",
+    }
+
+
+def test_row_as_dict():
+    from pyspark.errors import PySparkTypeError
+    from pyspark.sql import Row
+
+    nested = Row(a=Row(b=1), c=[Row(d=2)], e={"k": Row(f=3)})
+    assert nested.asDict() == {"a": Row(b=1), "c": [Row(d=2)], "e": {"k": 
Row(f=3)}}
+    assert type(nested.asDict()["a"]) is Row
+    assert nested.asDict(recursive=True) == {"a": {"b": 1}, "c": [{"d": 2}], 
"e": {"k": {"f": 3}}}
+    for row in [Row(1, 2), Row()]:
+        with pytest.raises(PySparkTypeError) as error:
+            row.asDict()
+        assert error.value.getCondition() == "CANNOT_CONVERT_TYPE"
+
+
 def test_datatype_from_ddl_parsing():
     """Test DataType.fromDDL parsing of DDL strings."""
     # StructField DDL parsing
@@ -2713,10 +2834,10 @@ def test_drop_metadata_and_create_row():
     _drop_metadata(_IT())
     r = _create_row(["a", "b"], [1, 2])
     assert r["a"] == 1 and r["b"] == 2
-    # _create_row accepts an existing Row as the field source (Row.__fields__ 
is a list attr).
-    r2 = _create_row(r, [3, 4])
-    assert r2["a"] == 3 and r2["b"] == 4
     assert list(r.__fields__) == ["a", "b"]
+    # The fields can be a row class, whose values name them, as Row.__call__ 
passes it.
+    r2 = _create_row(T.Row("a", "b"), [3, 4])
+    assert r2["a"] == 3 and r2.b == 4
 
 
 # ------------------------------------------- API parity: version, properties, 
signatures


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to