This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 1fac7f8 [SPARK-60912][CONNECT] Make Row behave as PySpark's Row
1fac7f8 is described below
commit 1fac7f82912c49c4293724d1b8cb76d43810e264
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Thu Nov 5 18:54:47 2026 +0900
[SPARK-60912][CONNECT] Make Row behave as PySpark's Row
### What changes were proposed in this pull request?
[SPARK-60912](https://issues.apache.org/jira/browse/SPARK-60912): the
Python client's `Row` now behaves as PySpark's `Row`, a `tuple` subclass, does.
- `repr()` and `str()` give `Row(x=1, y='a')`, with the `repr()` of each
value, and `<Row(1, 2)>` for a row without field names, so
`print(df.collect())` prints `[Row(id=0)]` rather than `[[0]]`.
- A row made in Python holds the values that it was made with, as they are,
as PySpark's does. A row from the server converts its values on access, as
before. The rest of the client converts the values of a row made in Python when
it needs them, as it converts those of a tuple.
- As the tuple of its values does, a row hashes, compares with rows and
tuples regardless of field names, orders, slices, concatenates and repeats into
tuples, reverses, and pickles. It pickles through `_create_row`, now PySpark's,
so that field names survive, duplicate ones too, and the server's PySpark can
unpickle it.
- `Row("name", "age")` is a row class: calling it makes a row named by its
values, and too many values raise `TOO_MANY_VALUES`.
- A row without field names, such as `Row(1, 2)`, has no `__fields__`, its
`asDict()` raises `CANNOT_CONVERT_TYPE`, and `in` tests its values. For a row
with names, `in` tests the names. `asDict(recursive=True)` turns nested rows
into dicts, also in lists and dicts.
- The errors are PySpark's: a missing field raises `PySparkAttributeError`
(`ATTRIBUTE_NOT_SUPPORTED`) as an attribute and `PySparkValueError` as an item,
setting an attribute other than `__fields__`, which renames the fields, raises
`READ_ONLY`, and mixing positional and keyword arguments raises
`CANNOT_SET_TOGETHER`.
`isinstance(row, tuple)` stays false, as a PyO3 class cannot subclass
`tuple`.
### Why are the changes needed?
A row printed as a list of its values in a Rust format:
`print(spark.range(1).collect())` printed `[[0]]`, and a date printed as its
day number and `None` as `NULL`. `Row("name", "age")("Alice", 11)` raised
`TypeError`, as did `hash()`, `sorted()`, `pickle.dumps()`, slicing and
concatenation, `Row(x=1) == (1,)` was false, `"x" in row` tested the values,
`Row(1, 2).asDict()` returned a dict keyed by `_1` and `_2`, and missing fields
raised a plain `AttributeError` and a `KeyError`. A r [...]
### Does this PR introduce _any_ user-facing change?
Yes, `Row` now behaves as in PySpark Connect 4.3.0, except that it is not
an instance of `tuple`. Two existing tests that expected the previous behavior,
`in` over the values of a row with names, and `_create_row` reading the names
of a row with names, now expect PySpark's.
### How was this patch tested?
- Probes of 59 cases, covering making rows, the repr of made and collected
rows, field access and its errors, the tuple operations, pickling, `asDict`,
row classes, and the row of a join with duplicate names, give output identical
to official PySpark 4.3.0, except for `isinstance(row, tuple)`.
- New offline tests check the repr of ten kinds of rows, the tuple
behaviors, the field names and their errors, and `asDict`. They also pass with
official PySpark 4.3.0.
- A new live test checks collected rows: their repr, equality and hash with
a made row, pickling, nested values, and duplicate names. It also passes with
official PySpark 4.3.0.
- The offline Python suites (433 passed, 8 live cases skipped), and the
live drop-in and Arrow suites (22 passed) and live cases (9 passed). Clippy
reports no diagnostics in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #154 from HyukjinKwon/SPARK-60912-row.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/row.rs | 438 ++++++++++++++++++++++++++++--------
crates/pyspark-rs/src/session.rs | 5 +-
python/pyspark/sql/types.py | 14 +-
python/tests/test_dropin_e2e.py | 25 ++
python/tests/test_dropin_offline.py | 135 ++++++++++-
5 files changed, 501 insertions(+), 116 deletions(-)
diff --git a/crates/pyspark-rs/src/row.rs b/crates/pyspark-rs/src/row.rs
index d90f426..480fdc9 100644
--- a/crates/pyspark-rs/src/row.rs
+++ b/crates/pyspark-rs/src/row.rs
@@ -1,19 +1,128 @@
//! PyO3 wrapper for spark_connect::row::Row, mirroring `pyspark.sql.Row`.
-use pyo3::exceptions::{PyAttributeError, PyIndexError, PyKeyError};
+use pyo3::basic::CompareOp;
+use pyo3::exceptions::PyIndexError;
use pyo3::prelude::*;
-use pyo3::types::{PyBytes, PyDict, PyList};
+use pyo3::types::{PyBytes, PyDict, PyList, PySlice, PyString, PyTuple};
use spark_connect::row::{Row, Value};
-/// Python wrapper for a Spark Row.
+use crate::column::raise_pyspark;
+
+/// Python wrapper for a Spark Row. Behaves as `pyspark.sql.Row`, a `tuple`
subclass, does,
+/// though it cannot subclass `tuple` itself.
#[pyclass(name = "Row", module = "pyspark.sql.types")]
pub struct PyRow {
- pub(crate) row: Row,
+ data: RowData,
+ /// Whether the row has field names (`__fields__`). As in PySpark, a row
made of values
+ /// alone, such as `Row(1, 2)` or the row class `Row("name", "age")`, has
none; the rest
+ /// of the client knows its fields as `_1`, `_2`, and so on.
+ named: bool,
+}
+
+enum RowData {
+ /// A row that the server returned, whose Spark values become Python
values on access.
+ Spark(Row),
+ /// A row made in Python, which holds its Python values as they are, as
PySpark's does.
+ Python {
+ fields: Vec<String>,
+ values: Py<PyTuple>,
+ },
}
impl PyRow {
pub fn new(row: Row) -> Self {
- PyRow { row }
+ PyRow {
+ data: RowData::Spark(row),
+ named: true,
+ }
+ }
+
+ /// A row made in Python of `values`, named by `fields`.
+ fn of_python(fields: Vec<String>, values: Bound<'_, PyTuple>, named: bool)
-> Self {
+ PyRow {
+ data: RowData::Python {
+ fields,
+ values: values.unbind(),
+ },
+ named,
+ }
+ }
+
+ fn fields(&self) -> &[String] {
+ match &self.data {
+ RowData::Spark(row) => row.fields(),
+ RowData::Python { fields, .. } => fields,
+ }
+ }
+
+ /// The values, as the tuple that a PySpark row is.
+ fn values<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyTuple>> {
+ match &self.data {
+ RowData::Spark(row) => {
+ let values = row
+ .values()
+ .iter()
+ .map(|v| value_to_py(py, v))
+ .collect::<PyResult<Vec<_>>>()?;
+ PyTuple::new(py, values)
+ }
+ RowData::Python { values, .. } => Ok(values.bind(py).clone()),
+ }
+ }
+
+ /// The value of the field at `index`, which is in range.
+ fn value<'py>(&self, py: Python<'py>, index: usize) -> PyResult<Bound<'py,
PyAny>> {
+ match &self.data {
+ RowData::Spark(row) => value_to_py(py, row.get_unchecked(index)),
+ RowData::Python { values, .. } => values.bind(py).get_item(index),
+ }
+ }
+
+ /// The value of the first field named `name`, if any.
+ fn value_by_name<'py>(
+ &self,
+ py: Python<'py>,
+ name: &str,
+ ) -> PyResult<Option<Bound<'py, PyAny>>> {
+ match self.fields().iter().position(|field| field == name) {
+ Some(index) => self.value(py, index).map(Some),
+ None => Ok(None),
+ }
+ }
+
+ /// The row with Spark values, for the rest of the client, which converts
the values of a
+ /// row made in Python as it converts those of a tuple.
+ pub(crate) fn to_row(&self, py: Python<'_>) -> PyResult<Row> {
+ match &self.data {
+ RowData::Spark(row) => Ok(row.clone()),
+ RowData::Python { fields, values } => {
+ let values = values
+ .bind(py)
+ .iter()
+ .map(|v| crate::session::py_to_value(&v))
+ .collect::<PyResult<Vec<_>>>()?;
+ Ok(Row::new(fields.clone(), values))
+ }
+ }
+ }
+
+ /// What PySpark raises for an attribute that a row does not have.
+ fn no_attribute(py: Python<'_>, name: &str) -> PyErr {
+ raise_pyspark(
+ py,
+ "PySparkAttributeError",
+ "ATTRIBUTE_NOT_SUPPORTED",
+ &[("attr_name", name)],
+ )
+ }
+
+ /// The values of `other` as a tuple, if it is a row or a tuple, as a
tuple operation
+ /// takes them.
+ fn tuple_of<'py>(other: &Bound<'py, PyAny>) -> PyResult<Option<Bound<'py,
PyTuple>>> {
+ if let Ok(row) = other.cast::<PyRow>() {
+ return row.borrow().values(other.py()).map(Some);
+ }
+ Ok(other.cast::<PyTuple>().ok().cloned())
}
}
@@ -104,169 +213,304 @@ pub fn value_to_py<'py>(py: Python<'py>, v: &Value) ->
PyResult<Bound<'py, PyAny
#[pymethods]
impl PyRow {
- /// Construct a Row, mirroring `pyspark.sql.Row`:
- /// `Row(name=value, ...)` yields named fields; `Row(v1, v2, ...)` yields
- /// positional fields named `_1`, `_2`, ....
+ /// Construct a Row, mirroring `pyspark.sql.Row`: `Row(name=value, ...)`
makes a row with
+ /// field names, and `Row(v1, v2, ...)` one of values alone, which can
also serve as a row
+ /// class: `Row("name", "age")("Alice", 11)`.
#[new]
#[pyo3(signature = (*args, **kwargs))]
fn __new__(
- args: &Bound<'_, pyo3::types::PyTuple>,
+ py: Python<'_>,
+ args: &Bound<'_, PyTuple>,
kwargs: Option<&Bound<'_, PyDict>>,
) -> PyResult<Self> {
+ let kwargs = kwargs.filter(|kw| !kw.is_empty());
if let Some(kw) = kwargs {
if !args.is_empty() {
- return Err(pyo3::exceptions::PyValueError::new_err(
- "Row cannot mix positional and keyword arguments",
+ return Err(raise_pyspark(
+ py,
+ "PySparkValueError",
+ "CANNOT_SET_TOGETHER",
+ &[("arg_list", "args and kwargs")],
));
}
let mut fields = Vec::with_capacity(kw.len());
let mut values = Vec::with_capacity(kw.len());
for (k, v) in kw.iter() {
fields.push(k.extract::<String>()?);
- values.push(crate::session::py_to_value(&v)?);
+ values.push(v);
}
- Ok(PyRow::new(Row::new(fields, values)))
+ Ok(PyRow::of_python(fields, PyTuple::new(py, values)?, true))
} else {
- let mut values = Vec::with_capacity(args.len());
- for v in args.iter() {
- values.push(crate::session::py_to_value(&v)?);
- }
- let fields = (0..values.len()).map(|i| format!("_{}", i +
1)).collect();
- Ok(PyRow::new(Row::new(fields, values)))
+ let fields = (0..args.len()).map(|i| format!("_{}", i +
1)).collect();
+ Ok(PyRow::of_python(fields, args.clone(), false))
}
}
/// Number of fields.
fn __len__(&self) -> usize {
- self.row.len()
+ self.fields().len()
+ }
+
+ fn __iter__<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py,
pyo3::types::PyIterator>> {
+ self.values(py)?.try_iter()
+ }
+
+ fn __reversed__<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py,
PyAny>> {
+ py.import("builtins")?
+ .getattr("reversed")?
+ .call1((self.values(py)?,))
}
- /// `x in row` — membership over the row's values (pyspark `Row`
subclasses `tuple`,
- /// so `value in row` tests the values). Compares with Python equality.
+ /// `x in row` tests the field names of a row that has them, and the
values otherwise.
fn __contains__(&self, py: Python<'_>, item: &Bound<'_, PyAny>) ->
PyResult<bool> {
- for v in self.row.values() {
- if value_to_py(py, v)?.eq(item)? {
- return Ok(true);
- }
+ if self.named {
+ return PyList::new(py, self.fields())?.contains(item);
}
- Ok(false)
+ self.values(py)?.contains(item)
}
- /// Index (int, supports negative) or field-name (str) access. Raises
- /// IndexError / KeyError like pyspark so the Row is a proper sequence.
+ /// An index or a slice, as of a tuple, or a field name.
fn __getitem__<'py>(
&self,
py: Python<'py>,
- key: &Bound<'_, PyAny>,
+ key: &Bound<'py, PyAny>,
) -> PyResult<Bound<'py, PyAny>> {
+ if key.cast::<PySlice>().is_ok() {
+ return self.values(py)?.as_any().get_item(key);
+ }
if let Ok(mut i) = key.extract::<isize>() {
- let n = self.row.len() as isize;
+ let n = self.fields().len() as isize;
if i < 0 {
i += n;
}
if i < 0 || i >= n {
- return Err(PyIndexError::new_err("Row index out of range"));
+ return Err(PyIndexError::new_err("tuple index out of range"));
}
- return value_to_py(py, self.row.get(i as usize).unwrap());
+ return self.value(py, i as usize);
}
- if let Ok(name) = key.extract::<String>() {
- return match self.row.get_by_name(&name) {
- Some(v) => value_to_py(py, v),
- None => Err(PyKeyError::new_err(name)),
- };
+ // As PySpark looks the key up in `__fields__`, which a row without
names lacks.
+ if !self.named {
+ return Err(Self::no_attribute(py, "__fields__"));
}
- Err(PyKeyError::new_err("Row key must be int or str"))
+ if let Ok(name) = key.cast::<PyString>() {
+ if let Some(value) = self.value_by_name(py, &name.to_cow()?)? {
+ return Ok(value);
+ }
+ }
+ let error = py.import("pyspark.errors")?.getattr("PySparkValueError")?;
+ Err(PyErr::from_value(error.call1((key,))?))
}
/// Attribute access by field name (`row.id`).
fn __getattr__<'py>(&self, py: Python<'py>, name: &str) ->
PyResult<Bound<'py, PyAny>> {
- match self.row.get_by_name(name) {
- Some(v) => value_to_py(py, v),
- None => Err(PyAttributeError::new_err(name.to_string())),
+ if !name.starts_with("__") && self.named {
+ if let Some(value) = self.value_by_name(py, name)? {
+ return Ok(value);
+ }
+ }
+ Err(Self::no_attribute(py, name))
+ }
+
+ /// Only `__fields__` can be set, which names the fields, as in PySpark.
+ fn __setattr__(
+ &mut self,
+ py: Python<'_>,
+ name: &str,
+ value: &Bound<'_, PyAny>,
+ ) -> PyResult<()> {
+ if name != "__fields__" {
+ return Err(raise_pyspark(
+ py,
+ "PySparkRuntimeError",
+ "READ_ONLY",
+ &[("object", "Row")],
+ ));
}
+ let mut names = value
+ .try_iter()?
+ .map(|name| name?.str().map(|name| name.to_string()))
+ .collect::<PyResult<Vec<_>>>()?;
+ // PySpark pairs names with values as `zip` does; fields beyond the
names keep theirs.
+ let fields = self.fields();
+ names.truncate(fields.len());
+ names.extend(fields[names.len()..].iter().cloned());
+ match &mut self.data {
+ RowData::Spark(row) => *row = Row::new(names,
row.values().to_vec()),
+ RowData::Python { fields, .. } => *fields = names,
+ }
+ self.named = true;
+ Ok(())
}
- /// Field names, in order. A list ATTRIBUTE (pyspark's `Row.__fields__` is
a list, not a
- /// method), so `list(row.__fields__)` and `row.__fields__` work as in the
reference.
+ /// Field names, in order: a list attribute, as PySpark's `Row.__fields__`
is. A row
+ /// without names has none.
#[getter]
- fn __fields__(&self) -> Vec<String> {
- self.row.fields().to_vec()
+ fn __fields__(&self, py: Python<'_>) -> PyResult<Vec<String>> {
+ if !self.named {
+ return Err(Self::no_attribute(py, "__fields__"));
+ }
+ Ok(self.fields().to_vec())
}
- /// As a dict of field name -> value (mirrors `Row.asDict`).
+ /// As a dict of field name -> value (mirrors `Row.asDict`), with nested
rows, also in
+ /// lists and dicts, as dicts if `recursive`.
#[pyo3(name = "asDict")]
#[pyo3(signature = (recursive=false))]
fn as_dict<'py>(&self, py: Python<'py>, recursive: bool) ->
PyResult<Bound<'py, PyDict>> {
- let _ = recursive; // nested structs are already converted to dicts by
value_to_py
+ fn conv<'py>(obj: Bound<'py, PyAny>) -> PyResult<Bound<'py, PyAny>> {
+ let py = obj.py();
+ if let Ok(row) = obj.cast::<PyRow>() {
+ return Ok(row.borrow().as_dict(py, true)?.into_any());
+ }
+ if let Ok(list) = obj.cast::<PyList>() {
+ let items =
list.iter().map(conv).collect::<PyResult<Vec<_>>>()?;
+ return Ok(PyList::new(py, items)?.into_any());
+ }
+ if let Ok(dict) = obj.cast::<PyDict>() {
+ let converted = PyDict::new(py);
+ for (k, v) in dict.iter() {
+ converted.set_item(k, conv(v)?)?;
+ }
+ return Ok(converted.into_any());
+ }
+ Ok(obj)
+ }
+ if !self.named {
+ return Err(raise_pyspark(
+ py,
+ "PySparkTypeError",
+ "CANNOT_CONVERT_TYPE",
+ &[("from_type", "Row"), ("to_type", "dict")],
+ ));
+ }
let dict = PyDict::new(py);
- for (name, val) in self.row.fields().iter().zip(self.row.values()) {
- dict.set_item(name, value_to_py(py, val)?)?;
+ for (name, value) in self.fields().iter().zip(self.values(py)?.iter())
{
+ dict.set_item(name, if recursive { conv(value)? } else { value })?;
}
Ok(dict)
}
- fn __repr__(&self) -> String {
- format!("{}", self.row)
+ /// A row class, such as `Row("name", "age")`, makes a row of `args` named
by its values.
+ #[pyo3(signature = (*args))]
+ fn __call__(&self, py: Python<'_>, args: &Bound<'_, PyTuple>) ->
PyResult<PyRow> {
+ if args.len() > self.fields().len() {
+ let (expected, actual) = (self.fields().len().to_string(),
args.len().to_string());
+ return Err(raise_pyspark(
+ py,
+ "PySparkValueError",
+ "TOO_MANY_VALUES",
+ &[
+ ("expected", expected.as_str()),
+ ("item", "fields"),
+ ("actual", actual.as_str()),
+ ],
+ ));
+ }
+ let fields = self
+ .values(py)?
+ .iter()
+ .take(args.len())
+ .map(|name| name.str().map(|name| name.to_string()))
+ .collect::<PyResult<Vec<_>>>()?;
+ Ok(PyRow::of_python(fields, args.clone(), true))
}
- fn __eq__(&self, other: &PyRow) -> bool {
- self.row == other.row
+ /// Pickles as PySpark's row does: with `_create_row` from the field names
and values, or
+ /// as `Row(*values)` for a row without names.
+ fn __reduce__<'py>(slf: &Bound<'py, Self>) -> PyResult<Bound<'py,
PyTuple>> {
+ let py = slf.py();
+ let row = slf.borrow();
+ let values = row.values(py)?;
+ if row.named {
+ let create_row =
py.import("pyspark.sql.types")?.getattr("_create_row")?;
+ let args = (PyList::new(py, row.fields())?,
values).into_pyobject(py)?;
+ return (create_row, args).into_pyobject(py);
+ }
+ (slf.get_type(), values).into_pyobject(py)
}
- /// Hash like a tuple of the row's values, matching pyspark (`Row`
subclasses
- /// `tuple`, so `hash(Row(a=1)) == hash((1,))`). Consistent with `__eq__`:
equal
- /// rows have equal values, hence equal hashes. Unhashable values (array
-> list,
- /// map -> dict) raise TypeError, as they do inside a tuple.
+ /// `Row(name='Alice', age=11)`, with the `repr()` of each value, or
`<Row('a', 'b')>` for
+ /// a row without names, as PySpark prints a row.
+ fn __repr__(&self, py: Python<'_>) -> PyResult<String> {
+ let values = self.values(py)?;
+ let mut parts = Vec::with_capacity(values.len());
+ if self.named {
+ for (name, value) in self.fields().iter().zip(values.iter()) {
+ parts.push(format!("{}={}", name, value.repr()?));
+ }
+ return Ok(format!("Row({})", parts.join(", ")));
+ }
+ for value in values.iter() {
+ parts.push(value.repr()?.to_string());
+ }
+ Ok(format!("<Row({})>", parts.join(", ")))
+ }
+
+ /// Hashes as the tuple of its values does.
fn __hash__(&self, py: Python<'_>) -> PyResult<isize> {
- let vals = self
- .row
- .values()
- .iter()
- .map(|v| value_to_py(py, v))
- .collect::<PyResult<Vec<_>>>()?;
- pyo3::types::PyTuple::new(py, vals)?.hash()
+ self.values(py)?.hash()
+ }
+
+ /// Compares as the tuple of its values does, with another row or a tuple.
+ fn __richcmp__<'py>(
+ &self,
+ other: &Bound<'py, PyAny>,
+ op: CompareOp,
+ ) -> PyResult<Bound<'py, PyAny>> {
+ let py = other.py();
+ match Self::tuple_of(other)? {
+ Some(other) => self.values(py)?.rich_compare(other, op),
+ None => Ok(py.NotImplemented().into_bound(py)),
+ }
+ }
+
+ /// Concatenates as a tuple does, into a tuple.
+ fn __add__<'py>(&self, other: &Bound<'py, PyAny>) -> PyResult<Bound<'py,
PyAny>> {
+ let py = other.py();
+ match Self::tuple_of(other)? {
+ Some(other) => self.values(py)?.as_any().add(other),
+ None => Ok(py.NotImplemented().into_bound(py)),
+ }
+ }
+
+ fn __radd__<'py>(&self, other: &Bound<'py, PyAny>) -> PyResult<Bound<'py,
PyAny>> {
+ let py = other.py();
+ match Self::tuple_of(other)? {
+ Some(other) => other.as_any().add(self.values(py)?),
+ None => Ok(py.NotImplemented().into_bound(py)),
+ }
+ }
+
+ /// Repeats as a tuple does, into a tuple.
+ fn __mul__<'py>(&self, times: &Bound<'py, PyAny>) -> PyResult<Bound<'py,
PyAny>> {
+ self.values(times.py())?.as_any().mul(times)
+ }
+
+ fn __rmul__<'py>(&self, times: &Bound<'py, PyAny>) -> PyResult<Bound<'py,
PyAny>> {
+ self.__mul__(times)
}
/// Number of occurrences of `value` among the row's values (tuple.count).
fn count(&self, py: Python<'_>, value: &Bound<'_, PyAny>) ->
PyResult<usize> {
- let mut n = 0;
- for v in self.row.values() {
- if value_to_py(py, v)?.eq(value)? {
- n += 1;
- }
- }
- Ok(n)
+ self.values(py)?
+ .as_any()
+ .call_method1("count", (value,))?
+ .extract()
}
- /// Index of the first occurrence of `value` (tuple.index); raises
ValueError if
- /// not found. `start`/`stop` bound the search, with Python's
negative-index rules.
- #[pyo3(signature = (value, start=0, stop=None))]
+ /// Index of the first occurrence of `value` among the row's values
(tuple.index).
+ #[pyo3(signature = (value, *args))]
fn index(
&self,
py: Python<'_>,
value: &Bound<'_, PyAny>,
- start: isize,
- stop: Option<isize>,
+ args: &Bound<'_, PyTuple>,
) -> PyResult<usize> {
- let len = self.row.len() as isize;
- let norm = |i: isize| -> isize {
- if i < 0 {
- (len + i).max(0)
- } else {
- i.min(len)
- }
- };
- let lo = norm(start);
- let hi = stop.map(norm).unwrap_or(len);
- let mut i = lo;
- while i < hi {
- if value_to_py(py, self.row.get(i as usize).unwrap())?.eq(value)? {
- return Ok(i as usize);
- }
- i += 1;
- }
- Err(PyErr::new::<pyo3::exceptions::PyValueError, _>(
- "tuple.index(x): x not in tuple",
- ))
+ let mut call = vec![value.clone()];
+ call.extend(args.iter());
+ self.values(py)?
+ .as_any()
+ .call_method1("index", PyTuple::new(py, call)?)?
+ .extract()
}
}
diff --git a/crates/pyspark-rs/src/session.rs b/crates/pyspark-rs/src/session.rs
index f302757..cd40ee8 100644
--- a/crates/pyspark-rs/src/session.rs
+++ b/crates/pyspark-rs/src/session.rs
@@ -527,7 +527,8 @@ impl PySparkSession {
items.push(if let Ok(dict) = item.cast::<PyDict>() {
record(dict)?
} else if let Ok(pyrow) = item.extract::<PyRef<PyRow>>() {
- Item::Named(pyrow.row.fields().to_vec(),
pyrow.row.values().to_vec())
+ let row = pyrow.to_row(py)?;
+ Item::Named(row.fields().to_vec(), row.values().to_vec())
} else if let Ok(lst) = item.cast::<PyList>() {
Item::Plain(
lst.iter()
@@ -1259,7 +1260,7 @@ pub(crate) fn py_to_value(obj: &Bound<'_, PyAny>) ->
PyResult<Value> {
// pyspark Row -> struct: its named fields (values already core Values),
so a nested
// Row inside createDataFrame data (e.g. Row(s=Row(x=1))) converts
recursively.
if let Ok(pyrow) = obj.extract::<PyRef<crate::row::PyRow>>() {
- return Ok(Value::Struct(pyrow.row.clone().into()));
+ return Ok(Value::Struct(pyrow.to_row(obj.py())?.into()));
}
// dict -> map (keys coerced to their string form, values recursively
converted)
diff --git a/python/pyspark/sql/types.py b/python/pyspark/sql/types.py
index 3e8a051..6fb1d17 100644
--- a/python/pyspark/sql/types.py
+++ b/python/pyspark/sql/types.py
@@ -377,13 +377,7 @@ def _drop_metadata(d):
return d
def _create_row(fields, values):
- """Build a Row with the given field names (mirrors
pyspark.sql.types._create_row).
-
- Upstream builds ``Row(*values)`` then sets ``__fields__``; our Rust
``Row`` carries its
- field names from construction instead, so build it from the (name, value)
pairs.
- """
- try:
- names = list(fields.__fields__) # `fields` may itself be a Row
- except AttributeError:
- names = list(fields)
- return Row(**{name: value for name, value in zip(names, values)})
+ """A Row of ``values`` named by ``fields``, as in PySpark, whose pickled
rows call it."""
+ row = Row(*values)
+ row.__fields__ = fields
+ return row
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 060552a..682afbd 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -297,3 +297,28 @@ def
test_server_errors_carry_their_condition_and_details(session):
assert context.fragment() == "1/0"
# spark.sql.connect.serverStacktrace.enabled is true by default.
assert "\n\nJVM stacktrace:\norg.apache.spark.SparkArithmeticException" in
str(e)
+
+
+def test_collected_rows_behave_as_pyspark_rows(session):
+ import pickle
+
+ from pyspark.sql import Row
+
+ rows = session.range(1).collect()
+ assert repr(rows) == "[Row(id=0)]"
+ assert rows[0] == Row(id=0) and hash(rows[0]) == hash(Row(id=0))
+ assert repr(pickle.loads(pickle.dumps(rows[0]))) == "Row(id=0)"
+ row = session.range(1).selectExpr(
+ "named_struct('b', 1, 'c', array(1, 2)) AS a",
+ "date'2020-01-02' AS d",
+ "1.50BD AS n",
+ "CAST(NULL AS INT) AS z",
+ ).first()
+ assert repr(row) == (
+ "Row(a=Row(b=1, c=[1, 2]), d=datetime.date(2020, 1, 2),
n=Decimal('1.50'), z=None)"
+ )
+ assert row.asDict(recursive=True)["a"] == {"b": 1, "c": [1, 2]}
+ # Columns of the same name, as of a join, keep both values through
pickling.
+ duplicate = session.range(1).selectExpr("1 AS a", "2 AS a").first()
+ assert repr(duplicate) == "Row(a=1, a=2)" and duplicate.asDict() == {"a":
2}
+ assert repr(pickle.loads(pickle.dumps(duplicate))) == "Row(a=1, a=2)"
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index 413fcef..0fb6533 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -6,6 +6,8 @@ Python side has real, measurable coverage in CI without a
running server. Tests
that require a live server live in test_dropin_e2e.py and are gated on
SPARK_REMOTE.
"""
+import datetime
+import decimal
import json
import pickle
import time
@@ -1205,10 +1207,11 @@ def test_row_construction_and_access():
assert r2["a"] == 10
assert r2.b == 20
- # contains check
- assert 1 in r
- assert 10 in r2
- assert 99 not in r
+ # `in` tests the field names of a row that has them, as in PySpark, and
the values of
+ # one that does not.
+ assert "x" in r and "b" in r2
+ assert 1 not in r
+ assert 1 in Row(1, 2) and 99 not in Row(1, 2)
def test_row_iteration():
@@ -1220,6 +1223,124 @@ def test_row_iteration():
assert values == [1, 2, 3]
+class _Thing:
+ def __repr__(self):
+ return "Thing()"
+
+
[email protected](
+ "make, expected",
+ [
+ (lambda R: R(x=1, y="a"), "Row(x=1, y='a')"),
+ (lambda R: R(1, 2), "<Row(1, 2)>"),
+ (lambda R: R("a", "b"), "<Row('a', 'b')>"),
+ (lambda R: R(), "<Row()>"),
+ (lambda R: R(a=R(b=1, c=[1, 2]), d={"k": None}), "Row(a=Row(b=1, c=[1,
2]), d={'k': None})"),
+ (
+ lambda R: R(d=datetime.date(2020, 1, 2),
n=decimal.Decimal("1.50"), b=b"\x00"),
+ "Row(d=datetime.date(2020, 1, 2), n=Decimal('1.50'), b=b'\\x00')",
+ ),
+ (lambda R: R(a=None, f=1.5, t=True), "Row(a=None, f=1.5, t=True)"),
+ # A row keeps the values it was made with, as they are.
+ (lambda R: R(a=(1, 2), s={1}, o=_Thing()), "Row(a=(1, 2), s={1},
o=Thing())"),
+ (lambda R: R("name", "age")("Alice", 11), "Row(name='Alice', age=11)"),
+ (lambda R: R("name", "age")("Alice"), "Row(name='Alice')"),
+ ],
+)
+def test_row_repr(make, expected):
+ from pyspark.sql import Row
+
+ row = make(Row)
+ assert repr(row) == expected
+ assert str(row) == expected
+
+
+def test_row_behaves_as_a_tuple_of_its_values():
+ import pickle
+
+ from pyspark.sql import Row
+
+ r = Row(a=1, b="x")
+ # Equality, order and hash are those of the tuple of values; the names do
not count.
+ assert r == (1, "x") and (1, "x") == r and r == Row(c=1, d="x")
+ assert r != [1, "x"] and r != (1,)
+ assert Row(a=1) < Row(a=2) and Row(a=1) < (2,) and not Row(a=2) <= (1,)
+ assert sorted([Row(x=2), Row(x=1)]) == [Row(x=1), Row(x=2)]
+ assert hash(r) == hash((1, "x"))
+ assert len({Row(a=1), Row(a=1)}) == 1 and {Row(a=1): 5}[Row(a=1)] == 5
+ # Slices, concatenation and repetition are tuples.
+ assert r[0:1] == (1,) and type(r[0:1]) is tuple and r[::-1] == ("x", 1)
+ assert r + (3,) == (1, "x", 3) and (0,) + r == (0, 1, "x")
+ assert r * 2 == (1, "x", 1, "x") and 2 * r == (1, "x", 1, "x")
+ assert list(reversed(r)) == ["x", 1] and r[-1] == "x" and list(r) == [1,
"x"]
+ with pytest.raises(IndexError, match="tuple index out of range"):
+ r[5]
+ assert (r.count(1), r.index("x")) == (1, 1)
+ with pytest.raises(ValueError, match="x not in tuple"):
+ Row(a=1, b=2).index(2, 0, 1)
+ # Pickled rows keep their names, also when two fields share one.
+ for row in [r, Row(1, 2), Row(a=Row(b=[Row(c=1)])), Row("a", "b")(1, 2)]:
+ restored = pickle.loads(pickle.dumps(row))
+ assert repr(restored) == repr(row)
+ assert pickle.loads(pickle.dumps(Row(1, 2))) == (1, 2)
+
+
+def test_row_fields():
+ from pyspark.errors import (
+ PySparkAttributeError,
+ PySparkRuntimeError,
+ PySparkTypeError,
+ PySparkValueError,
+ )
+ from pyspark.sql import Row
+
+ r = Row(a=1, b="x")
+ assert r.__fields__ == ["a", "b"] and hasattr(r, "__fields__")
+ assert not hasattr(Row(1, 2), "__fields__")
+ # `in` tests the names of a row with names, and the values of one without.
+ assert "a" in r and 1 not in r and 1 in Row(1, 2)
+ with pytest.raises(PySparkAttributeError) as error:
+ r.missing
+ assert error.value.getCondition() == "ATTRIBUTE_NOT_SUPPORTED"
+ assert error.value.getMessageParameters() == {"attr_name": "missing"}
+ with pytest.raises(PySparkValueError):
+ r["missing"]
+ with pytest.raises(PySparkAttributeError) as error:
+ Row(1, 2)["a"]
+ assert error.value.getMessageParameters() == {"attr_name": "__fields__"}
+ with pytest.raises(PySparkRuntimeError) as error:
+ r.a = 2
+ assert error.value.getCondition() == "READ_ONLY"
+ renamed = Row(a=1, b=2)
+ renamed.__fields__ = ["p", "q"]
+ assert repr(renamed) == "Row(p=1, q=2)" and renamed.p == 1
+ with pytest.raises(PySparkValueError) as error:
+ Row(1, a=2)
+ assert error.value.getCondition() == "CANNOT_SET_TOGETHER"
+ with pytest.raises(PySparkValueError) as error:
+ Row("name", "age")(1, 2, 3)
+ assert error.value.getCondition() == "TOO_MANY_VALUES"
+ assert error.value.getMessageParameters() == {
+ "expected": "2",
+ "item": "fields",
+ "actual": "3",
+ }
+
+
+def test_row_as_dict():
+ from pyspark.errors import PySparkTypeError
+ from pyspark.sql import Row
+
+ nested = Row(a=Row(b=1), c=[Row(d=2)], e={"k": Row(f=3)})
+ assert nested.asDict() == {"a": Row(b=1), "c": [Row(d=2)], "e": {"k":
Row(f=3)}}
+ assert type(nested.asDict()["a"]) is Row
+ assert nested.asDict(recursive=True) == {"a": {"b": 1}, "c": [{"d": 2}],
"e": {"k": {"f": 3}}}
+ for row in [Row(1, 2), Row()]:
+ with pytest.raises(PySparkTypeError) as error:
+ row.asDict()
+ assert error.value.getCondition() == "CANNOT_CONVERT_TYPE"
+
+
def test_datatype_from_ddl_parsing():
"""Test DataType.fromDDL parsing of DDL strings."""
# StructField DDL parsing
@@ -2713,10 +2834,10 @@ def test_drop_metadata_and_create_row():
_drop_metadata(_IT())
r = _create_row(["a", "b"], [1, 2])
assert r["a"] == 1 and r["b"] == 2
- # _create_row accepts an existing Row as the field source (Row.__fields__
is a list attr).
- r2 = _create_row(r, [3, 4])
- assert r2["a"] == 3 and r2["b"] == 4
assert list(r.__fields__) == ["a", "b"]
+ # The fields can be a row class, whose values name them, as Row.__call__
passes it.
+ r2 = _create_row(T.Row("a", "b"), [3, 4])
+ assert r2["a"] == 3 and r2.b == 4
# ------------------------------------------- API parity: version, properties,
signatures
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]