This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 586286bd [SPARK-61100][CONNECT][PYTHON] Preserve tuple semantics for
Python Row
586286bd is described below
commit 586286bd82b6b9ba07c75e0ab14e3b48849980b7
Author: DB Tsai <[email protected]>
AuthorDate: Mon Nov 9 09:29:31 2026 +0900
[SPARK-61100][CONNECT][PYTHON] Preserve tuple semantics for Python Row
### What changes were proposed in this pull request?
Return one public tuple-based `Row` through the native extension and Python
imports. Embed PySpark 4.3.0's Row implementation, preserve mutable
`__fields__`, and materialize Rows for collection and iteration. Share field
names per collected result, per top-level iterator Arrow batch, and per nested
schema path.
Update local-input conversion, pickling and UDT conversion to use that
identity. Add tuple, field-mutation, nested-input, batch and UDT/spatial
regressions.
### Why are the changes needed?
[SPARK-61100](https://issues.apache.org/jira/browse/SPARK-61100). The
native object wrapper lacks tuple inheritance and has different identity and
field-mutation behavior, affecting tuple-aware serializers and pandas-on-Spark
paths.
Includes the [NumPy
prerequisite](https://github.com/apache/spark-connect-rust/pull/381)
(SPARK-61102) and [binary configuration/iterator
follow-up](https://github.com/apache/spark-connect-rust/pull/382) (SPARK-61099).
### Does this PR introduce _any_ user-facing change?
Yes. Rows support PySpark's tuple interface, public identity and field-name
semantics. The inherited customized-pandas named-Row conversion behavior is
retained; complete pandas parity is not established.
Tuple compatibility has a measured performance cost. For 100,000 collected
rows followed by full recursive consumption, comparing the final binary parent
with this change:
| Shape | Client CPU, parent → Row (ms) | Wall time (ms) | RSS after
validation (MiB) |
| --- | --- | --- | --- |
| One integer | 47.8 → 88.2 | 59.9 → 99.8 | 109.8 → 124.5 |
| Eight integers | 155.9 → 196.6 | 173.0 → 220.9 | 150.3 → 164.7 |
| Four fields, three nested Rows | 418.1 → 670.1 | 441.4 → 707.7 | 285.7 →
344.6 |
Paired CPU ratios were 1.31–1.86× and wall ratios 1.26–1.65×; retained RSS
increased by 14–59 MiB. Collect-only CPU ratios were 6.18–16.29× because values
now materialize before return. Wide iterator consumption improved; narrow and
nested iteration remained slower. These costs are an explicit tradeoff for
tuple compatibility.
### How was this patch tested?
- Formatting, Rust core/API units, integration-test compilation with and
without WASM, and native-extension Python suites: 2,291 passed, eight
live-server skips.
- Twelve native live cases and all 36 setup/call/teardown reports passed
with verified imports; 172 unrelated cases were deselected. Eight smoke actions
covered nested Rows, JSON, pickle, shared names and local inputs. The NTZ
test's nested subprocess had no separate import telemetry.
- Scoped official transport checks: 85 passed, 34 skipped, no retries; all
12 mixed Classic/Connect collection cases skipped. API/member checks confirmed
tuple inheritance and inspected members, with annotation differences retained.
- The three offline files named by CI passed 2,363 cases with eight
remote-only skips. Configured Python-shim coverage was 84.3% across 72 files;
embedded Row and compiled Rust are excluded. A causal control reproduced
`NoSource` without the synthetic Row omission. The workflow's 90/75 values
select badge colors, not fail-under thresholds.
- Both paired benchmarks completed 54 fresh workers, 27 pairs and 162
validated observations each. Results above are medians of process medians;
paired ratios use paired samples. This is one macOS/Python/dependency set with
three pairs, no confidence intervals or server-CPU measurements. RSS is an
endpoint observation, not allocation or leak evidence. Official
Python/newest-master performance was not measured. Auxiliary profiling ended
with a cleanup error and its timings are excluded.
The shared full and cluster runs executed the NumPy prerequisite binary.
This branch has the same 43 selected raw-RPC source files and separate
native-import/live validation. This is source-based reuse of transport
evidence; those full and cluster runs did not execute this branch's binary.
They use official Python plans and converters, with no per-child import
attestation, and do not validate native high-level conversion, native
ReleaseSession or native artifact upload.
The full run completed 87 normal files: 2,586 passed, 641 skipped, one
deselected and no failures, errors or retries. Eleven files were entirely
skipped. A separate advisory check of the existing metadata-column manifest
entry still failed because the test catalog plugin was unavailable (one
failure, 57 deselections). The cluster phase passed 15 invocations across four
modules (ten substantive cases and five helpers), with 14 JVM-guarded skips and
no failures or retries.
The original full run had a seeded-random failure on two threads. The
unchanged test failed identically in official Python and both saved Rust
transports on two threads, and passed in all three on its default four-thread
fixture. The complete successful rerun used that fixture. Server logs retain
deliberate negative-test diagnostics and shutdown races; successful direct
shutdown was recorded, without final descendant-process telemetry.
Clippy remains advisory exit 101; its source diagnostics were assessed
against the baseline. Linux, Windows and the newest-master combined tree were
not runtime tested locally. These results establish neither full converter
parity nor performance parity.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex
This pull request and its description were written by OpenAI Codex.
Closes #383 from dbtsai/codex/python-rust-gap-row.
Lead-authored-by: DB Tsai <[email protected]>
Co-authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
.github/workflows/lint.yml | 6 +-
AGENTS.md | 16 +-
crates/pyspark-rs/src/dataframe.rs | 95 ++++--
crates/pyspark-rs/src/infer.rs | 22 +-
crates/pyspark-rs/src/lib.rs | 3 +-
crates/pyspark-rs/src/row.py | 228 ++++++++++++++
crates/pyspark-rs/src/row.rs | 605 ++++++++----------------------------
crates/pyspark-rs/src/session.rs | 33 +-
crates/spark-connect/src/row.rs | 23 ++
pyproject.toml | 14 +-
python/.coveragerc | 6 +
python/pyspark/sql/types.py | 26 +-
python/tests/test_arrow_handoff.py | 92 ++++++
python/tests/test_dropin_offline.py | 343 +++++++++++++++++++-
14 files changed, 936 insertions(+), 576 deletions(-)
diff --git a/.github/workflows/lint.yml b/.github/workflows/lint.yml
index 1f226e29..c1473891 100644
--- a/.github/workflows/lint.yml
+++ b/.github/workflows/lint.yml
@@ -66,7 +66,7 @@ jobs:
run: python3 -m unittest discover -s scripts/tests
python-ruff:
- name: ruff (our scripts)
+ name: ruff (our scripts and embedded Row)
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
@@ -76,6 +76,6 @@ jobs:
- name: Install ruff
run: python -m pip install "ruff==0.14.0"
- name: ruff check - blocking
- run: ruff check scripts/
+ run: ruff check scripts/ crates/pyspark-rs/src/row.py
- name: ruff format (check) - blocking
- run: ruff format --check scripts/
+ run: ruff format --check scripts/ crates/pyspark-rs/src/row.py
diff --git a/AGENTS.md b/AGENTS.md
index 4810ae4d..bbe4115c 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -243,7 +243,9 @@ Vendor the ones we don't have a Rust implementation for.
(the gRPC channel connects lazily, so proto/plan construction needs no
server)
and server-gated `tests/e2e_*.rs` (they check
`std::env::var("SPARK_REMOTE")`).
- **Python drop-in**: `python/.coveragerc` scopes to the hand-written shim
- (omits vendored/generated/Rust-only). Run:
+ (omits vendored/generated/Rust-only). The embedded
`crates/pyspark-rs/src/row.py`
+ runs as `pyspark/_row.py` without a file at that path and is explicitly
omitted;
+ Row tests exercise its behavior and Ruff checks its source. Run:
```
cd python && PYTHONPATH=$(pwd) python -m coverage run --rcfile=.coveragerc \
-m pytest tests/test_dropin_offline.py && python -m coverage report
--rcfile=.coveragerc
@@ -414,9 +416,15 @@ For every method matching a pyspark name:
proto carrying the pickled object: `registerDataSource`,
`GroupedData.transformWithState[InPandas]`,
`createDataFrame(..., udf)`, etc. General rule: proto-building → Rust;
cloudpickled Python code that
runs server-side → vendored Python.
-- **Row materialization:** PyO3 cannot subclass `tuple`, so our `Row` is not a
tuple subclass.
- Some pandas-on-Spark code assumes a tuple-`Row` with mutable `__fields__`.
Full fidelity
- likely requires adopting the upstream Python tuple-`Row` at collect
boundaries.
+- **Row materialization:** `crates/pyspark-rs/src/row.py` embeds PySpark's
Python `Row(tuple)`
+ class, exported as `pyspark._pyspark.Row` and re-exported by
`pyspark.sql.types`,
+ `pyspark.sql`, and `pyspark`. Embedding avoids importing the partially
initialized
+ `pyspark.sql.types` shim while the extension loads. Rust eagerly builds
tuple values
+ through `pyspark.sql.types._create_row`; collected rows share a mutable
`__fields__`
+ list across the result, and iterator rows share top-level names within each
Arrow
+ record batch. Nested names are shared by schema path across the execution,
including
+ iterator batches, while distinct sibling paths keep separate lists.
+ Nested local-input Rows follow the tuple inference paths; named Rows expose
`__fields__`.
- **Streaming listeners and progress holders:** Keep progress holders
(`StreamingQueryProgress`,
`StateOperatorProgress`, etc.) as faithful pure-Python implementations
parsed from server
JSON, not Rust-backed. The listener bus (subscribe/dispatch) is genuinely
Rust-backed;
diff --git a/crates/pyspark-rs/src/dataframe.rs
b/crates/pyspark-rs/src/dataframe.rs
index cfd0e888..b98ffa34 100644
--- a/crates/pyspark-rs/src/dataframe.rs
+++ b/crates/pyspark-rs/src/dataframe.rs
@@ -9,7 +9,7 @@ use spark_connect::udf::{eval_type,
CommonInlineUserDefinedFunctionExpression, P
use crate::column::PyColumn;
use crate::errors::ResultExt;
use crate::group::PyGroupedData;
-use crate::row::{FieldConverters, PyRow};
+use crate::row::{FieldConverters, NestedFieldNames};
use crate::streaming::PyDataStreamWriter;
use crate::types::PyDataType;
@@ -2187,12 +2187,12 @@ impl PyDataFrame {
}
/// Collect all rows into memory.
- fn collect(&self, py: Python<'_>) -> PyResult<Vec<PyRow>> {
+ fn collect(&self, py: Python<'_>) -> PyResult<Vec<Py<PyAny>>> {
// Release the GIL across the blocking RPC so other Python threads run.
let (rows, schema) = py
.detach(|| self.dataframe.collect_with_schema())
.to_pyerr()?;
- rows_to_py(py, &self.dataframe, rows, schema.as_ref())
+ rows_to_py(py, &self.dataframe, &rows, schema.as_ref())
}
/// Return an iterator over rows without materializing the entire result
in memory.
@@ -2396,12 +2396,11 @@ impl PyDataFrame {
}
/// Get the first row.
- fn first(&self, py: Python<'_>) -> PyResult<Option<PyRow>> {
+ fn first(&self, py: Python<'_>) -> PyResult<Option<Py<PyAny>>> {
let (row, schema) = py
.detach(|| self.dataframe.first_with_schema())
.to_pyerr()?;
- let rows = row.into_iter().collect();
- Ok(rows_to_py(py, &self.dataframe, rows, schema.as_ref())?.pop())
+ Ok(rows_to_py(py, &self.dataframe, row.as_slice(),
schema.as_ref())?.pop())
}
/// Alias for first.
@@ -2416,17 +2415,17 @@ impl PyDataFrame {
let (rows, schema) = py
.detach(|| self.dataframe.limit(k).collect_with_schema())
.to_pyerr()?;
- rows_to_py(py, &self.dataframe, rows,
schema.as_ref())?.into_py_any(py)
+ rows_to_py(py, &self.dataframe, &rows,
schema.as_ref())?.into_py_any(py)
}
}
}
/// Get the first n rows.
- fn take(&self, py: Python<'_>, num: usize) -> PyResult<Vec<PyRow>> {
+ fn take(&self, py: Python<'_>, num: usize) -> PyResult<Vec<Py<PyAny>>> {
let (rows, schema) = py
.detach(|| self.dataframe.take_with_schema(num))
.to_pyerr()?;
- rows_to_py(py, &self.dataframe, rows, schema.as_ref())
+ rows_to_py(py, &self.dataframe, &rows, schema.as_ref())
}
/// Column names (a property, mirroring `DataFrame.columns`).
@@ -2436,11 +2435,11 @@ impl PyDataFrame {
}
/// Get the last `num` rows as a list of Rows (eager, like pyspark `tail`).
- fn tail(&self, py: Python<'_>, num: i32) -> PyResult<Vec<PyRow>> {
+ fn tail(&self, py: Python<'_>, num: i32) -> PyResult<Vec<Py<PyAny>>> {
let (rows, schema) = py
.detach(|| self.dataframe.tail(num).collect_with_schema())
.to_pyerr()?;
- rows_to_py(py, &self.dataframe, rows, schema.as_ref())
+ rows_to_py(py, &self.dataframe, &rows, schema.as_ref())
}
/// Sample a fraction of rows, as PySpark's
`DataFrame.sample(withReplacement=None,
@@ -4281,23 +4280,36 @@ fn read_binary_as_bytes(py: Python<'_>, dataframe:
&DataFrame) -> PyResult<bool>
Ok(value.is_some_and(|value| value.eq_ignore_ascii_case("true")))
}
-/// The collected `rows` of a result of `schema` of `dataframe`, whose values
of user-defined
-/// types are deserialized when first read. Reads the binary result setting
after execution
-/// and applies it to binary values, including when only an Arrow schema is
available.
+/// The collected `rows` of a result of `schema`, with configured binary
values and
+/// user-defined values materialized before constructing the tuples returned
to Python.
+/// Reads the binary setting after execution, including with only an Arrow
schema.
fn rows_to_py(
py: Python<'_>,
dataframe: &DataFrame,
- rows: Vec<spark_connect::row::Row>,
+ rows: &[spark_connect::row::Row],
schema: Option<&spark_connect::types::DataType>,
-) -> PyResult<Vec<PyRow>> {
+) -> PyResult<Vec<Py<PyAny>>> {
check_arrow_schema(py, schema)?;
let binary_as_bytes = read_binary_as_bytes(py, dataframe)?;
check_collectable(py, schema)?;
let converters = udt_field_converters(py, schema)?;
- Ok(rows
- .into_iter()
- .map(|row| PyRow::with_converters(row, converters.clone(),
binary_as_bytes))
- .collect())
+ let Some(first) = rows.first() else {
+ return Ok(Vec::new());
+ };
+ let fields = PyList::new(py, first.fields())?;
+ let mut nested_fields = NestedFieldNames::default();
+ rows.iter()
+ .map(|row| {
+ crate::row::row_values_to_py(
+ py,
+ &fields,
+ row.values(),
+ converters.as_ref(),
+ binary_as_bytes,
+ &mut nested_fields,
+ )
+ })
+ .collect()
}
/// Python wrapper for LocalRowIterator yielding Row objects lazily.
@@ -4316,6 +4328,10 @@ pub struct PyLocalRowIterator {
converters: Option<Option<FieldConverters>>,
/// Snapshotted before the first fetch, as the body of PySpark's generator
does.
binary_as_bytes: Option<bool>,
+ /// Top-level names shared only by rows of the current Arrow record batch.
+ fields: Option<(std::sync::Arc<[String]>, Py<PyList>)>,
+ /// Nested names belong to schema paths and stay shared across iterator
batches.
+ nested_fields: NestedFieldNames,
}
impl PyLocalRowIterator {
@@ -4327,6 +4343,8 @@ impl PyLocalRowIterator {
finished: false,
converters: None,
binary_as_bytes: None,
+ fields: None,
+ nested_fields: NestedFieldNames::default(),
}
}
@@ -4342,7 +4360,7 @@ impl PyLocalRowIterator {
}
/// Prepare and fetch one Python row; the public wrapper closes on
exhaustion or error.
- fn next_python_row(&mut self, py: Python<'_>) -> PyResult<Option<PyRow>> {
+ fn next_python_row(&mut self, py: Python<'_>) ->
PyResult<Option<Py<PyAny>>> {
let binary_as_bytes = match self.binary_as_bytes {
Some(value) => value,
None => {
@@ -4362,11 +4380,34 @@ impl PyLocalRowIterator {
match next {
Some(Ok(row)) => {
let converters = self.converters.clone().flatten();
- Ok(Some(PyRow::with_converters(
- row,
- converters,
+ let fields = match row.shared_fields() {
+ Some(native_fields) => match self.fields.as_ref() {
+ Some((cached, fields)) if
std::sync::Arc::ptr_eq(cached, native_fields) => {
+ fields.bind(py).clone()
+ }
+ _ => {
+ let fields = PyList::new(py, row.fields())?;
+ self.fields = Some((
+ std::sync::Arc::clone(native_fields),
+ fields.clone().unbind(),
+ ));
+ fields
+ }
+ },
+ None => {
+ self.fields = None;
+ PyList::new(py, row.fields())?
+ }
+ };
+ crate::row::row_values_to_py(
+ py,
+ &fields,
+ row.values(),
+ converters.as_ref(),
binary_as_bytes,
- )))
+ &mut self.nested_fields,
+ )
+ .map(Some)
}
Some(Err(error)) =>
Err(crate::errors::spark_error_to_py_exception(error)),
None => Ok(None),
@@ -4386,6 +4427,8 @@ impl PyLocalRowIterator {
fn close(&mut self, py: Python<'_>) {
let iterator = self.iterator.take();
self.finished = true;
+ self.fields = None;
+ self.nested_fields.clear();
py.detach(move || drop(iterator));
}
@@ -4393,7 +4436,7 @@ impl PyLocalRowIterator {
///
/// The first call snapshots the binary result setting before starting
execution.
/// Both the Config RPC and blocking row fetch release the GIL.
- fn __next__(&mut self, py: Python<'_>) -> PyResult<Option<PyRow>> {
+ fn __next__(&mut self, py: Python<'_>) -> PyResult<Option<Py<PyAny>>> {
if self.finished {
return Ok(None);
}
diff --git a/crates/pyspark-rs/src/infer.rs b/crates/pyspark-rs/src/infer.rs
index 0b682543..c2e51146 100644
--- a/crates/pyspark-rs/src/infer.rs
+++ b/crates/pyspark-rs/src/infer.rs
@@ -23,7 +23,6 @@ use spark_connect::types::{DataType, GeographyType,
GeometryType, StructField};
use crate::column::raise_pyspark;
use crate::errors::ResultExt;
-use crate::row::PyRow;
use crate::values::{PyGeography, PyGeometry, PyVariantVal};
static DECIMAL: PyOnceLock<Py<PyType>> = PyOnceLock::new();
@@ -656,18 +655,6 @@ fn row_items<'py>(
if row.is_instance_of::<PyDict>() {
return sorted_items(row);
}
- if let Ok(pyrow) = row.cast::<PyRow>() {
- let pyrow = pyrow.borrow();
- let (fields, values) = pyrow.parts(py)?;
- return Ok(match fields {
- Some(fields) => fields
- .iter()
- .map(|f| PyString::new(py, f).into_any())
- .zip(values.iter())
- .collect(),
- None => positional(py, values.iter().collect(), names),
- });
- }
if row.is_instance_of::<PyTuple>() || row.is_instance_of::<PyList>() {
let fields = if row.hasattr(intern!(py, "__fields__"))? {
Some(row.getattr(intern!(py, "__fields__"))?)
@@ -1663,10 +1650,7 @@ fn values_by_position<'py>(
/// and another object by the names of its `__dict__`.
fn convert_struct(obj: &Bound<'_, PyAny>, fields: &[StructField]) ->
PyResult<Value> {
let py = obj.py();
- let values = if let Ok(row) = obj.cast::<PyRow>() {
- let (_, values) = row.borrow().parts(py)?;
- values_by_position(py, values.iter().collect(), fields)?
- } else if let Ok(tuple) = obj.cast::<PyTuple>() {
+ let values = if let Ok(tuple) = obj.cast::<PyTuple>() {
values_by_position(py, tuple.iter().collect(), fields)?
} else if obj.is_instance_of::<PyDict>() {
values_by_name(obj, fields)?
@@ -1722,10 +1706,6 @@ pub(crate) fn row_values<'py>(
if item.is_none() {
return Ok(vec![py.None().into_bound(py); fields.len()]);
}
- if let Ok(row) = item.cast::<PyRow>() {
- let (_, values) = row.borrow().parts(py)?;
- return values_by_position(py, values.iter().collect(), fields);
- }
if let Ok(tuple) = item.cast::<PyTuple>() {
return values_by_position(py, tuple.iter().collect(), fields);
}
diff --git a/crates/pyspark-rs/src/lib.rs b/crates/pyspark-rs/src/lib.rs
index 084a4797..b6042714 100644
--- a/crates/pyspark-rs/src/lib.rs
+++ b/crates/pyspark-rs/src/lib.rs
@@ -74,7 +74,6 @@ use readwriter::PyDataFrameReader;
use resource::{
PyExecutorResourceRequests, PyResourceProfile, PyResourceProfileBuilder,
PyTaskResourceRequests,
};
-use row::PyRow;
use session::{PyHook, PySparkSession, PySparkSessionBuilder};
use stat::PyStatFunctions;
use streaming::{
@@ -109,7 +108,7 @@ fn _pyspark(_py: Python<'_>, m: &Bound<'_, PyModule>) ->
PyResult<()> {
m.add_class::<PyWhenNotMatched>()?;
m.add_class::<PyWhenNotMatchedBySource>()?;
m.add_class::<PyColumn>()?;
- m.add_class::<PyRow>()?;
+ row::register(m)?;
m.add_class::<PyGroupedData>()?;
m.add_class::<PyCoGroupedData>()?;
m.add_class::<PyLocalRowIterator>()?;
diff --git a/crates/pyspark-rs/src/row.py b/crates/pyspark-rs/src/row.py
new file mode 100644
index 00000000..6af09bec
--- /dev/null
+++ b/crates/pyspark-rs/src/row.py
@@ -0,0 +1,228 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Tuple-backed Row semantics from PySpark's ``pyspark.sql.types.Row``.
+
+Mirrors ``class Row`` in apache/spark v4.3.0, ``python/pyspark/sql/types.py``.
+Embedded by the extension so ``pyspark._pyspark.Row`` and the public imports
+refer to the same class without importing the partially initialized types shim.
+"""
+
+
+class Row(tuple):
+ """
+ A row in :class:`DataFrame`.
+ The fields in it can be accessed:
+
+ * like attributes (``row.key``)
+ * like dictionary values (``row[key]``)
+
+ ``key in row`` will search through row keys.
+
+ Row can be used to create a row object by using named arguments.
+ It is not allowed to omit a named argument to represent that the value is
+ None or missing. This should be explicitly set to None in this case.
+
+ .. versionchanged:: 3.0.0
+ Rows created from named arguments no longer have
+ field names sorted alphabetically and will be ordered in the position
as
+ entered.
+
+ Examples
+ --------
+ >>> from pyspark.sql import Row
+ >>> row = Row(name="Alice", age=11)
+ >>> row
+ Row(name='Alice', age=11)
+ >>> row['name'], row['age']
+ ('Alice', 11)
+ >>> row.name, row.age
+ ('Alice', 11)
+ >>> 'name' in row
+ True
+ >>> 'wrong_key' in row
+ False
+
+ Row also can be used to create another Row like class, then it
+ could be used to create Row objects, such as
+
+ >>> Person = Row("name", "age")
+ >>> Person
+ <Row('name', 'age')>
+ >>> 'name' in Person
+ True
+ >>> 'wrong_key' in Person
+ False
+ >>> Person("Alice", 11)
+ Row(name='Alice', age=11)
+
+ This form can also be used to create rows as tuple values, i.e. with
unnamed
+ fields.
+
+ >>> row1 = Row("Alice", 11)
+ >>> row2 = Row(name="Alice", age=11)
+ >>> row1 == row2
+ True
+ """
+
+ __module__ = "pyspark.sql.types"
+
+ def __new__(cls, *args, **kwargs):
+ if args and kwargs:
+ from pyspark.errors import PySparkValueError
+
+ raise PySparkValueError(
+ errorClass="CANNOT_SET_TOGETHER",
+ messageParameters={"arg_list": "args and kwargs"},
+ )
+ if kwargs:
+ row = tuple.__new__(cls, list(kwargs.values()))
+ row.__fields__ = list(kwargs.keys())
+ return row
+ return tuple.__new__(cls, args)
+
+ def asDict(self, recursive=False):
+ """
+ Return as a dict
+
+ Parameters
+ ----------
+ recursive : bool, optional
+ turns the nested Rows to dict (default: False).
+
+ Notes
+ -----
+ If a row contains duplicate field names, e.g., the rows of a join
+ between two :class:`DataFrame` that both have the fields of same names,
+ one of the duplicate fields will be selected by ``asDict``.
``__getitem__``
+ will also return one of the duplicate fields, however returned value
might
+ be different to ``asDict``.
+
+ Examples
+ --------
+ >>> from pyspark.sql import Row
+ >>> Row(name="Alice", age=11).asDict() == {'name': 'Alice', 'age': 11}
+ True
+ >>> row = Row(key=1, value=Row(name='a', age=2))
+ >>> row.asDict() == {'key': 1, 'value': Row(name='a', age=2)}
+ True
+ >>> row.asDict(True) == {'key': 1, 'value': {'name': 'a', 'age': 2}}
+ True
+ """
+ if not hasattr(self, "__fields__"):
+ from pyspark.errors import PySparkTypeError
+
+ raise PySparkTypeError(
+ errorClass="CANNOT_CONVERT_TYPE",
+ messageParameters={"from_type": "Row", "to_type": "dict"},
+ )
+ if recursive:
+
+ def conv(obj):
+ if isinstance(obj, Row):
+ return obj.asDict(True)
+ if isinstance(obj, list):
+ return [conv(value) for value in obj]
+ if isinstance(obj, dict):
+ return {key: conv(value) for key, value in obj.items()}
+ return obj
+
+ return dict(zip(self.__fields__, (conv(value) for value in self)))
+ return dict(zip(self.__fields__, self))
+
+ def __contains__(self, item):
+ if hasattr(self, "__fields__"):
+ return item in self.__fields__
+ return super().__contains__(item)
+
+ def __call__(self, *args):
+ """create new Row object"""
+ from pyspark.sql.types import _create_row
+
+ if len(args) > len(self):
+ from pyspark.errors import PySparkValueError
+
+ raise PySparkValueError(
+ errorClass="TOO_MANY_VALUES",
+ messageParameters={
+ "expected": str(len(self)),
+ "item": "fields",
+ "actual": str(len(args)),
+ },
+ )
+ return _create_row(self, args)
+
+ def __getitem__(self, item):
+ if isinstance(item, (int, slice)):
+ return super().__getitem__(item)
+ try:
+ index = self.__fields__.index(item)
+ return super().__getitem__(index)
+ except IndexError:
+ from pyspark.errors import PySparkKeyError
+
+ raise PySparkKeyError(errorClass="KEY_NOT_EXISTS",
messageParameters={"key": str(item)})
+ except ValueError:
+ from pyspark.errors import PySparkValueError
+
+ raise PySparkValueError(item)
+
+ def __getattr__(self, item):
+ if item.startswith("__"):
+ from pyspark.errors import PySparkAttributeError
+
+ raise PySparkAttributeError(
+ errorClass="ATTRIBUTE_NOT_SUPPORTED",
messageParameters={"attr_name": item}
+ )
+ try:
+ index = self.__fields__.index(item)
+ return self[index]
+ except IndexError:
+ from pyspark.errors import PySparkAttributeError
+
+ raise PySparkAttributeError(
+ errorClass="ATTRIBUTE_NOT_SUPPORTED",
messageParameters={"attr_name": item}
+ )
+ except ValueError:
+ from pyspark.errors import PySparkAttributeError
+
+ raise PySparkAttributeError(
+ errorClass="ATTRIBUTE_NOT_SUPPORTED",
messageParameters={"attr_name": item}
+ )
+
+ def __setattr__(self, key, value):
+ if key != "__fields__":
+ from pyspark.errors import PySparkRuntimeError
+
+ raise PySparkRuntimeError(errorClass="READ_ONLY",
messageParameters={"object": "Row"})
+ self.__dict__[key] = value
+
+ def __reduce__(self):
+ """Returns a tuple so Python knows how to pickle Row."""
+ if hasattr(self, "__fields__"):
+ from pyspark.sql.types import _create_row
+
+ return _create_row, (self.__fields__, tuple(self))
+ return tuple.__reduce__(self)
+
+ def __repr__(self):
+ """Printable representation of Row used in Python REPL."""
+ if hasattr(self, "__fields__"):
+ return "Row(%s)" % ", ".join(
+ "%s=%r" % (key, value) for key, value in zip(self.__fields__,
tuple(self))
+ )
+ return "<Row(%s)>" % ", ".join(repr(value) for value in self)
diff --git a/crates/pyspark-rs/src/row.rs b/crates/pyspark-rs/src/row.rs
index 75bd78af..0fbf771a 100644
--- a/crates/pyspark-rs/src/row.rs
+++ b/crates/pyspark-rs/src/row.rs
@@ -1,191 +1,123 @@
-//! PyO3 wrapper for spark_connect::row::Row, mirroring `pyspark.sql.Row`.
+//! Conversion of native Spark values into tuple-backed Python Rows.
+
+use std::collections::HashMap;
+use std::sync::Arc;
-use pyo3::basic::CompareOp;
-use pyo3::exceptions::PyIndexError;
use pyo3::prelude::*;
use pyo3::sync::PyOnceLock;
-use pyo3::types::{PyByteArray, PyBytes, PyDict, PyList, PySlice, PyString,
PyTuple};
-use spark_connect::row::{Row, Value};
-
-use crate::column::raise_pyspark;
-
-/// Python wrapper for a Spark Row. Behaves as `pyspark.sql.Row`, a `tuple`
subclass, does,
-/// though it cannot subclass `tuple` itself.
-#[pyclass(name = "Row", module = "pyspark.sql.types")]
-pub struct PyRow {
- data: RowData,
- /// Whether the row has field names (`__fields__`). As in PySpark, a row
made of values
- /// alone, such as `Row(1, 2)` or the row class `Row("name", "age")`, has
none; the rest
- /// of the client knows its fields as `_1`, `_2`, and so on.
- named: bool,
+use pyo3::types::{PyByteArray, PyBytes, PyDict, PyList, PyTuple};
+use spark_connect::row::{StructValue, Value};
+
+/// Functions that deserialize user-defined or spatial values in each result
field.
+pub(crate) type FieldConverters = Arc<Vec<Option<Py<PyAny>>>>;
+
+static CREATE_ROW: PyOnceLock<Py<PyAny>> = PyOnceLock::new();
+const ROW_SOURCE: &std::ffi::CStr = pyo3::ffi::c_str!(include_str!("row.py"));
+
+/// Expose the embedded tuple `Row` as `pyspark._pyspark.Row`, re-exported by
the Python shims.
+/// Its source imports PySpark modules only inside methods, avoiding import
cycles
+/// while `pyspark.sql.types` is importing this extension.
+/// The synthetic filename avoids embedding a build-machine path in
distributed wheels;
+/// the source is compiled into the extension and need not be installed
separately.
+pub(crate) fn register(module: &Bound<'_, PyModule>) -> PyResult<()> {
+ let row_module =
+ PyModule::from_code(module.py(), ROW_SOURCE, c"pyspark/_row.py",
c"pyspark._row")?;
+ module.add("Row", row_module.getattr("Row")?)
}
-/// The Python functions that turn the values of the fields of collected rows
whose types have
-/// user-defined types into what PySpark gives for them, by deserializing
those values; None
-/// for the other fields.
-pub(crate) type FieldConverters = std::sync::Arc<Vec<Option<Py<PyAny>>>>;
+/// One step of a nested schema path. Array elements and map keys/values carry
no
+/// index, so all of them share the names at their path.
+#[derive(Clone, Copy, Eq, Hash, PartialEq)]
+enum NestedFieldPath {
+ Field(usize),
+ ArrayElement,
+ MapKey,
+ MapValue,
+}
-enum RowData {
- /// A row that the server returned, whose Spark values become Python
values when first
- /// read, and stay those objects, as a PySpark row is a tuple of them: so
the row equals
- /// itself, with a NaN or a VARIANT value, and keeps its hash.
- Spark {
- row: Row,
- values: PyOnceLock<Py<PyTuple>>,
- /// The converters of its fields, when one has a user-defined type.
- converters: Option<FieldConverters>,
- /// Whether its binary values are `bytes`, or `bytearray`, as PySpark
gives them per
- /// `spark.sql.execution.pyspark.binaryAsBytes`.
- binary_as_bytes: bool,
- },
- /// A row made in Python, which holds its Python values as they are, as
PySpark's does.
- Python {
- fields: Vec<String>,
- values: Py<PyTuple>,
- },
+/// Mutable names for each nested schema path, shared across the result's
Arrow batches.
+/// Distinct sibling structs retain independent lists even when their names
are identical.
+#[derive(Default)]
+pub(crate) struct NestedFieldNames {
+ path: Vec<NestedFieldPath>,
+ fields: HashMap<Vec<NestedFieldPath>, Py<PyList>>,
}
-impl PyRow {
- pub fn new(row: Row) -> Self {
- Self::with_converters(row, None, true)
+impl NestedFieldNames {
+ /// Release names when an iterator closes, including when materialization
fails.
+ pub(crate) fn clear(&mut self) {
+ self.path.clear();
+ self.fields.clear();
}
- /// A row that the server returned, whose values of user-defined types
`converters`
- /// deserializes when they are first read, and whose binary values are
`bytes` if
- /// `binary_as_bytes`, or else `bytearray`.
- pub(crate) fn with_converters(
- row: Row,
- converters: Option<FieldConverters>,
+ /// Convert one schema step below this path, restoring it even if
conversion fails.
+ fn value_at<'py>(
+ &mut self,
+ py: Python<'py>,
+ value: &Value,
binary_as_bytes: bool,
- ) -> Self {
- PyRow {
- data: RowData::Spark {
- row,
- values: PyOnceLock::new(),
- converters,
- binary_as_bytes,
- },
- named: true,
- }
- }
-
- /// A row made in Python of `values`, named by `fields`.
- fn of_python(fields: Vec<String>, values: Bound<'_, PyTuple>, named: bool)
-> Self {
- PyRow {
- data: RowData::Python {
- fields,
- values: values.unbind(),
- },
- named,
- }
- }
-
- fn fields(&self) -> &[String] {
- match &self.data {
- RowData::Spark { row, .. } => row.fields(),
- RowData::Python { fields, .. } => fields,
- }
- }
-
- /// The values, as the tuple that a PySpark row is.
- fn values<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyTuple>> {
- match &self.data {
- RowData::Spark {
- row,
- values,
- converters,
- binary_as_bytes,
- } => {
- let values = values.get_or_try_init(py, || ->
PyResult<Py<PyTuple>> {
- let convert = |i: usize| {
- converters
- .as_ref()
- .and_then(|c| c.get(i))
- .and_then(Option::as_ref)
- };
- let values = row
- .values()
- .iter()
- .enumerate()
- .map(|(i, v)| {
- let value = value_to_py(py, v, *binary_as_bytes)?;
- match convert(i) {
- Some(converter) =>
converter.bind(py).call1((value,)),
- None => Ok(value),
- }
- })
- .collect::<PyResult<Vec<_>>>()?;
- Ok(PyTuple::new(py, values)?.unbind())
- })?;
- Ok(values.bind(py).clone())
- }
- RowData::Python { values, .. } => Ok(values.bind(py).clone()),
- }
- }
-
- /// The value of the field at `index`, which is in range.
- fn value<'py>(&self, py: Python<'py>, index: usize) -> PyResult<Bound<'py,
PyAny>> {
- match &self.data {
- RowData::Spark { .. } => self.values(py)?.get_item(index),
- RowData::Python { values, .. } => values.bind(py).get_item(index),
- }
+ path: NestedFieldPath,
+ ) -> PyResult<Bound<'py, PyAny>> {
+ self.path.push(path);
+ let result = value_to_py(py, value, binary_as_bytes, self);
+ self.path.pop();
+ result
}
- /// The value of the first field named `name`, if any.
- fn value_by_name<'py>(
- &self,
+ /// Get mutable names for this path, using the first struct seen to
initialize them.
+ fn names_at_path<'py>(
+ &mut self,
py: Python<'py>,
- name: &str,
- ) -> PyResult<Option<Bound<'py, PyAny>>> {
- match self.fields().iter().position(|field| field == name) {
- Some(index) => self.value(py, index).map(Some),
- None => Ok(None),
+ row: &StructValue,
+ ) -> PyResult<Bound<'py, PyList>> {
+ if let Some(fields) = self.fields.get(&self.path) {
+ return Ok(fields.bind(py).clone());
}
+ let fields = PyList::new(py, row.fields())?;
+ self.fields
+ .insert(self.path.clone(), fields.clone().unbind());
+ Ok(fields)
}
+}
- /// The field names, if the row has them (`__fields__`), and the values,
as the tuple
- /// that a PySpark row is, for reading the row as PySpark's schema
inference does.
- pub(crate) fn parts<'py>(
- &self,
- py: Python<'py>,
- ) -> PyResult<(Option<&[String]>, Bound<'py, PyTuple>)> {
- Ok((self.named.then(|| self.fields()), self.values(py)?))
- }
-
- /// The row with Spark values, for the rest of the client, which converts
the values of a
- /// row made in Python as it converts those of a tuple.
- pub(crate) fn to_row(&self, py: Python<'_>) -> PyResult<Row> {
- match &self.data {
- RowData::Spark { row, .. } => Ok(row.clone()),
- RowData::Python { fields, values } => {
- let values = values
- .bind(py)
- .iter()
- .map(|v| crate::session::py_to_value(&v))
- .collect::<PyResult<Vec<_>>>()?;
- Ok(Row::new(fields.clone(), values))
+/// Build a Row from named values, deserializing user-defined fields with
`converters`.
+/// The caller shares `fields` across result rows and nested names across the
execution.
+pub(crate) fn row_values_to_py(
+ py: Python<'_>,
+ fields: &Bound<'_, PyList>,
+ values: &[Value],
+ converters: Option<&FieldConverters>,
+ binary_as_bytes: bool,
+ nested_fields: &mut NestedFieldNames,
+) -> PyResult<Py<PyAny>> {
+ let values = values
+ .iter()
+ .enumerate()
+ .map(|(index, value)| {
+ let value = nested_fields.value_at(
+ py,
+ value,
+ binary_as_bytes,
+ NestedFieldPath::Field(index),
+ )?;
+ match converters
+ .and_then(|converters| converters.get(index))
+ .and_then(Option::as_ref)
+ {
+ Some(converter) => converter.bind(py).call1((value,)),
+ None => Ok(value),
}
- }
- }
-
- /// What PySpark raises for an attribute that a row does not have.
- fn no_attribute(py: Python<'_>, name: &str) -> PyErr {
- raise_pyspark(
- py,
- "PySparkAttributeError",
- "ATTRIBUTE_NOT_SUPPORTED",
- &[("attr_name", name)],
- )
- }
-
- /// The values of `other` as a tuple, if it is a row or a tuple, as a
tuple operation
- /// takes them.
- fn tuple_of<'py>(other: &Bound<'py, PyAny>) -> PyResult<Option<Bound<'py,
PyTuple>>> {
- if let Ok(row) = other.cast::<PyRow>() {
- return row.borrow().values(other.py()).map(Some);
- }
- Ok(other.cast::<PyTuple>().ok().cloned())
- }
+ })
+ .collect::<PyResult<Vec<_>>>()?;
+ let create_row = CREATE_ROW.get_or_try_init(py, || {
+ py.import("pyspark.sql.types")?
+ .getattr("_create_row")
+ .map(Bound::unbind)
+ })?;
+ Ok(create_row
+ .bind(py)
+ .call1((fields, PyTuple::new(py, values)?))?
+ .unbind())
}
/// `datetime.datetime(1970, 1, 1)`, the epoch of Spark's timestamps, without
a time zone.
@@ -219,10 +151,13 @@ fn micros_timedelta(py: Python<'_>, micros: i64) ->
PyResult<Bound<'_, PyAny>> {
/// Convert a Spark `Value` into a Python object (PyO3 0.28 `IntoPyObject`),
whose binary values
/// are `bytes` if `binary_as_bytes`, or else `bytearray`, also in arrays,
maps and structs.
-pub fn value_to_py<'py>(
+/// `nested_fields` supplies shared struct names. Call through
`NestedFieldNames::value_at`
+/// so its schema path is set for this value and restored after conversion.
+fn value_to_py<'py>(
py: Python<'py>,
v: &Value,
binary_as_bytes: bool,
+ nested_fields: &mut NestedFieldNames,
) -> PyResult<Bound<'py, PyAny>> {
// Propagate conversion failures as Python exceptions with `?` rather than
// panicking across the FFI boundary (the conversions are near-infallible,
@@ -304,7 +239,12 @@ pub fn value_to_py<'py>(
Value::List(items) => {
let list = PyList::empty(py);
for it in items {
- list.append(value_to_py(py, it, binary_as_bytes)?)?;
+ list.append(nested_fields.value_at(
+ py,
+ it,
+ binary_as_bytes,
+ NestedFieldPath::ArrayElement,
+ )?)?;
}
list.into_any()
}
@@ -314,21 +254,24 @@ pub fn value_to_py<'py>(
let dict = PyDict::new(py);
for (k, val) in m {
dict.set_item(
- value_to_py(py, k, binary_as_bytes)?,
- value_to_py(py, val, binary_as_bytes)?,
+ nested_fields.value_at(py, k, binary_as_bytes,
NestedFieldPath::MapKey)?,
+ nested_fields.value_at(py, val, binary_as_bytes,
NestedFieldPath::MapValue)?,
)?;
}
dict.into_any()
}
Value::Struct(row) => {
- // A struct value materializes as a Row (named fields), matching
PySpark,
- // rather than a plain dict. Cloning keeps shared names and
independent values.
- Py::new(
+ // A struct value materializes as a Row with the mutable names of
its schema path.
+ let fields = nested_fields.names_at_path(py, row)?;
+ row_values_to_py(
py,
- PyRow::with_converters(row.clone().into(), None,
binary_as_bytes),
+ &fields,
+ row.values(),
+ None,
+ binary_as_bytes,
+ nested_fields,
)?
.into_bound(py)
- .into_any()
}
Value::Variant { value, metadata } => {
// A VARIANT materializes as a VariantVal (toJson/toPython decode
lazily),
@@ -342,307 +285,3 @@ pub fn value_to_py<'py>(
}
})
}
-
-#[pymethods]
-impl PyRow {
- /// Construct a Row, mirroring `pyspark.sql.Row`: `Row(name=value, ...)`
makes a row with
- /// field names, and `Row(v1, v2, ...)` one of values alone, which can
also serve as a row
- /// class: `Row("name", "age")("Alice", 11)`.
- #[new]
- #[pyo3(signature = (*args, **kwargs))]
- fn __new__(
- py: Python<'_>,
- args: &Bound<'_, PyTuple>,
- kwargs: Option<&Bound<'_, PyDict>>,
- ) -> PyResult<Self> {
- let kwargs = kwargs.filter(|kw| !kw.is_empty());
- if let Some(kw) = kwargs {
- if !args.is_empty() {
- return Err(raise_pyspark(
- py,
- "PySparkValueError",
- "CANNOT_SET_TOGETHER",
- &[("arg_list", "args and kwargs")],
- ));
- }
- let mut fields = Vec::with_capacity(kw.len());
- let mut values = Vec::with_capacity(kw.len());
- for (k, v) in kw.iter() {
- fields.push(k.extract::<String>()?);
- values.push(v);
- }
- Ok(PyRow::of_python(fields, PyTuple::new(py, values)?, true))
- } else {
- let fields = (0..args.len()).map(|i| format!("_{}", i +
1)).collect();
- Ok(PyRow::of_python(fields, args.clone(), false))
- }
- }
-
- /// Number of fields.
- fn __len__(&self) -> usize {
- self.fields().len()
- }
-
- fn __iter__<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py,
pyo3::types::PyIterator>> {
- self.values(py)?.try_iter()
- }
-
- fn __reversed__<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py,
PyAny>> {
- py.import("builtins")?
- .getattr("reversed")?
- .call1((self.values(py)?,))
- }
-
- /// `x in row` tests the field names of a row that has them, and the
values otherwise.
- fn __contains__(&self, py: Python<'_>, item: &Bound<'_, PyAny>) ->
PyResult<bool> {
- if self.named {
- return PyList::new(py, self.fields())?.contains(item);
- }
- self.values(py)?.contains(item)
- }
-
- /// An index or a slice, as of a tuple, or a field name.
- fn __getitem__<'py>(
- &self,
- py: Python<'py>,
- key: &Bound<'py, PyAny>,
- ) -> PyResult<Bound<'py, PyAny>> {
- if key.cast::<PySlice>().is_ok() {
- return self.values(py)?.as_any().get_item(key);
- }
- if let Ok(mut i) = key.extract::<isize>() {
- let n = self.fields().len() as isize;
- if i < 0 {
- i += n;
- }
- if i < 0 || i >= n {
- return Err(PyIndexError::new_err("tuple index out of range"));
- }
- return self.value(py, i as usize);
- }
- // As PySpark looks the key up in `__fields__`, which a row without
names lacks.
- if !self.named {
- return Err(Self::no_attribute(py, "__fields__"));
- }
- if let Ok(name) = key.cast::<PyString>() {
- if let Some(value) = self.value_by_name(py, &name.to_cow()?)? {
- return Ok(value);
- }
- }
- let error = py.import("pyspark.errors")?.getattr("PySparkValueError")?;
- Err(PyErr::from_value(error.call1((key,))?))
- }
-
- /// Attribute access by field name (`row.id`).
- fn __getattr__<'py>(&self, py: Python<'py>, name: &str) ->
PyResult<Bound<'py, PyAny>> {
- if !name.starts_with("__") && self.named {
- if let Some(value) = self.value_by_name(py, name)? {
- return Ok(value);
- }
- }
- Err(Self::no_attribute(py, name))
- }
-
- /// Only `__fields__` can be set, which names the fields, as in PySpark.
- fn __setattr__(
- &mut self,
- py: Python<'_>,
- name: &str,
- value: &Bound<'_, PyAny>,
- ) -> PyResult<()> {
- if name != "__fields__" {
- return Err(raise_pyspark(
- py,
- "PySparkRuntimeError",
- "READ_ONLY",
- &[("object", "Row")],
- ));
- }
- let mut names = value
- .try_iter()?
- .map(|name| name?.str().map(|name| name.to_string()))
- .collect::<PyResult<Vec<_>>>()?;
- // PySpark pairs names with values as `zip` does; fields beyond the
names keep theirs.
- let fields = self.fields();
- names.truncate(fields.len());
- names.extend(fields[names.len()..].iter().cloned());
- match &mut self.data {
- RowData::Spark { row, .. } => *row = Row::new(names,
row.values().to_vec()),
- RowData::Python { fields, .. } => *fields = names,
- }
- self.named = true;
- Ok(())
- }
-
- /// Field names, in order: a list attribute, as PySpark's `Row.__fields__`
is. A row
- /// without names has none.
- #[getter]
- fn __fields__(&self, py: Python<'_>) -> PyResult<Vec<String>> {
- if !self.named {
- return Err(Self::no_attribute(py, "__fields__"));
- }
- Ok(self.fields().to_vec())
- }
-
- /// As a dict of field name -> value (mirrors `Row.asDict`), with nested
rows, also in
- /// lists and dicts, as dicts if `recursive`.
- #[pyo3(name = "asDict")]
- #[pyo3(signature = (recursive=false))]
- fn as_dict<'py>(&self, py: Python<'py>, recursive: bool) ->
PyResult<Bound<'py, PyDict>> {
- fn conv<'py>(obj: Bound<'py, PyAny>) -> PyResult<Bound<'py, PyAny>> {
- let py = obj.py();
- if let Ok(row) = obj.cast::<PyRow>() {
- return Ok(row.borrow().as_dict(py, true)?.into_any());
- }
- if let Ok(list) = obj.cast::<PyList>() {
- let items =
list.iter().map(conv).collect::<PyResult<Vec<_>>>()?;
- return Ok(PyList::new(py, items)?.into_any());
- }
- if let Ok(dict) = obj.cast::<PyDict>() {
- let converted = PyDict::new(py);
- for (k, v) in dict.iter() {
- converted.set_item(k, conv(v)?)?;
- }
- return Ok(converted.into_any());
- }
- Ok(obj)
- }
- if !self.named {
- return Err(raise_pyspark(
- py,
- "PySparkTypeError",
- "CANNOT_CONVERT_TYPE",
- &[("from_type", "Row"), ("to_type", "dict")],
- ));
- }
- let dict = PyDict::new(py);
- for (name, value) in self.fields().iter().zip(self.values(py)?.iter())
{
- dict.set_item(name, if recursive { conv(value)? } else { value })?;
- }
- Ok(dict)
- }
-
- /// A row class, such as `Row("name", "age")`, makes a row of `args` named
by its values.
- #[pyo3(signature = (*args))]
- fn __call__(&self, py: Python<'_>, args: &Bound<'_, PyTuple>) ->
PyResult<PyRow> {
- if args.len() > self.fields().len() {
- let (expected, actual) = (self.fields().len().to_string(),
args.len().to_string());
- return Err(raise_pyspark(
- py,
- "PySparkValueError",
- "TOO_MANY_VALUES",
- &[
- ("expected", expected.as_str()),
- ("item", "fields"),
- ("actual", actual.as_str()),
- ],
- ));
- }
- let fields = self
- .values(py)?
- .iter()
- .take(args.len())
- .map(|name| name.str().map(|name| name.to_string()))
- .collect::<PyResult<Vec<_>>>()?;
- Ok(PyRow::of_python(fields, args.clone(), true))
- }
-
- /// Pickles as PySpark's row does: with `_create_row` from the field names
and values, or
- /// as `Row(*values)` for a row without names.
- fn __reduce__<'py>(slf: &Bound<'py, Self>) -> PyResult<Bound<'py,
PyTuple>> {
- let py = slf.py();
- let row = slf.borrow();
- let values = row.values(py)?;
- if row.named {
- let create_row =
py.import("pyspark.sql.types")?.getattr("_create_row")?;
- let args = (PyList::new(py, row.fields())?,
values).into_pyobject(py)?;
- return (create_row, args).into_pyobject(py);
- }
- (slf.get_type(), values).into_pyobject(py)
- }
-
- /// `Row(name='Alice', age=11)`, with the `repr()` of each value, or
`<Row('a', 'b')>` for
- /// a row without names, as PySpark prints a row.
- fn __repr__(&self, py: Python<'_>) -> PyResult<String> {
- let values = self.values(py)?;
- let mut parts = Vec::with_capacity(values.len());
- if self.named {
- for (name, value) in self.fields().iter().zip(values.iter()) {
- parts.push(format!("{}={}", name, value.repr()?));
- }
- return Ok(format!("Row({})", parts.join(", ")));
- }
- for value in values.iter() {
- parts.push(value.repr()?.to_string());
- }
- Ok(format!("<Row({})>", parts.join(", ")))
- }
-
- /// Hashes as the tuple of its values does.
- fn __hash__(&self, py: Python<'_>) -> PyResult<isize> {
- self.values(py)?.hash()
- }
-
- /// Compares as the tuple of its values does, with another row or a tuple.
- fn __richcmp__<'py>(
- &self,
- other: &Bound<'py, PyAny>,
- op: CompareOp,
- ) -> PyResult<Bound<'py, PyAny>> {
- let py = other.py();
- match Self::tuple_of(other)? {
- Some(other) => self.values(py)?.rich_compare(other, op),
- None => Ok(py.NotImplemented().into_bound(py)),
- }
- }
-
- /// Concatenates as a tuple does, into a tuple.
- fn __add__<'py>(&self, other: &Bound<'py, PyAny>) -> PyResult<Bound<'py,
PyAny>> {
- let py = other.py();
- match Self::tuple_of(other)? {
- Some(other) => self.values(py)?.as_any().add(other),
- None => Ok(py.NotImplemented().into_bound(py)),
- }
- }
-
- fn __radd__<'py>(&self, other: &Bound<'py, PyAny>) -> PyResult<Bound<'py,
PyAny>> {
- let py = other.py();
- match Self::tuple_of(other)? {
- Some(other) => other.as_any().add(self.values(py)?),
- None => Ok(py.NotImplemented().into_bound(py)),
- }
- }
-
- /// Repeats as a tuple does, into a tuple.
- fn __mul__<'py>(&self, times: &Bound<'py, PyAny>) -> PyResult<Bound<'py,
PyAny>> {
- self.values(times.py())?.as_any().mul(times)
- }
-
- fn __rmul__<'py>(&self, times: &Bound<'py, PyAny>) -> PyResult<Bound<'py,
PyAny>> {
- self.__mul__(times)
- }
-
- /// Number of occurrences of `value` among the row's values (tuple.count).
- fn count(&self, py: Python<'_>, value: &Bound<'_, PyAny>) ->
PyResult<usize> {
- self.values(py)?
- .as_any()
- .call_method1("count", (value,))?
- .extract()
- }
-
- /// Index of the first occurrence of `value` among the row's values
(tuple.index).
- #[pyo3(signature = (value, *args))]
- fn index(
- &self,
- py: Python<'_>,
- value: &Bound<'_, PyAny>,
- args: &Bound<'_, PyTuple>,
- ) -> PyResult<usize> {
- let mut call = vec![value.clone()];
- call.extend(args.iter());
- self.values(py)?
- .as_any()
- .call_method1("index", PyTuple::new(py, call)?)?
- .extract()
- }
-}
diff --git a/crates/pyspark-rs/src/session.rs b/crates/pyspark-rs/src/session.rs
index 6a42d5dc..ef75ad9e 100644
--- a/crates/pyspark-rs/src/session.rs
+++ b/crates/pyspark-rs/src/session.rs
@@ -717,8 +717,7 @@ impl PySparkSession {
} else {
// A VariantVal of PySpark has a `__dict__`, so that it is taken
as a row, which
// PySpark's `to_row` rejects, as `row_values` does.
- !(first.is_instance_of::<crate::row::PyRow>()
- || first.is_instance_of::<pyo3::types::PyTuple>()
+ !(first.is_instance_of::<pyo3::types::PyTuple>()
|| first.is_instance_of::<PyList>()
|| first.is_instance_of::<crate::values::PyVariantVal>()
|| first.hasattr("__dict__")?)
@@ -3048,19 +3047,31 @@ pub(crate) fn py_to_value(obj: &Bound<'_, PyAny>) ->
PyResult<Value> {
return Ok(Value::List(items));
}
if let Ok(tuple) = obj.cast::<pyo3::types::PyTuple>() {
- let mut items = Vec::with_capacity(tuple.len());
- for item in tuple.iter() {
- items.push(py_to_value(&item)?);
+ let mut items = tuple
+ .iter()
+ .map(|item| py_to_value(&item))
+ .collect::<PyResult<Vec<_>>>()?;
+ // Named Rows (tuple subclasses with `__fields__`) become structs on
the generic
+ // object path, e.g. customized pandas object columns. Unnamed tuples
stay arrays.
+ if !obj.is_exact_instance_of::<pyo3::types::PyTuple>() {
+ match obj.getattr(pyo3::intern!(obj.py(), "__fields__")) {
+ Ok(fields) => {
+ let mut fields: Vec<String> = fields.extract()?;
+ // Row factories may receive fewer values than names, and
__fields__
+ // is mutable. Pair them as schema inference does, without
panicking
+ // in the native Row constructor when their lengths differ.
+ fields.truncate(items.len());
+ items.truncate(fields.len());
+ return Ok(Value::Struct(Row::new(fields, items).into()));
+ }
+ Err(error)
+ if
error.is_instance_of::<pyo3::exceptions::PyAttributeError>(obj.py()) => {}
+ Err(error) => return Err(error),
+ }
}
return Ok(Value::List(items));
}
- // pyspark Row -> struct: its named fields (values already core Values),
so a nested
- // Row inside createDataFrame data (e.g. Row(s=Row(x=1))) converts
recursively.
- if let Ok(pyrow) = obj.extract::<PyRef<crate::row::PyRow>>() {
- return Ok(Value::Struct(pyrow.to_row(obj.py())?.into()));
- }
-
// dict -> map, its entries in order with keys and values of their own
types
if let Ok(dict) = obj.cast::<pyo3::types::PyDict>() {
let mut map = Vec::with_capacity(dict.len());
diff --git a/crates/spark-connect/src/row.rs b/crates/spark-connect/src/row.rs
index 06edcadd..ec9f22c8 100644
--- a/crates/spark-connect/src/row.rs
+++ b/crates/spark-connect/src/row.rs
@@ -428,6 +428,17 @@ impl Row {
self.fields.as_slice()
}
+ /// The immutable field-name allocation shared by rows of an Arrow record
batch.
+ /// Language bindings use its identity to preserve per-batch field-name
sharing.
+ /// Locally constructed rows may own their names instead.
+ #[doc(hidden)]
+ pub fn shared_fields(&self) -> Option<&Arc<[String]>> {
+ match &self.fields {
+ RowFields::Shared(fields) => Some(fields),
+ RowFields::Owned(_) => None,
+ }
+ }
+
/// Get values.
pub fn values(&self) -> &[Value] {
&self.values
@@ -493,6 +504,18 @@ mod tests {
assert!(std::ptr::eq(row.fields(), cloned.fields()));
}
+ #[test]
+ fn shared_fields_exposes_only_shared_name_storage() {
+ let fields: Arc<[String]> = vec!["id".to_string()].into();
+ let row = Row::with_shared_fields(Arc::clone(&fields),
vec![Value::Long(1)]);
+ let cloned = row.clone();
+
+ assert!(Arc::ptr_eq(row.shared_fields().unwrap(), &fields));
+ assert!(Arc::ptr_eq(cloned.shared_fields().unwrap(), &fields));
+ let owned = Row::new(vec!["id".to_string()], vec![Value::Long(1)]);
+ assert!(owned.shared_fields().is_none());
+ }
+
#[test]
fn shared_field_names_preserve_order_duplicates_equality_and_debug() {
let names = vec![
diff --git a/pyproject.toml b/pyproject.toml
index e6f77f83..0fb7b92f 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -44,18 +44,24 @@ Repository = "https://github.com/apache/spark-connect-rust"
"Bug Tracker" = "https://issues.apache.org/jira/browse/SPARK"
[tool.ruff]
-# Lint only our authored tooling/scripts. The python/pyspark tree is largely
vendored
-# from Apache Spark (pandas-on-Spark, the drop-in skin) and must stay
-# byte-compatible with upstream, so it is intentionally excluded from our
linter.
+# Lint our authored tooling/scripts and the embedded Row implementation.
+# The python/pyspark tree is largely vendored from Apache Spark
(pandas-on-Spark
+# and the drop-in skin). It must stay byte-compatible with upstream and is
excluded
+# from this linter.
line-length = 100
target-version = "py39"
extend-exclude = ["python/pyspark", "target", ".claude"]
-include = ["scripts/**/*.py"]
+include = ["scripts/**/*.py", "crates/pyspark-rs/src/row.py"]
[tool.ruff.lint]
select = ["E", "F", "W", "I", "UP", "B"]
ignore = ["E501"] # line length handled by the formatter, not enforced hard
+[tool.ruff.lint.per-file-ignores]
+# Keep the upstream Row implementation's implicit exception context and percent
+# formatting so error behavior and source comparisons stay faithful to PySpark
4.3.0.
+"crates/pyspark-rs/src/row.py" = ["B904", "UP031"]
+
[tool.maturin]
# Mixed Python/Rust layout: the pure-Python drop-in skin lives under
python/pyspark,
# and the PyO3 extension is placed inside it as pyspark._pyspark.
diff --git a/python/.coveragerc b/python/.coveragerc
index 284e3c6e..426e1c51 100644
--- a/python/.coveragerc
+++ b/python/.coveragerc
@@ -3,6 +3,9 @@
# the Python glue only, excluding vendored third-party code (cloudpickle), the
# test-support utilities, pandas-on-spark, generated wrappers, and the compiled
# extension itself.
+# The embedded crates/pyspark-rs/src/row.py runs as pyspark/_row.py.
+# That path has no file and is explicitly omitted. Row tests exercise its
behavior;
+# Ruff checks the embedded source.
[run]
source = pyspark
branch = True
@@ -62,7 +65,10 @@ omit =
*/pyspark/sql/connect/logging.py
*/pyspark/sql/connect/_typing.py
*/pyspark/sql/connect/conversion.py
+ # The native extension and its embedded Row source, which has a synthetic
filename.
*/pyspark/_pyspark*
+ */pyspark/_row.py
+ # Test helpers.
*/tests/*
[report]
diff --git a/python/pyspark/sql/types.py b/python/pyspark/sql/types.py
index 33a92e30..09f1df76 100644
--- a/python/pyspark/sql/types.py
+++ b/python/pyspark/sql/types.py
@@ -21,10 +21,11 @@ try:
# Spark's nanosecond timestamp types (no PySpark 4.3.0 class; not in
__all__).
TimestampNTZNanosType, TimestampLTZNanosType,
)
- # Row lives in the Rust core; upstream code imports it from
pyspark.sql.types too.
+ # The extension exports the same tuple-backed Row as the public Python
imports.
from pyspark._pyspark import Row
except ImportError: # pragma: no cover - defensive fallback when the
extension is absent
- # Fallback: define minimal stubs for testing purposes
+ # Minimal DataType stubs for import-only tests. Row and runtime conversion
+ # helpers require the compiled extension.
class DataType:
"""Base class for all data types."""
def __init__(self):
@@ -614,12 +615,12 @@ def _udt_converter(dataType):
fields = _udt_field_converters(dataType)
if fields is None:
return None
- row = Row(*dataType.names)
+ field_names = dataType.names
def convert_struct(value):
if value is None:
return None
- return row(*[v if f is None else f(v) for f, v in zip(fields,
value)])
+ return _create_row(field_names, [v if f is None else f(v) for f, v
in zip(fields, value)])
return convert_struct
return None
@@ -710,7 +711,6 @@ def _struct_to_internal(dataType, obj):
if isinstance(obj, VariantVal):
raise PySparkValueError("Rows cannot be of type VariantVal")
names, fields = dataType.names, dataType.fields
- # A Row is a tuple in PySpark.
if dataType._needSerializeAnyField:
# Only calling toInternal function for fields that need conversion
need = dataType._needConversion
@@ -719,7 +719,7 @@ def _struct_to_internal(dataType, obj):
f.toInternal(obj.get(n)) if c else obj.get(n)
for n, f, c in zip(names, fields, need)
)
- elif isinstance(obj, (tuple, list, Row)):
+ elif isinstance(obj, (tuple, list)):
return tuple(f.toInternal(v) if c else v for f, v, c in
zip(fields, obj, need))
elif hasattr(obj, "__dict__"):
d = obj.__dict__
@@ -729,7 +729,7 @@ def _struct_to_internal(dataType, obj):
else:
if isinstance(obj, dict):
return tuple(obj.get(n) for n in names)
- elif isinstance(obj, (list, tuple, Row)):
+ elif isinstance(obj, (list, tuple)):
return tuple(obj)
elif hasattr(obj, "__dict__"):
d = obj.__dict__
@@ -822,8 +822,7 @@ def _from_internal(dataType, obj):
# The private helpers of PySpark's pyspark.sql.types, as of Apache Spark
v4.3.0: the inference,
# merging, conversion and verification of the types of Python objects, and
their tables, which
# PySpark Connect's modules, Apache Spark's tests and libraries import. They
are PySpark's, as
-# the client's types share the classes and the attributes of PySpark's; but
the client's Row is
-# not a tuple, so where PySpark takes a tuple of a Row, they take the client's
Row too.
+# the client's types share the classes and attributes of PySpark's, including
its tuple Row.
import ctypes # noqa: E402
import datetime # noqa: E402
import decimal # noqa: E402
@@ -1158,7 +1157,7 @@ def _infer_schema(
if isinstance(row, dict):
items = sorted(row.items())
- elif isinstance(row, (tuple, list, Row)):
+ elif isinstance(row, (tuple, list)):
if hasattr(row, "__fields__"): # Row
items = zip(row.__fields__, tuple(row))
elif hasattr(row, "_fields"): # namedtuple
@@ -1369,7 +1368,7 @@ def _create_converter(dataType: DataType) -> Callable:
if obj is None:
return None
- if isinstance(obj, (tuple, list, Row)):
+ if isinstance(obj, (tuple, list)):
if convert_fields:
return tuple(conv(v) for v, conv in zip(obj, converters))
else:
@@ -1413,7 +1412,7 @@ _acceptable_types = {
DayTimeIntervalType: (datetime.timedelta,),
ArrayType: (list, tuple, array),
MapType: (dict,),
- StructType: (tuple, list, dict, Row),
+ StructType: (tuple, list, dict),
GeometryType: (Geometry,),
GeographyType: (Geography,),
VariantType: (
@@ -1428,7 +1427,6 @@ _acceptable_types = {
datetime.datetime,
datetime.timedelta,
tuple,
- Row,
list,
dict,
array,
@@ -1714,7 +1712,7 @@ def _make_type_verifier(
if isinstance(obj, dict):
for f, verifier in verifiers:
verifier(obj.get(f))
- elif isinstance(obj, (tuple, list, Row)):
+ elif isinstance(obj, (tuple, list)):
if len(obj) != len(verifiers):
if name is not None:
raise PySparkValueError(
diff --git a/python/tests/test_arrow_handoff.py
b/python/tests/test_arrow_handoff.py
index 91422b7c..70f20f2c 100644
--- a/python/tests/test_arrow_handoff.py
+++ b/python/tests/test_arrow_handoff.py
@@ -249,6 +249,98 @@ def
test_collect_reads_binary_default_with_arrow_only_schema(protocol):
assert calls == {"execute": 1, "config": 1}
[email protected](
+ "action",
+ ["collect", "take", "tail", "first", "head", "head_many", "iterator",
"prefetch"],
+)
+def test_native_result_actions_return_tuple_rows_recursively(protocol, action):
+ from pyspark import Row
+
+ child_type = pa.struct([pa.field("value", pa.int64())])
+ batch = pa.RecordBatch.from_arrays(
+ [
+ pa.array([1, 2], type=pa.int64()),
+ pa.array([{"value": 3}, None], type=child_type),
+ pa.array([[{"value": 4}, None], []], type=pa.list_(child_type)),
+ pa.array([[("child", {"value": 5})], []],
type=pa.map_(pa.string(), child_type)),
+ ],
+ names=["id", "child", "children", "by_name"],
+ )
+ with _dataframe(protocol=protocol, batches=[batch]) as (frame, _):
+ if action in ("collect", "first", "head"):
+ result = getattr(frame, action)()
+ rows = result if action == "collect" else [result]
+ elif action in ("take", "tail"):
+ rows = getattr(frame, action)(2)
+ elif action == "head_many":
+ rows = frame.head(2)
+ else:
+ rows = list(frame.toLocalIterator(prefetchPartitions=action ==
"prefetch"))
+
+ assert rows and all(type(row) is Row and isinstance(row, tuple) for row in
rows)
+ row = rows[0]
+ assert type(row.child) is type(row.children[0]) is
type(row.by_name["child"]) is Row
+ assert tuple.__getitem__(row.child, 0) == 3
+ assert row.children[1] is None
+ assert row.asDict(recursive=True) == {
+ "id": 1,
+ "child": {"value": 3},
+ "children": [{"value": 4}, None],
+ "by_name": {"child": {"value": 5}},
+ }
+ if len(rows) == 2:
+ assert rows[1] == (2, None, [], {})
+
+
+def test_native_collect_rows_share_mutable_field_names(protocol):
+ batch = pa.RecordBatch.from_arrays([pa.array([1, 2])], names=["value"])
+ with _dataframe(protocol=protocol, batches=[batch.slice(0, 1),
batch.slice(1, 1)]) as (frame, _):
+ rows = frame.collect()
+
+ assert rows[0].__fields__ is rows[1].__fields__
+ rows[0].__fields__[0] = "renamed"
+ assert rows[1].renamed == 2
+ assert rows[1].asDict() == {"renamed": 2}
+
+
[email protected]("prefetch", [False, True])
[email protected]("one_response", [False, True])
+def test_native_iterator_rows_share_field_names_only_within_each_arrow_batch(
+ protocol, prefetch, one_response
+):
+ batch = pa.RecordBatch.from_arrays([pa.array([1, 2, 3, 4])],
names=["value"])
+ batches = [batch.slice(0, 2), batch.slice(2, 2)]
+ payloads = None
+ if one_response:
+ sink = pa.BufferOutputStream()
+ with pa.ipc.new_stream(sink, batch.schema) as writer:
+ for part in batches:
+ writer.write_batch(part)
+ payloads = [(batch.num_rows, sink.getvalue().to_pybytes())]
+ with _dataframe(protocol=protocol, batches=batches, payloads=payloads) as
(frame, _):
+ rows = list(frame.toLocalIterator(prefetchPartitions=prefetch))
+
+ assert rows[0].__fields__ is rows[1].__fields__
+ assert rows[2].__fields__ is rows[3].__fields__
+ assert rows[0].__fields__ is not rows[2].__fields__
+ rows[0].__fields__[0] = "renamed"
+ assert rows[1].renamed == 2
+ assert rows[2].value == 3
+ assert rows[3].asDict() == {"value": 4}
+
+
+def test_native_iterator_ends_after_row_materialization_fails(protocol):
+ # The first timestamp is year 10000, outside Python datetime's range.
+ batch = pa.RecordBatch.from_arrays(
+ [pa.array([253402300800000000, 0], type=pa.timestamp("us"))],
names=["value"]
+ )
+ with _dataframe(protocol=protocol, batches=[batch]) as (frame, _):
+ rows = frame.toLocalIterator()
+ with pytest.raises(OverflowError):
+ next(rows)
+ assert list(rows) == []
+
+
def test_handoff_preserves_values_metadata_and_chunks(protocol):
batch = _mixed_batch()
batches = [batch.slice(1, 2), batch.slice(0, 1), batch.slice(3, 1)]
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index 86dd2d65..a85ed8b0 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -248,8 +248,7 @@ def test_spatial_and_collated_json_forms_parse():
def test_private_type_helpers_are_those_of_pyspark():
# As PySpark's pyspark.sql.types: the inference, merging, conversion and
verification of the
- # types of Python objects, and their tables. They take the client's Row as
PySpark's take
- # theirs, a tuple.
+ # types of Python objects and their tables, including tuple-based Rows.
from array import array
from pyspark.errors import PySparkTypeError, PySparkValueError
@@ -358,8 +357,7 @@ def _type_verifier_cases():
def test_type_verifiers_take_and_reject_the_values_of_pyspark():
# As Apache Spark's DataTypeVerificationTests: _make_type_verifier takes
the values of each
- # type, of the client's Row too, and raises the errors of PySpark for the
others, with the
- # names of the fields.
+ # type, including Rows, and raises the errors of PySpark for the others
with field names.
from pyspark.errors import PySparkTypeError, PySparkValueError
from pyspark.sql.types import _make_type_verifier
@@ -383,7 +381,7 @@ def
test_type_verifiers_take_and_reject_the_values_of_pyspark():
def
test_types_of_python_objects_are_inferred_merged_and_converted_as_pyspark():
# As PySpark's _infer_type, _infer_schema, _merge_type and
_create_converter, of the
- # values of each type, also nested, of the client's Row too.
+ # values of each type, including nested Rows.
import datetime
import decimal
@@ -4308,6 +4306,243 @@ def test_row_behaves_as_a_tuple_of_its_values():
assert pickle.loads(pickle.dumps(Row(1, 2))) == (1, 2)
+def test_row_is_the_public_tuple_subclass():
+ from pyspark import Row as root_row
+ from pyspark._pyspark import Row as extension_row
+ from pyspark.sql import Row
+ from pyspark.sql.types import Row as types_row
+
+ assert root_row is extension_row is Row is types_row
+ assert Row.__module__ == "pyspark.sql.types"
+ assert Row.__bases__ == (tuple,)
+ row = Row(a=1, child=Row(b=2))
+ assert isinstance(row, tuple)
+ assert tuple.__getitem__(row, 0) == 1
+ assert json.loads(json.dumps(row)) == [1, [2]]
+
+ class DerivedRow(Row):
+ def total(self):
+ return sum(self)
+
+ derived = DerivedRow(a=1, b=2)
+ assert type(derived) is DerivedRow
+ assert isinstance(derived, Row) and isinstance(derived, tuple)
+ assert derived.total() == 3 and derived.asDict() == {"a": 1, "b": 2}
+
+
+def test_row_field_names_keep_their_identity_and_mutations():
+ from pyspark.errors import PySparkKeyError
+ from pyspark.sql import Row
+ from pyspark.sql.types import _create_row
+
+ row = Row(a=1, b=2)
+ fields = row.__fields__
+ assert row.__fields__ is fields
+ fields[0] = "renamed"
+ assert row.renamed == 1 and "a" not in row
+ assert row.asDict() == {"renamed": 1, "b": 2}
+ assert repr(row) == "Row(renamed=1, b=2)"
+ fields.append("missing")
+ assert len(row) == 2 and tuple(row) == (1, 2)
+ assert "missing" in row
+ with pytest.raises(PySparkKeyError):
+ row["missing"]
+
+ shared = ["x", "y"]
+ left = _create_row(shared, (1, 2))
+ right = _create_row(shared, (3, 4))
+ assert left.__fields__ is right.__fields__ is shared
+ shared[1] = "z"
+ assert (left.z, right.z) == (2, 4)
+ row.__fields__ = shared
+ assert row.__fields__ is shared and row.z == 2
+
+ factory = Row("first", "second")
+ made = factory(1)
+ assert made.__fields__ is factory
+ assert made.asDict() == {"first": 1}
+ assert repr(made) == "Row(first=1)"
+
+
[email protected]("serializer", ["pickle", "cloudpickle"])
+def test_row_serialization_preserves_public_type_and_shared_names(serializer):
+ from pyspark import cloudpickle
+ from pyspark.sql import Row
+ from pyspark.sql.types import _create_row
+
+ codec = pickle if serializer == "pickle" else cloudpickle
+ fields = ["value"]
+ rows = [_create_row(fields, (Row(child=1),)), _create_row(fields,
(Row(child=2),))]
+ restored = codec.loads(codec.dumps(rows))
+ assert all(type(row) is Row and type(row.value) is Row for row in restored)
+ assert restored[0].__fields__ is restored[1].__fields__
+ restored[0].__fields__[0] = "renamed"
+ assert restored[1].renamed == Row(child=2)
+ unnamed = codec.loads(codec.dumps(Row(1, 2)))
+ assert type(unnamed) is Row and isinstance(unnamed, tuple)
+ assert not hasattr(unnamed, "__fields__")
+ assert codec.loads(codec.dumps(Row)) is Row
+
+
+def _tuple_row_results(mock_connect, table, schema, method):
+ import pyarrow as pa
+
+ spark, service = mock_connect
+ sink = pa.BufferOutputStream()
+ with pa.ipc.new_stream(sink, table.schema) as writer:
+ writer.write_table(table, max_chunksize=2)
+ service.execute_payload = message_field(7, schema) + message_field(
+ 2, varint_field(1, table.num_rows) + message_field(2,
sink.getvalue().to_pybytes()))
+ frame = spark.range(table.num_rows)
+ return frame.collect() if method == "collect" else list(
+ frame.toLocalIterator(method == "prefetch"))
+
+
[email protected]("method", ["collect", "iterator", "prefetch"])
+def test_collected_tuple_rows_share_names_by_schema_path(mock_connect, method):
+ import pyarrow as pa
+ from _mock_connect_server import data_type_proto
+
+ # PySpark keeps top-level names per collection or iterator batch, but
nested
+ # names belong to the schema and span batches. Equal sibling names are
independent.
+ child = pa.struct([("v", pa.int64())])
+ values = [{"v": index} for index in range(4)]
+ table = pa.table({
+ "left": pa.array(values, child),
+ "right": pa.array(values, child),
+ "items": pa.array([[value, value] for value in values],
pa.list_(child)),
+ "lookup": pa.array([[('a', value), ('b', value)] for value in values],
+ pa.map_(pa.string(), child)),
+ "struct_map": pa.array([[(value, value)] for value in values],
pa.map_(child, child)),
+ })
+ rows = _tuple_row_results(mock_connect, table, data_type_proto(ddl_json(
+ "left struct<v:bigint>, right struct<v:bigint>, "
+ "items array<struct<v:bigint>>, lookup map<string,struct<v:bigint>>, "
+ "struct_map map<struct<v:bigint>,struct<v:bigint>>")), method)
+ assert len(rows) == 4
+ for index, row in enumerate(rows):
+ assert type(row) is Row and isinstance(row, tuple)
+ assert type(row.left) is Row and row.left == (index,)
+ assert type(row.items[0]) is Row and row.items == [(index,), (index,)]
+ assert type(row.lookup['a']) is Row and row.lookup == {'a': (index,),
'b': (index,)}
+ assert type(row.left.__fields__) is list
+ assert row.left.__fields__ is rows[0].left.__fields__
+ assert row.right.__fields__ is rows[0].right.__fields__
+ assert row.items[0].__fields__ is row.items[1].__fields__ is
rows[0].items[0].__fields__
+ assert row.lookup['a'].__fields__ is row.lookup['b'].__fields__ is
rows[0].lookup['a'].__fields__
+ key, value = next(iter(row.struct_map.items()))
+ assert type(key) is type(value) is Row and key == value == (index,)
+ first_key, first_value = next(iter(rows[0].struct_map.items()))
+ assert key.__fields__ is first_key.__fields__
+ assert value.__fields__ is first_value.__fields__
+ assert key.__fields__ is not value.__fields__
+ assert len({id(rows[0][index].__fields__) for index in (0, 1)}) == 2
+ assert rows[0].left.__fields__ is not rows[0].items[0].__fields__
+ assert rows[0].left.__fields__ is not rows[0].lookup['a'].__fields__
+
+ rows[0].left.__fields__[0] = "renamed"
+ rows[0].items[0].__fields__[0] = "array_value"
+ rows[0].lookup['a'].__fields__[0] = "map_value"
+ next(iter(rows[0].struct_map)).__fields__[0] = "map_key"
+ assert [row.left.renamed for row in rows] == list(range(4))
+ assert [row.right.v for row in rows] == list(range(4))
+ assert [row.items[1].array_value for row in rows] == list(range(4))
+ assert [row.lookup['b'].map_value for row in rows] == list(range(4))
+ assert [next(iter(row.struct_map)).map_key for row in rows] ==
list(range(4))
+
+ assert rows[0].__fields__ is rows[1].__fields__
+ assert rows[2].__fields__ is rows[3].__fields__
+ assert (rows[0].__fields__ is rows[2].__fields__) == (method == "collect")
+ rows[0].__fields__[0] = "renamed_left"
+ assert rows[1].renamed_left.renamed == 1
+ if method != "collect":
+ assert rows[2].left.renamed == 2
+
+
+class _TupleBinaryUDT(T.UserDefinedType):
+ @classmethod
+ def sqlType(cls):
+ return T.StructType([T.StructField("b", T.BinaryType())])
+
+ @classmethod
+ def module(cls):
+ return __name__
+
+ def deserialize(self, datum):
+ return datum, datum.__fields__
+
+
+def _tuple_row_type_proto(data_type):
+ from _mock_connect_server import data_type_proto
+
+ if isinstance(data_type, T.UserDefinedType):
+ value = data_type.jsonValue()
+ # DataType.udt=23: type=1, python_class=3, serialized_python_class=4,
sql_type=5.
+ return message_field(23, message_field(1, "udt") + message_field(3,
value["pyClass"])
+ + message_field(4, value["serializedClass"])
+ + message_field(5,
_tuple_row_type_proto(data_type.sqlType())))
+ if isinstance(data_type, T.StructType):
+ return message_field(21, b"".join(message_field(
+ 1, message_field(1, field.name) + message_field(2,
_tuple_row_type_proto(field.dataType))
+ + varint_field(3, int(field.nullable))) for field in
data_type.fields))
+ return data_type_proto(data_type.jsonValue())
+
+
[email protected]("method", ["collect", "iterator", "prefetch"])
+def test_tuple_rows_keep_configured_binary_and_udt_inputs(mock_connect,
method):
+ import pyarrow as pa
+
+ _, service = mock_connect
+ service.confs = {"spark.sql.execution.pyspark.binaryAsBytes": "false"}
+ child = pa.struct([("b", pa.binary())])
+ converted = pa.struct([("b", pa.binary()), ("u", child)])
+ values = [{"b": bytes([index])} for index in range(4)]
+ table = pa.table({
+ "s": pa.array(values, child),
+ "converted": pa.array([{"b": value["b"], "u": value} for value in
values], converted),
+ })
+ schema = T.StructType([
+ T.StructField("s", T.StructType([T.StructField("b", T.BinaryType())])),
+ T.StructField("converted", T.StructType([
+ T.StructField("b", T.BinaryType()), T.StructField("u",
_TupleBinaryUDT()),
+ ])),
+ ])
+ rows = _tuple_row_results(mock_connect, table,
_tuple_row_type_proto(schema), method)
+ assert len(rows) == 4
+ for index, row in enumerate(rows):
+ assert type(row) is Row and type(row.s) is Row and type(row.converted)
is Row
+ assert type(row.s.b) is type(row.converted.b) is bytearray
+ assert row.s.b == row.converted.b == bytes([index])
+ udt_input, input_fields = row.converted.u
+ assert type(udt_input) is Row and isinstance(udt_input, tuple)
+ assert type(udt_input.b) is bytearray and udt_input.b == bytes([index])
+ assert input_fields is udt_input.__fields__ and type(input_fields) is
list
+ assert row.converted.__fields__ is rows[0].converted.__fields__
+ assert type(rows[0].converted.__fields__) is list
+ rows[0].converted.__fields__[0] = "renamed_binary"
+ assert rows[-1].converted.renamed_binary == b"\x03"
+
+
[email protected]("spatial_type", [T.GeometryType, T.GeographyType])
+def
test_tuple_struct_converter_keeps_spatial_payloads_and_mutable_names(spatial_type):
+ from pyspark.sql.types import _udt_field_converters
+
+ wkb = bytes.fromhex("0101000000000000000000f03f0000000000000040")
+ schema = T.StructType([T.StructField("s", T.StructType([
+ T.StructField("b", T.BinaryType()), T.StructField("g",
spatial_type(4326)),
+ ]))])
+ convert = _udt_field_converters(schema)[0]
+ left = convert(Row(b=bytearray(b"x"), g=Row(wkb=wkb, srid=4326)))
+ right = convert(Row(b=bytearray(b"y"), g=Row(wkb=wkb, srid=4326)))
+ assert type(left) is Row and isinstance(left, tuple)
+ assert type(left.b) is bytearray and left.b == b"x"
+ assert type(left.g.wkb) is bytes and left.g.wkb == wkb
+ assert left.g == right.g
+ assert type(left.__fields__) is list and left.__fields__ is
right.__fields__
+ left.__fields__[0] = "renamed_binary"
+ assert right.renamed_binary == b"y"
+
+
def test_row_fields():
from pyspark.errors import (
PySparkAttributeError,
@@ -7710,9 +7945,9 @@ def
test_create_dataframe_raises_recursion_error_for_a_value_that_contains_itsel
_local_create_dataframe([(value,)])
-# py_to_value converts the cells of object-dtype pandas input, as well as the
values of
-# fillna, pivot and Row conversion; local rows go through infer.convert
instead, so the
-# "local" source is a control for values that the exact-number fast path must
not change.
+# py_to_value converts object-dtype cells of customized pandas frames, plus
fillna
+# and pivot values; local rows use infer.convert. The "local" source below is a
+# control for values that the exact-number fast path must not change.
def _RowPathFrame(pd):
"""A subclass of pandas.DataFrame: createDataFrame converts a customized
frame with its
row path (py_to_value), where it converts a plain one with PySpark's Arrow
conversion."""
@@ -7847,6 +8082,98 @@ def
test_local_input_native_numbers_inside_nested_values(mock_connect):
assert dict(actual["mapping"]) == {"x": 2**63 - 1}
[email protected]("source", ["local", "pandas"])
[email protected]("make, expected", [
+ (lambda: Row(a=1, b="x"), {"a": 1, "b": "x"}),
+ (lambda: Row("a", "b")(1, "x"), {"a": 1, "b": "x"}),
+ (lambda: Row("a", "b")(1), {"a": 1}),
+])
+def test_tuple_rows_remain_named_structs_in_native_input(mock_connect, source,
make, expected):
+ # A customized pandas subclass retains the native row path. Plain pandas
+ # frames take the separate Arrow path checked below.
+ value = make()
+ actual = _native_number_input_table(
+ mock_connect, [(value,)], source=source
+ ).column(0)[0].as_py()
+ assert actual == expected
+
+
[email protected]("source", ["local", "pandas"])
+def test_unnamed_nested_row_inference_matches_pyspark(mock_connect, source):
+ import pyarrow as pa
+
+ # Local inference makes positional structs; the customized pandas row path
+ # keeps unnamed tuple values as arrays.
+ column = _native_number_input_table(
+ mock_connect, [(Row(1, 2),)], source=source
+ ).column(0)
+ if source == "local":
+ assert column.type == pa.struct([pa.field("_1", pa.int64()),
pa.field("_2", pa.int64())])
+ assert column[0].as_py() == {"_1": 1, "_2": 2}
+ else:
+ assert pa.types.is_list(column.type) or
pa.types.is_large_list(column.type)
+ assert column.type.value_type == pa.int64()
+ assert column[0].as_py() == [1, 2]
+
+
[email protected]("value", [Row(1, 2), Row(a=1, b=2)])
+def test_plain_pandas_tuple_rows_keep_arrow_array_inference(mock_connect,
value):
+ import pandas as pd
+ import pyarrow as pa
+
+ # The default branch preserves this plain frame and its Arrow conversion
route.
+ table = _native_number_input_table(
+ mock_connect, pd.DataFrame({"value": [value]}, dtype=object))
+ table.validate(full=True)
+ assert (table.num_rows, table.num_columns) == (1, 1)
+ # IPC keeps temporary names; the enclosing ToDF applies the public label.
+ assert table.column_names == ["_0"]
+ _, service = mock_connect
+ request = parse_message(service.execute_requests[-1])
+ relation = parse_message(parse_message(request[3][0])[1][0])
+ rename = parse_message(relation[18][0])
+ assert [name.decode() for name in rename[2]] == ["value"]
+ column = table.column(0)
+ assert pa.types.is_list(column.type) or pa.types.is_large_list(column.type)
+ assert column.type.value_type == pa.int64()
+ assert column[0].as_py() == [1, 2]
+
+
+def test_customized_pandas_exact_nested_tuple_rows_remain_arrays(mock_connect):
+ import pyarrow as pa
+
+ values = [
+ ((1, None, -3), (), None, (2**63 - 1, -(2**63))),
+ (),
+ None,
+ ((4,), (5, 6)),
+ ]
+ table = _native_number_input_table(
+ mock_connect, [(value,) for value in values], source="pandas")
+ table.validate(full=True)
+ assert (table.num_rows, table.num_columns) == (4, 1)
+ assert table.column_names == ["value"]
+ assert table.column(0).type == pa.list_(pa.list_(pa.int64()))
+ assert table.column(0).to_pylist() == [
+ [[1, None, -3], [], None, [2**63 - 1, -(2**63)]],
+ [],
+ None,
+ [[4], [5, 6]],
+ ]
+
+
+def test_pandas_tuple_field_lookup_propagates_errors(mock_connect):
+ class InvalidFields(tuple):
+ @property
+ def __fields__(self):
+ raise RuntimeError("field lookup failed")
+
+ with pytest.raises(RuntimeError, match="^field lookup failed$"):
+ _native_number_input_table(
+ mock_connect, [(InvalidFields((1,)),)], source="pandas"
+ )
+
+
# Exact list/tuple conversion and pandas input IPC regressions.
from datetime import date as container_input_date, datetime as
container_input_datetime
from decimal import Decimal as ContainerInputDecimal
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]