This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 586286bd [SPARK-61100][CONNECT][PYTHON] Preserve tuple semantics for 
Python Row
586286bd is described below

commit 586286bd82b6b9ba07c75e0ab14e3b48849980b7
Author: DB Tsai <[email protected]>
AuthorDate: Mon Nov 9 09:29:31 2026 +0900

    [SPARK-61100][CONNECT][PYTHON] Preserve tuple semantics for Python Row
    
    ### What changes were proposed in this pull request?
    
    Return one public tuple-based `Row` through the native extension and Python 
imports. Embed PySpark 4.3.0's Row implementation, preserve mutable 
`__fields__`, and materialize Rows for collection and iteration. Share field 
names per collected result, per top-level iterator Arrow batch, and per nested 
schema path.
    
    Update local-input conversion, pickling and UDT conversion to use that 
identity. Add tuple, field-mutation, nested-input, batch and UDT/spatial 
regressions.
    
    ### Why are the changes needed?
    
    [SPARK-61100](https://issues.apache.org/jira/browse/SPARK-61100). The 
native object wrapper lacks tuple inheritance and has different identity and 
field-mutation behavior, affecting tuple-aware serializers and pandas-on-Spark 
paths.
    
    Includes the [NumPy 
prerequisite](https://github.com/apache/spark-connect-rust/pull/381) 
(SPARK-61102) and [binary configuration/iterator 
follow-up](https://github.com/apache/spark-connect-rust/pull/382) (SPARK-61099).
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. Rows support PySpark's tuple interface, public identity and field-name 
semantics. The inherited customized-pandas named-Row conversion behavior is 
retained; complete pandas parity is not established.
    
    Tuple compatibility has a measured performance cost. For 100,000 collected 
rows followed by full recursive consumption, comparing the final binary parent 
with this change:
    
    | Shape | Client CPU, parent → Row (ms) | Wall time (ms) | RSS after 
validation (MiB) |
    | --- | --- | --- | --- |
    | One integer | 47.8 → 88.2 | 59.9 → 99.8 | 109.8 → 124.5 |
    | Eight integers | 155.9 → 196.6 | 173.0 → 220.9 | 150.3 → 164.7 |
    | Four fields, three nested Rows | 418.1 → 670.1 | 441.4 → 707.7 | 285.7 → 
344.6 |
    
    Paired CPU ratios were 1.31–1.86× and wall ratios 1.26–1.65×; retained RSS 
increased by 14–59 MiB. Collect-only CPU ratios were 6.18–16.29× because values 
now materialize before return. Wide iterator consumption improved; narrow and 
nested iteration remained slower. These costs are an explicit tradeoff for 
tuple compatibility.
    
    ### How was this patch tested?
    
    - Formatting, Rust core/API units, integration-test compilation with and 
without WASM, and native-extension Python suites: 2,291 passed, eight 
live-server skips.
    - Twelve native live cases and all 36 setup/call/teardown reports passed 
with verified imports; 172 unrelated cases were deselected. Eight smoke actions 
covered nested Rows, JSON, pickle, shared names and local inputs. The NTZ 
test's nested subprocess had no separate import telemetry.
    - Scoped official transport checks: 85 passed, 34 skipped, no retries; all 
12 mixed Classic/Connect collection cases skipped. API/member checks confirmed 
tuple inheritance and inspected members, with annotation differences retained.
    - The three offline files named by CI passed 2,363 cases with eight 
remote-only skips. Configured Python-shim coverage was 84.3% across 72 files; 
embedded Row and compiled Rust are excluded. A causal control reproduced 
`NoSource` without the synthetic Row omission. The workflow's 90/75 values 
select badge colors, not fail-under thresholds.
    - Both paired benchmarks completed 54 fresh workers, 27 pairs and 162 
validated observations each. Results above are medians of process medians; 
paired ratios use paired samples. This is one macOS/Python/dependency set with 
three pairs, no confidence intervals or server-CPU measurements. RSS is an 
endpoint observation, not allocation or leak evidence. Official 
Python/newest-master performance was not measured. Auxiliary profiling ended 
with a cleanup error and its timings are excluded.
    
    The shared full and cluster runs executed the NumPy prerequisite binary. 
This branch has the same 43 selected raw-RPC source files and separate 
native-import/live validation. This is source-based reuse of transport 
evidence; those full and cluster runs did not execute this branch's binary. 
They use official Python plans and converters, with no per-child import 
attestation, and do not validate native high-level conversion, native 
ReleaseSession or native artifact upload.
    
    The full run completed 87 normal files: 2,586 passed, 641 skipped, one 
deselected and no failures, errors or retries. Eleven files were entirely 
skipped. A separate advisory check of the existing metadata-column manifest 
entry still failed because the test catalog plugin was unavailable (one 
failure, 57 deselections). The cluster phase passed 15 invocations across four 
modules (ten substantive cases and five helpers), with 14 JVM-guarded skips and 
no failures or retries.
    
    The original full run had a seeded-random failure on two threads. The 
unchanged test failed identically in official Python and both saved Rust 
transports on two threads, and passed in all three on its default four-thread 
fixture. The complete successful rerun used that fixture. Server logs retain 
deliberate negative-test diagnostics and shutdown races; successful direct 
shutdown was recorded, without final descendant-process telemetry.
    
    Clippy remains advisory exit 101; its source diagnostics were assessed 
against the baseline. Linux, Windows and the newest-master combined tree were 
not runtime tested locally. These results establish neither full converter 
parity nor performance parity.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: OpenAI Codex
    
    This pull request and its description were written by OpenAI Codex.
    
    Closes #383 from dbtsai/codex/python-rust-gap-row.
    
    Lead-authored-by: DB Tsai <[email protected]>
    Co-authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 .github/workflows/lint.yml          |   6 +-
 AGENTS.md                           |  16 +-
 crates/pyspark-rs/src/dataframe.rs  |  95 ++++--
 crates/pyspark-rs/src/infer.rs      |  22 +-
 crates/pyspark-rs/src/lib.rs        |   3 +-
 crates/pyspark-rs/src/row.py        | 228 ++++++++++++++
 crates/pyspark-rs/src/row.rs        | 605 ++++++++----------------------------
 crates/pyspark-rs/src/session.rs    |  33 +-
 crates/spark-connect/src/row.rs     |  23 ++
 pyproject.toml                      |  14 +-
 python/.coveragerc                  |   6 +
 python/pyspark/sql/types.py         |  26 +-
 python/tests/test_arrow_handoff.py  |  92 ++++++
 python/tests/test_dropin_offline.py | 343 +++++++++++++++++++-
 14 files changed, 936 insertions(+), 576 deletions(-)

diff --git a/.github/workflows/lint.yml b/.github/workflows/lint.yml
index 1f226e29..c1473891 100644
--- a/.github/workflows/lint.yml
+++ b/.github/workflows/lint.yml
@@ -66,7 +66,7 @@ jobs:
         run: python3 -m unittest discover -s scripts/tests
 
   python-ruff:
-    name: ruff (our scripts)
+    name: ruff (our scripts and embedded Row)
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4
@@ -76,6 +76,6 @@ jobs:
       - name: Install ruff
         run: python -m pip install "ruff==0.14.0"
       - name: ruff check - blocking
-        run: ruff check scripts/
+        run: ruff check scripts/ crates/pyspark-rs/src/row.py
       - name: ruff format (check) - blocking
-        run: ruff format --check scripts/
+        run: ruff format --check scripts/ crates/pyspark-rs/src/row.py
diff --git a/AGENTS.md b/AGENTS.md
index 4810ae4d..bbe4115c 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -243,7 +243,9 @@ Vendor the ones we don't have a Rust implementation for.
   (the gRPC channel connects lazily, so proto/plan construction needs no 
server)
   and server-gated `tests/e2e_*.rs` (they check 
`std::env::var("SPARK_REMOTE")`).
 - **Python drop-in**: `python/.coveragerc` scopes to the hand-written shim
-  (omits vendored/generated/Rust-only). Run:
+  (omits vendored/generated/Rust-only). The embedded 
`crates/pyspark-rs/src/row.py`
+  runs as `pyspark/_row.py` without a file at that path and is explicitly 
omitted;
+  Row tests exercise its behavior and Ruff checks its source. Run:
   ```
   cd python && PYTHONPATH=$(pwd) python -m coverage run --rcfile=.coveragerc \
     -m pytest tests/test_dropin_offline.py && python -m coverage report 
--rcfile=.coveragerc
@@ -414,9 +416,15 @@ For every method matching a pyspark name:
   proto carrying the pickled object: `registerDataSource`, 
`GroupedData.transformWithState[InPandas]`,
   `createDataFrame(..., udf)`, etc. General rule: proto-building → Rust; 
cloudpickled Python code that
   runs server-side → vendored Python.
-- **Row materialization:** PyO3 cannot subclass `tuple`, so our `Row` is not a 
tuple subclass.
-  Some pandas-on-Spark code assumes a tuple-`Row` with mutable `__fields__`. 
Full fidelity
-  likely requires adopting the upstream Python tuple-`Row` at collect 
boundaries.
+- **Row materialization:** `crates/pyspark-rs/src/row.py` embeds PySpark's 
Python `Row(tuple)`
+  class, exported as `pyspark._pyspark.Row` and re-exported by 
`pyspark.sql.types`,
+  `pyspark.sql`, and `pyspark`. Embedding avoids importing the partially 
initialized
+  `pyspark.sql.types` shim while the extension loads. Rust eagerly builds 
tuple values
+  through `pyspark.sql.types._create_row`; collected rows share a mutable 
`__fields__`
+  list across the result, and iterator rows share top-level names within each 
Arrow
+  record batch. Nested names are shared by schema path across the execution, 
including
+  iterator batches, while distinct sibling paths keep separate lists.
+  Nested local-input Rows follow the tuple inference paths; named Rows expose 
`__fields__`.
 - **Streaming listeners and progress holders:** Keep progress holders 
(`StreamingQueryProgress`,
   `StateOperatorProgress`, etc.) as faithful pure-Python implementations 
parsed from server
   JSON, not Rust-backed. The listener bus (subscribe/dispatch) is genuinely 
Rust-backed;
diff --git a/crates/pyspark-rs/src/dataframe.rs 
b/crates/pyspark-rs/src/dataframe.rs
index cfd0e888..b98ffa34 100644
--- a/crates/pyspark-rs/src/dataframe.rs
+++ b/crates/pyspark-rs/src/dataframe.rs
@@ -9,7 +9,7 @@ use spark_connect::udf::{eval_type, 
CommonInlineUserDefinedFunctionExpression, P
 use crate::column::PyColumn;
 use crate::errors::ResultExt;
 use crate::group::PyGroupedData;
-use crate::row::{FieldConverters, PyRow};
+use crate::row::{FieldConverters, NestedFieldNames};
 use crate::streaming::PyDataStreamWriter;
 use crate::types::PyDataType;
 
@@ -2187,12 +2187,12 @@ impl PyDataFrame {
     }
 
     /// Collect all rows into memory.
-    fn collect(&self, py: Python<'_>) -> PyResult<Vec<PyRow>> {
+    fn collect(&self, py: Python<'_>) -> PyResult<Vec<Py<PyAny>>> {
         // Release the GIL across the blocking RPC so other Python threads run.
         let (rows, schema) = py
             .detach(|| self.dataframe.collect_with_schema())
             .to_pyerr()?;
-        rows_to_py(py, &self.dataframe, rows, schema.as_ref())
+        rows_to_py(py, &self.dataframe, &rows, schema.as_ref())
     }
 
     /// Return an iterator over rows without materializing the entire result 
in memory.
@@ -2396,12 +2396,11 @@ impl PyDataFrame {
     }
 
     /// Get the first row.
-    fn first(&self, py: Python<'_>) -> PyResult<Option<PyRow>> {
+    fn first(&self, py: Python<'_>) -> PyResult<Option<Py<PyAny>>> {
         let (row, schema) = py
             .detach(|| self.dataframe.first_with_schema())
             .to_pyerr()?;
-        let rows = row.into_iter().collect();
-        Ok(rows_to_py(py, &self.dataframe, rows, schema.as_ref())?.pop())
+        Ok(rows_to_py(py, &self.dataframe, row.as_slice(), 
schema.as_ref())?.pop())
     }
 
     /// Alias for first.
@@ -2416,17 +2415,17 @@ impl PyDataFrame {
                 let (rows, schema) = py
                     .detach(|| self.dataframe.limit(k).collect_with_schema())
                     .to_pyerr()?;
-                rows_to_py(py, &self.dataframe, rows, 
schema.as_ref())?.into_py_any(py)
+                rows_to_py(py, &self.dataframe, &rows, 
schema.as_ref())?.into_py_any(py)
             }
         }
     }
 
     /// Get the first n rows.
-    fn take(&self, py: Python<'_>, num: usize) -> PyResult<Vec<PyRow>> {
+    fn take(&self, py: Python<'_>, num: usize) -> PyResult<Vec<Py<PyAny>>> {
         let (rows, schema) = py
             .detach(|| self.dataframe.take_with_schema(num))
             .to_pyerr()?;
-        rows_to_py(py, &self.dataframe, rows, schema.as_ref())
+        rows_to_py(py, &self.dataframe, &rows, schema.as_ref())
     }
 
     /// Column names (a property, mirroring `DataFrame.columns`).
@@ -2436,11 +2435,11 @@ impl PyDataFrame {
     }
 
     /// Get the last `num` rows as a list of Rows (eager, like pyspark `tail`).
-    fn tail(&self, py: Python<'_>, num: i32) -> PyResult<Vec<PyRow>> {
+    fn tail(&self, py: Python<'_>, num: i32) -> PyResult<Vec<Py<PyAny>>> {
         let (rows, schema) = py
             .detach(|| self.dataframe.tail(num).collect_with_schema())
             .to_pyerr()?;
-        rows_to_py(py, &self.dataframe, rows, schema.as_ref())
+        rows_to_py(py, &self.dataframe, &rows, schema.as_ref())
     }
 
     /// Sample a fraction of rows, as PySpark's 
`DataFrame.sample(withReplacement=None,
@@ -4281,23 +4280,36 @@ fn read_binary_as_bytes(py: Python<'_>, dataframe: 
&DataFrame) -> PyResult<bool>
     Ok(value.is_some_and(|value| value.eq_ignore_ascii_case("true")))
 }
 
-/// The collected `rows` of a result of `schema` of `dataframe`, whose values 
of user-defined
-/// types are deserialized when first read. Reads the binary result setting 
after execution
-/// and applies it to binary values, including when only an Arrow schema is 
available.
+/// The collected `rows` of a result of `schema`, with configured binary 
values and
+/// user-defined values materialized before constructing the tuples returned 
to Python.
+/// Reads the binary setting after execution, including with only an Arrow 
schema.
 fn rows_to_py(
     py: Python<'_>,
     dataframe: &DataFrame,
-    rows: Vec<spark_connect::row::Row>,
+    rows: &[spark_connect::row::Row],
     schema: Option<&spark_connect::types::DataType>,
-) -> PyResult<Vec<PyRow>> {
+) -> PyResult<Vec<Py<PyAny>>> {
     check_arrow_schema(py, schema)?;
     let binary_as_bytes = read_binary_as_bytes(py, dataframe)?;
     check_collectable(py, schema)?;
     let converters = udt_field_converters(py, schema)?;
-    Ok(rows
-        .into_iter()
-        .map(|row| PyRow::with_converters(row, converters.clone(), 
binary_as_bytes))
-        .collect())
+    let Some(first) = rows.first() else {
+        return Ok(Vec::new());
+    };
+    let fields = PyList::new(py, first.fields())?;
+    let mut nested_fields = NestedFieldNames::default();
+    rows.iter()
+        .map(|row| {
+            crate::row::row_values_to_py(
+                py,
+                &fields,
+                row.values(),
+                converters.as_ref(),
+                binary_as_bytes,
+                &mut nested_fields,
+            )
+        })
+        .collect()
 }
 
 /// Python wrapper for LocalRowIterator yielding Row objects lazily.
@@ -4316,6 +4328,10 @@ pub struct PyLocalRowIterator {
     converters: Option<Option<FieldConverters>>,
     /// Snapshotted before the first fetch, as the body of PySpark's generator 
does.
     binary_as_bytes: Option<bool>,
+    /// Top-level names shared only by rows of the current Arrow record batch.
+    fields: Option<(std::sync::Arc<[String]>, Py<PyList>)>,
+    /// Nested names belong to schema paths and stay shared across iterator 
batches.
+    nested_fields: NestedFieldNames,
 }
 
 impl PyLocalRowIterator {
@@ -4327,6 +4343,8 @@ impl PyLocalRowIterator {
             finished: false,
             converters: None,
             binary_as_bytes: None,
+            fields: None,
+            nested_fields: NestedFieldNames::default(),
         }
     }
 
@@ -4342,7 +4360,7 @@ impl PyLocalRowIterator {
     }
 
     /// Prepare and fetch one Python row; the public wrapper closes on 
exhaustion or error.
-    fn next_python_row(&mut self, py: Python<'_>) -> PyResult<Option<PyRow>> {
+    fn next_python_row(&mut self, py: Python<'_>) -> 
PyResult<Option<Py<PyAny>>> {
         let binary_as_bytes = match self.binary_as_bytes {
             Some(value) => value,
             None => {
@@ -4362,11 +4380,34 @@ impl PyLocalRowIterator {
         match next {
             Some(Ok(row)) => {
                 let converters = self.converters.clone().flatten();
-                Ok(Some(PyRow::with_converters(
-                    row,
-                    converters,
+                let fields = match row.shared_fields() {
+                    Some(native_fields) => match self.fields.as_ref() {
+                        Some((cached, fields)) if 
std::sync::Arc::ptr_eq(cached, native_fields) => {
+                            fields.bind(py).clone()
+                        }
+                        _ => {
+                            let fields = PyList::new(py, row.fields())?;
+                            self.fields = Some((
+                                std::sync::Arc::clone(native_fields),
+                                fields.clone().unbind(),
+                            ));
+                            fields
+                        }
+                    },
+                    None => {
+                        self.fields = None;
+                        PyList::new(py, row.fields())?
+                    }
+                };
+                crate::row::row_values_to_py(
+                    py,
+                    &fields,
+                    row.values(),
+                    converters.as_ref(),
                     binary_as_bytes,
-                )))
+                    &mut self.nested_fields,
+                )
+                .map(Some)
             }
             Some(Err(error)) => 
Err(crate::errors::spark_error_to_py_exception(error)),
             None => Ok(None),
@@ -4386,6 +4427,8 @@ impl PyLocalRowIterator {
     fn close(&mut self, py: Python<'_>) {
         let iterator = self.iterator.take();
         self.finished = true;
+        self.fields = None;
+        self.nested_fields.clear();
         py.detach(move || drop(iterator));
     }
 
@@ -4393,7 +4436,7 @@ impl PyLocalRowIterator {
     ///
     /// The first call snapshots the binary result setting before starting 
execution.
     /// Both the Config RPC and blocking row fetch release the GIL.
-    fn __next__(&mut self, py: Python<'_>) -> PyResult<Option<PyRow>> {
+    fn __next__(&mut self, py: Python<'_>) -> PyResult<Option<Py<PyAny>>> {
         if self.finished {
             return Ok(None);
         }
diff --git a/crates/pyspark-rs/src/infer.rs b/crates/pyspark-rs/src/infer.rs
index 0b682543..c2e51146 100644
--- a/crates/pyspark-rs/src/infer.rs
+++ b/crates/pyspark-rs/src/infer.rs
@@ -23,7 +23,6 @@ use spark_connect::types::{DataType, GeographyType, 
GeometryType, StructField};
 
 use crate::column::raise_pyspark;
 use crate::errors::ResultExt;
-use crate::row::PyRow;
 use crate::values::{PyGeography, PyGeometry, PyVariantVal};
 
 static DECIMAL: PyOnceLock<Py<PyType>> = PyOnceLock::new();
@@ -656,18 +655,6 @@ fn row_items<'py>(
     if row.is_instance_of::<PyDict>() {
         return sorted_items(row);
     }
-    if let Ok(pyrow) = row.cast::<PyRow>() {
-        let pyrow = pyrow.borrow();
-        let (fields, values) = pyrow.parts(py)?;
-        return Ok(match fields {
-            Some(fields) => fields
-                .iter()
-                .map(|f| PyString::new(py, f).into_any())
-                .zip(values.iter())
-                .collect(),
-            None => positional(py, values.iter().collect(), names),
-        });
-    }
     if row.is_instance_of::<PyTuple>() || row.is_instance_of::<PyList>() {
         let fields = if row.hasattr(intern!(py, "__fields__"))? {
             Some(row.getattr(intern!(py, "__fields__"))?)
@@ -1663,10 +1650,7 @@ fn values_by_position<'py>(
 /// and another object by the names of its `__dict__`.
 fn convert_struct(obj: &Bound<'_, PyAny>, fields: &[StructField]) -> 
PyResult<Value> {
     let py = obj.py();
-    let values = if let Ok(row) = obj.cast::<PyRow>() {
-        let (_, values) = row.borrow().parts(py)?;
-        values_by_position(py, values.iter().collect(), fields)?
-    } else if let Ok(tuple) = obj.cast::<PyTuple>() {
+    let values = if let Ok(tuple) = obj.cast::<PyTuple>() {
         values_by_position(py, tuple.iter().collect(), fields)?
     } else if obj.is_instance_of::<PyDict>() {
         values_by_name(obj, fields)?
@@ -1722,10 +1706,6 @@ pub(crate) fn row_values<'py>(
     if item.is_none() {
         return Ok(vec![py.None().into_bound(py); fields.len()]);
     }
-    if let Ok(row) = item.cast::<PyRow>() {
-        let (_, values) = row.borrow().parts(py)?;
-        return values_by_position(py, values.iter().collect(), fields);
-    }
     if let Ok(tuple) = item.cast::<PyTuple>() {
         return values_by_position(py, tuple.iter().collect(), fields);
     }
diff --git a/crates/pyspark-rs/src/lib.rs b/crates/pyspark-rs/src/lib.rs
index 084a4797..b6042714 100644
--- a/crates/pyspark-rs/src/lib.rs
+++ b/crates/pyspark-rs/src/lib.rs
@@ -74,7 +74,6 @@ use readwriter::PyDataFrameReader;
 use resource::{
     PyExecutorResourceRequests, PyResourceProfile, PyResourceProfileBuilder, 
PyTaskResourceRequests,
 };
-use row::PyRow;
 use session::{PyHook, PySparkSession, PySparkSessionBuilder};
 use stat::PyStatFunctions;
 use streaming::{
@@ -109,7 +108,7 @@ fn _pyspark(_py: Python<'_>, m: &Bound<'_, PyModule>) -> 
PyResult<()> {
     m.add_class::<PyWhenNotMatched>()?;
     m.add_class::<PyWhenNotMatchedBySource>()?;
     m.add_class::<PyColumn>()?;
-    m.add_class::<PyRow>()?;
+    row::register(m)?;
     m.add_class::<PyGroupedData>()?;
     m.add_class::<PyCoGroupedData>()?;
     m.add_class::<PyLocalRowIterator>()?;
diff --git a/crates/pyspark-rs/src/row.py b/crates/pyspark-rs/src/row.py
new file mode 100644
index 00000000..6af09bec
--- /dev/null
+++ b/crates/pyspark-rs/src/row.py
@@ -0,0 +1,228 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Tuple-backed Row semantics from PySpark's ``pyspark.sql.types.Row``.
+
+Mirrors ``class Row`` in apache/spark v4.3.0, ``python/pyspark/sql/types.py``.
+Embedded by the extension so ``pyspark._pyspark.Row`` and the public imports
+refer to the same class without importing the partially initialized types shim.
+"""
+
+
+class Row(tuple):
+    """
+    A row in :class:`DataFrame`.
+    The fields in it can be accessed:
+
+    * like attributes (``row.key``)
+    * like dictionary values (``row[key]``)
+
+    ``key in row`` will search through row keys.
+
+    Row can be used to create a row object by using named arguments.
+    It is not allowed to omit a named argument to represent that the value is
+    None or missing. This should be explicitly set to None in this case.
+
+    .. versionchanged:: 3.0.0
+        Rows created from named arguments no longer have
+        field names sorted alphabetically and will be ordered in the position 
as
+        entered.
+
+    Examples
+    --------
+    >>> from pyspark.sql import Row
+    >>> row = Row(name="Alice", age=11)
+    >>> row
+    Row(name='Alice', age=11)
+    >>> row['name'], row['age']
+    ('Alice', 11)
+    >>> row.name, row.age
+    ('Alice', 11)
+    >>> 'name' in row
+    True
+    >>> 'wrong_key' in row
+    False
+
+    Row also can be used to create another Row like class, then it
+    could be used to create Row objects, such as
+
+    >>> Person = Row("name", "age")
+    >>> Person
+    <Row('name', 'age')>
+    >>> 'name' in Person
+    True
+    >>> 'wrong_key' in Person
+    False
+    >>> Person("Alice", 11)
+    Row(name='Alice', age=11)
+
+    This form can also be used to create rows as tuple values, i.e. with 
unnamed
+    fields.
+
+    >>> row1 = Row("Alice", 11)
+    >>> row2 = Row(name="Alice", age=11)
+    >>> row1 == row2
+    True
+    """
+
+    __module__ = "pyspark.sql.types"
+
+    def __new__(cls, *args, **kwargs):
+        if args and kwargs:
+            from pyspark.errors import PySparkValueError
+
+            raise PySparkValueError(
+                errorClass="CANNOT_SET_TOGETHER",
+                messageParameters={"arg_list": "args and kwargs"},
+            )
+        if kwargs:
+            row = tuple.__new__(cls, list(kwargs.values()))
+            row.__fields__ = list(kwargs.keys())
+            return row
+        return tuple.__new__(cls, args)
+
+    def asDict(self, recursive=False):
+        """
+        Return as a dict
+
+        Parameters
+        ----------
+        recursive : bool, optional
+            turns the nested Rows to dict (default: False).
+
+        Notes
+        -----
+        If a row contains duplicate field names, e.g., the rows of a join
+        between two :class:`DataFrame` that both have the fields of same names,
+        one of the duplicate fields will be selected by ``asDict``. 
``__getitem__``
+        will also return one of the duplicate fields, however returned value 
might
+        be different to ``asDict``.
+
+        Examples
+        --------
+        >>> from pyspark.sql import Row
+        >>> Row(name="Alice", age=11).asDict() == {'name': 'Alice', 'age': 11}
+        True
+        >>> row = Row(key=1, value=Row(name='a', age=2))
+        >>> row.asDict() == {'key': 1, 'value': Row(name='a', age=2)}
+        True
+        >>> row.asDict(True) == {'key': 1, 'value': {'name': 'a', 'age': 2}}
+        True
+        """
+        if not hasattr(self, "__fields__"):
+            from pyspark.errors import PySparkTypeError
+
+            raise PySparkTypeError(
+                errorClass="CANNOT_CONVERT_TYPE",
+                messageParameters={"from_type": "Row", "to_type": "dict"},
+            )
+        if recursive:
+
+            def conv(obj):
+                if isinstance(obj, Row):
+                    return obj.asDict(True)
+                if isinstance(obj, list):
+                    return [conv(value) for value in obj]
+                if isinstance(obj, dict):
+                    return {key: conv(value) for key, value in obj.items()}
+                return obj
+
+            return dict(zip(self.__fields__, (conv(value) for value in self)))
+        return dict(zip(self.__fields__, self))
+
+    def __contains__(self, item):
+        if hasattr(self, "__fields__"):
+            return item in self.__fields__
+        return super().__contains__(item)
+
+    def __call__(self, *args):
+        """create new Row object"""
+        from pyspark.sql.types import _create_row
+
+        if len(args) > len(self):
+            from pyspark.errors import PySparkValueError
+
+            raise PySparkValueError(
+                errorClass="TOO_MANY_VALUES",
+                messageParameters={
+                    "expected": str(len(self)),
+                    "item": "fields",
+                    "actual": str(len(args)),
+                },
+            )
+        return _create_row(self, args)
+
+    def __getitem__(self, item):
+        if isinstance(item, (int, slice)):
+            return super().__getitem__(item)
+        try:
+            index = self.__fields__.index(item)
+            return super().__getitem__(index)
+        except IndexError:
+            from pyspark.errors import PySparkKeyError
+
+            raise PySparkKeyError(errorClass="KEY_NOT_EXISTS", 
messageParameters={"key": str(item)})
+        except ValueError:
+            from pyspark.errors import PySparkValueError
+
+            raise PySparkValueError(item)
+
+    def __getattr__(self, item):
+        if item.startswith("__"):
+            from pyspark.errors import PySparkAttributeError
+
+            raise PySparkAttributeError(
+                errorClass="ATTRIBUTE_NOT_SUPPORTED", 
messageParameters={"attr_name": item}
+            )
+        try:
+            index = self.__fields__.index(item)
+            return self[index]
+        except IndexError:
+            from pyspark.errors import PySparkAttributeError
+
+            raise PySparkAttributeError(
+                errorClass="ATTRIBUTE_NOT_SUPPORTED", 
messageParameters={"attr_name": item}
+            )
+        except ValueError:
+            from pyspark.errors import PySparkAttributeError
+
+            raise PySparkAttributeError(
+                errorClass="ATTRIBUTE_NOT_SUPPORTED", 
messageParameters={"attr_name": item}
+            )
+
+    def __setattr__(self, key, value):
+        if key != "__fields__":
+            from pyspark.errors import PySparkRuntimeError
+
+            raise PySparkRuntimeError(errorClass="READ_ONLY", 
messageParameters={"object": "Row"})
+        self.__dict__[key] = value
+
+    def __reduce__(self):
+        """Returns a tuple so Python knows how to pickle Row."""
+        if hasattr(self, "__fields__"):
+            from pyspark.sql.types import _create_row
+
+            return _create_row, (self.__fields__, tuple(self))
+        return tuple.__reduce__(self)
+
+    def __repr__(self):
+        """Printable representation of Row used in Python REPL."""
+        if hasattr(self, "__fields__"):
+            return "Row(%s)" % ", ".join(
+                "%s=%r" % (key, value) for key, value in zip(self.__fields__, 
tuple(self))
+            )
+        return "<Row(%s)>" % ", ".join(repr(value) for value in self)
diff --git a/crates/pyspark-rs/src/row.rs b/crates/pyspark-rs/src/row.rs
index 75bd78af..0fbf771a 100644
--- a/crates/pyspark-rs/src/row.rs
+++ b/crates/pyspark-rs/src/row.rs
@@ -1,191 +1,123 @@
-//! PyO3 wrapper for spark_connect::row::Row, mirroring `pyspark.sql.Row`.
+//! Conversion of native Spark values into tuple-backed Python Rows.
+
+use std::collections::HashMap;
+use std::sync::Arc;
 
-use pyo3::basic::CompareOp;
-use pyo3::exceptions::PyIndexError;
 use pyo3::prelude::*;
 use pyo3::sync::PyOnceLock;
-use pyo3::types::{PyByteArray, PyBytes, PyDict, PyList, PySlice, PyString, 
PyTuple};
-use spark_connect::row::{Row, Value};
-
-use crate::column::raise_pyspark;
-
-/// Python wrapper for a Spark Row. Behaves as `pyspark.sql.Row`, a `tuple` 
subclass, does,
-/// though it cannot subclass `tuple` itself.
-#[pyclass(name = "Row", module = "pyspark.sql.types")]
-pub struct PyRow {
-    data: RowData,
-    /// Whether the row has field names (`__fields__`). As in PySpark, a row 
made of values
-    /// alone, such as `Row(1, 2)` or the row class `Row("name", "age")`, has 
none; the rest
-    /// of the client knows its fields as `_1`, `_2`, and so on.
-    named: bool,
+use pyo3::types::{PyByteArray, PyBytes, PyDict, PyList, PyTuple};
+use spark_connect::row::{StructValue, Value};
+
+/// Functions that deserialize user-defined or spatial values in each result 
field.
+pub(crate) type FieldConverters = Arc<Vec<Option<Py<PyAny>>>>;
+
+static CREATE_ROW: PyOnceLock<Py<PyAny>> = PyOnceLock::new();
+const ROW_SOURCE: &std::ffi::CStr = pyo3::ffi::c_str!(include_str!("row.py"));
+
+/// Expose the embedded tuple `Row` as `pyspark._pyspark.Row`, re-exported by 
the Python shims.
+/// Its source imports PySpark modules only inside methods, avoiding import 
cycles
+/// while `pyspark.sql.types` is importing this extension.
+/// The synthetic filename avoids embedding a build-machine path in 
distributed wheels;
+/// the source is compiled into the extension and need not be installed 
separately.
+pub(crate) fn register(module: &Bound<'_, PyModule>) -> PyResult<()> {
+    let row_module =
+        PyModule::from_code(module.py(), ROW_SOURCE, c"pyspark/_row.py", 
c"pyspark._row")?;
+    module.add("Row", row_module.getattr("Row")?)
 }
 
-/// The Python functions that turn the values of the fields of collected rows 
whose types have
-/// user-defined types into what PySpark gives for them, by deserializing 
those values; None
-/// for the other fields.
-pub(crate) type FieldConverters = std::sync::Arc<Vec<Option<Py<PyAny>>>>;
+/// One step of a nested schema path. Array elements and map keys/values carry 
no
+/// index, so all of them share the names at their path.
+#[derive(Clone, Copy, Eq, Hash, PartialEq)]
+enum NestedFieldPath {
+    Field(usize),
+    ArrayElement,
+    MapKey,
+    MapValue,
+}
 
-enum RowData {
-    /// A row that the server returned, whose Spark values become Python 
values when first
-    /// read, and stay those objects, as a PySpark row is a tuple of them: so 
the row equals
-    /// itself, with a NaN or a VARIANT value, and keeps its hash.
-    Spark {
-        row: Row,
-        values: PyOnceLock<Py<PyTuple>>,
-        /// The converters of its fields, when one has a user-defined type.
-        converters: Option<FieldConverters>,
-        /// Whether its binary values are `bytes`, or `bytearray`, as PySpark 
gives them per
-        /// `spark.sql.execution.pyspark.binaryAsBytes`.
-        binary_as_bytes: bool,
-    },
-    /// A row made in Python, which holds its Python values as they are, as 
PySpark's does.
-    Python {
-        fields: Vec<String>,
-        values: Py<PyTuple>,
-    },
+/// Mutable names for each nested schema path, shared across the result's 
Arrow batches.
+/// Distinct sibling structs retain independent lists even when their names 
are identical.
+#[derive(Default)]
+pub(crate) struct NestedFieldNames {
+    path: Vec<NestedFieldPath>,
+    fields: HashMap<Vec<NestedFieldPath>, Py<PyList>>,
 }
 
-impl PyRow {
-    pub fn new(row: Row) -> Self {
-        Self::with_converters(row, None, true)
+impl NestedFieldNames {
+    /// Release names when an iterator closes, including when materialization 
fails.
+    pub(crate) fn clear(&mut self) {
+        self.path.clear();
+        self.fields.clear();
     }
 
-    /// A row that the server returned, whose values of user-defined types 
`converters`
-    /// deserializes when they are first read, and whose binary values are 
`bytes` if
-    /// `binary_as_bytes`, or else `bytearray`.
-    pub(crate) fn with_converters(
-        row: Row,
-        converters: Option<FieldConverters>,
+    /// Convert one schema step below this path, restoring it even if 
conversion fails.
+    fn value_at<'py>(
+        &mut self,
+        py: Python<'py>,
+        value: &Value,
         binary_as_bytes: bool,
-    ) -> Self {
-        PyRow {
-            data: RowData::Spark {
-                row,
-                values: PyOnceLock::new(),
-                converters,
-                binary_as_bytes,
-            },
-            named: true,
-        }
-    }
-
-    /// A row made in Python of `values`, named by `fields`.
-    fn of_python(fields: Vec<String>, values: Bound<'_, PyTuple>, named: bool) 
-> Self {
-        PyRow {
-            data: RowData::Python {
-                fields,
-                values: values.unbind(),
-            },
-            named,
-        }
-    }
-
-    fn fields(&self) -> &[String] {
-        match &self.data {
-            RowData::Spark { row, .. } => row.fields(),
-            RowData::Python { fields, .. } => fields,
-        }
-    }
-
-    /// The values, as the tuple that a PySpark row is.
-    fn values<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyTuple>> {
-        match &self.data {
-            RowData::Spark {
-                row,
-                values,
-                converters,
-                binary_as_bytes,
-            } => {
-                let values = values.get_or_try_init(py, || -> 
PyResult<Py<PyTuple>> {
-                    let convert = |i: usize| {
-                        converters
-                            .as_ref()
-                            .and_then(|c| c.get(i))
-                            .and_then(Option::as_ref)
-                    };
-                    let values = row
-                        .values()
-                        .iter()
-                        .enumerate()
-                        .map(|(i, v)| {
-                            let value = value_to_py(py, v, *binary_as_bytes)?;
-                            match convert(i) {
-                                Some(converter) => 
converter.bind(py).call1((value,)),
-                                None => Ok(value),
-                            }
-                        })
-                        .collect::<PyResult<Vec<_>>>()?;
-                    Ok(PyTuple::new(py, values)?.unbind())
-                })?;
-                Ok(values.bind(py).clone())
-            }
-            RowData::Python { values, .. } => Ok(values.bind(py).clone()),
-        }
-    }
-
-    /// The value of the field at `index`, which is in range.
-    fn value<'py>(&self, py: Python<'py>, index: usize) -> PyResult<Bound<'py, 
PyAny>> {
-        match &self.data {
-            RowData::Spark { .. } => self.values(py)?.get_item(index),
-            RowData::Python { values, .. } => values.bind(py).get_item(index),
-        }
+        path: NestedFieldPath,
+    ) -> PyResult<Bound<'py, PyAny>> {
+        self.path.push(path);
+        let result = value_to_py(py, value, binary_as_bytes, self);
+        self.path.pop();
+        result
     }
 
-    /// The value of the first field named `name`, if any.
-    fn value_by_name<'py>(
-        &self,
+    /// Get mutable names for this path, using the first struct seen to 
initialize them.
+    fn names_at_path<'py>(
+        &mut self,
         py: Python<'py>,
-        name: &str,
-    ) -> PyResult<Option<Bound<'py, PyAny>>> {
-        match self.fields().iter().position(|field| field == name) {
-            Some(index) => self.value(py, index).map(Some),
-            None => Ok(None),
+        row: &StructValue,
+    ) -> PyResult<Bound<'py, PyList>> {
+        if let Some(fields) = self.fields.get(&self.path) {
+            return Ok(fields.bind(py).clone());
         }
+        let fields = PyList::new(py, row.fields())?;
+        self.fields
+            .insert(self.path.clone(), fields.clone().unbind());
+        Ok(fields)
     }
+}
 
-    /// The field names, if the row has them (`__fields__`), and the values, 
as the tuple
-    /// that a PySpark row is, for reading the row as PySpark's schema 
inference does.
-    pub(crate) fn parts<'py>(
-        &self,
-        py: Python<'py>,
-    ) -> PyResult<(Option<&[String]>, Bound<'py, PyTuple>)> {
-        Ok((self.named.then(|| self.fields()), self.values(py)?))
-    }
-
-    /// The row with Spark values, for the rest of the client, which converts 
the values of a
-    /// row made in Python as it converts those of a tuple.
-    pub(crate) fn to_row(&self, py: Python<'_>) -> PyResult<Row> {
-        match &self.data {
-            RowData::Spark { row, .. } => Ok(row.clone()),
-            RowData::Python { fields, values } => {
-                let values = values
-                    .bind(py)
-                    .iter()
-                    .map(|v| crate::session::py_to_value(&v))
-                    .collect::<PyResult<Vec<_>>>()?;
-                Ok(Row::new(fields.clone(), values))
+/// Build a Row from named values, deserializing user-defined fields with 
`converters`.
+/// The caller shares `fields` across result rows and nested names across the 
execution.
+pub(crate) fn row_values_to_py(
+    py: Python<'_>,
+    fields: &Bound<'_, PyList>,
+    values: &[Value],
+    converters: Option<&FieldConverters>,
+    binary_as_bytes: bool,
+    nested_fields: &mut NestedFieldNames,
+) -> PyResult<Py<PyAny>> {
+    let values = values
+        .iter()
+        .enumerate()
+        .map(|(index, value)| {
+            let value = nested_fields.value_at(
+                py,
+                value,
+                binary_as_bytes,
+                NestedFieldPath::Field(index),
+            )?;
+            match converters
+                .and_then(|converters| converters.get(index))
+                .and_then(Option::as_ref)
+            {
+                Some(converter) => converter.bind(py).call1((value,)),
+                None => Ok(value),
             }
-        }
-    }
-
-    /// What PySpark raises for an attribute that a row does not have.
-    fn no_attribute(py: Python<'_>, name: &str) -> PyErr {
-        raise_pyspark(
-            py,
-            "PySparkAttributeError",
-            "ATTRIBUTE_NOT_SUPPORTED",
-            &[("attr_name", name)],
-        )
-    }
-
-    /// The values of `other` as a tuple, if it is a row or a tuple, as a 
tuple operation
-    /// takes them.
-    fn tuple_of<'py>(other: &Bound<'py, PyAny>) -> PyResult<Option<Bound<'py, 
PyTuple>>> {
-        if let Ok(row) = other.cast::<PyRow>() {
-            return row.borrow().values(other.py()).map(Some);
-        }
-        Ok(other.cast::<PyTuple>().ok().cloned())
-    }
+        })
+        .collect::<PyResult<Vec<_>>>()?;
+    let create_row = CREATE_ROW.get_or_try_init(py, || {
+        py.import("pyspark.sql.types")?
+            .getattr("_create_row")
+            .map(Bound::unbind)
+    })?;
+    Ok(create_row
+        .bind(py)
+        .call1((fields, PyTuple::new(py, values)?))?
+        .unbind())
 }
 
 /// `datetime.datetime(1970, 1, 1)`, the epoch of Spark's timestamps, without 
a time zone.
@@ -219,10 +151,13 @@ fn micros_timedelta(py: Python<'_>, micros: i64) -> 
PyResult<Bound<'_, PyAny>> {
 
 /// Convert a Spark `Value` into a Python object (PyO3 0.28 `IntoPyObject`), 
whose binary values
 /// are `bytes` if `binary_as_bytes`, or else `bytearray`, also in arrays, 
maps and structs.
-pub fn value_to_py<'py>(
+/// `nested_fields` supplies shared struct names. Call through 
`NestedFieldNames::value_at`
+/// so its schema path is set for this value and restored after conversion.
+fn value_to_py<'py>(
     py: Python<'py>,
     v: &Value,
     binary_as_bytes: bool,
+    nested_fields: &mut NestedFieldNames,
 ) -> PyResult<Bound<'py, PyAny>> {
     // Propagate conversion failures as Python exceptions with `?` rather than
     // panicking across the FFI boundary (the conversions are near-infallible,
@@ -304,7 +239,12 @@ pub fn value_to_py<'py>(
         Value::List(items) => {
             let list = PyList::empty(py);
             for it in items {
-                list.append(value_to_py(py, it, binary_as_bytes)?)?;
+                list.append(nested_fields.value_at(
+                    py,
+                    it,
+                    binary_as_bytes,
+                    NestedFieldPath::ArrayElement,
+                )?)?;
             }
             list.into_any()
         }
@@ -314,21 +254,24 @@ pub fn value_to_py<'py>(
             let dict = PyDict::new(py);
             for (k, val) in m {
                 dict.set_item(
-                    value_to_py(py, k, binary_as_bytes)?,
-                    value_to_py(py, val, binary_as_bytes)?,
+                    nested_fields.value_at(py, k, binary_as_bytes, 
NestedFieldPath::MapKey)?,
+                    nested_fields.value_at(py, val, binary_as_bytes, 
NestedFieldPath::MapValue)?,
                 )?;
             }
             dict.into_any()
         }
         Value::Struct(row) => {
-            // A struct value materializes as a Row (named fields), matching 
PySpark,
-            // rather than a plain dict. Cloning keeps shared names and 
independent values.
-            Py::new(
+            // A struct value materializes as a Row with the mutable names of 
its schema path.
+            let fields = nested_fields.names_at_path(py, row)?;
+            row_values_to_py(
                 py,
-                PyRow::with_converters(row.clone().into(), None, 
binary_as_bytes),
+                &fields,
+                row.values(),
+                None,
+                binary_as_bytes,
+                nested_fields,
             )?
             .into_bound(py)
-            .into_any()
         }
         Value::Variant { value, metadata } => {
             // A VARIANT materializes as a VariantVal (toJson/toPython decode 
lazily),
@@ -342,307 +285,3 @@ pub fn value_to_py<'py>(
         }
     })
 }
-
-#[pymethods]
-impl PyRow {
-    /// Construct a Row, mirroring `pyspark.sql.Row`: `Row(name=value, ...)` 
makes a row with
-    /// field names, and `Row(v1, v2, ...)` one of values alone, which can 
also serve as a row
-    /// class: `Row("name", "age")("Alice", 11)`.
-    #[new]
-    #[pyo3(signature = (*args, **kwargs))]
-    fn __new__(
-        py: Python<'_>,
-        args: &Bound<'_, PyTuple>,
-        kwargs: Option<&Bound<'_, PyDict>>,
-    ) -> PyResult<Self> {
-        let kwargs = kwargs.filter(|kw| !kw.is_empty());
-        if let Some(kw) = kwargs {
-            if !args.is_empty() {
-                return Err(raise_pyspark(
-                    py,
-                    "PySparkValueError",
-                    "CANNOT_SET_TOGETHER",
-                    &[("arg_list", "args and kwargs")],
-                ));
-            }
-            let mut fields = Vec::with_capacity(kw.len());
-            let mut values = Vec::with_capacity(kw.len());
-            for (k, v) in kw.iter() {
-                fields.push(k.extract::<String>()?);
-                values.push(v);
-            }
-            Ok(PyRow::of_python(fields, PyTuple::new(py, values)?, true))
-        } else {
-            let fields = (0..args.len()).map(|i| format!("_{}", i + 
1)).collect();
-            Ok(PyRow::of_python(fields, args.clone(), false))
-        }
-    }
-
-    /// Number of fields.
-    fn __len__(&self) -> usize {
-        self.fields().len()
-    }
-
-    fn __iter__<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, 
pyo3::types::PyIterator>> {
-        self.values(py)?.try_iter()
-    }
-
-    fn __reversed__<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, 
PyAny>> {
-        py.import("builtins")?
-            .getattr("reversed")?
-            .call1((self.values(py)?,))
-    }
-
-    /// `x in row` tests the field names of a row that has them, and the 
values otherwise.
-    fn __contains__(&self, py: Python<'_>, item: &Bound<'_, PyAny>) -> 
PyResult<bool> {
-        if self.named {
-            return PyList::new(py, self.fields())?.contains(item);
-        }
-        self.values(py)?.contains(item)
-    }
-
-    /// An index or a slice, as of a tuple, or a field name.
-    fn __getitem__<'py>(
-        &self,
-        py: Python<'py>,
-        key: &Bound<'py, PyAny>,
-    ) -> PyResult<Bound<'py, PyAny>> {
-        if key.cast::<PySlice>().is_ok() {
-            return self.values(py)?.as_any().get_item(key);
-        }
-        if let Ok(mut i) = key.extract::<isize>() {
-            let n = self.fields().len() as isize;
-            if i < 0 {
-                i += n;
-            }
-            if i < 0 || i >= n {
-                return Err(PyIndexError::new_err("tuple index out of range"));
-            }
-            return self.value(py, i as usize);
-        }
-        // As PySpark looks the key up in `__fields__`, which a row without 
names lacks.
-        if !self.named {
-            return Err(Self::no_attribute(py, "__fields__"));
-        }
-        if let Ok(name) = key.cast::<PyString>() {
-            if let Some(value) = self.value_by_name(py, &name.to_cow()?)? {
-                return Ok(value);
-            }
-        }
-        let error = py.import("pyspark.errors")?.getattr("PySparkValueError")?;
-        Err(PyErr::from_value(error.call1((key,))?))
-    }
-
-    /// Attribute access by field name (`row.id`).
-    fn __getattr__<'py>(&self, py: Python<'py>, name: &str) -> 
PyResult<Bound<'py, PyAny>> {
-        if !name.starts_with("__") && self.named {
-            if let Some(value) = self.value_by_name(py, name)? {
-                return Ok(value);
-            }
-        }
-        Err(Self::no_attribute(py, name))
-    }
-
-    /// Only `__fields__` can be set, which names the fields, as in PySpark.
-    fn __setattr__(
-        &mut self,
-        py: Python<'_>,
-        name: &str,
-        value: &Bound<'_, PyAny>,
-    ) -> PyResult<()> {
-        if name != "__fields__" {
-            return Err(raise_pyspark(
-                py,
-                "PySparkRuntimeError",
-                "READ_ONLY",
-                &[("object", "Row")],
-            ));
-        }
-        let mut names = value
-            .try_iter()?
-            .map(|name| name?.str().map(|name| name.to_string()))
-            .collect::<PyResult<Vec<_>>>()?;
-        // PySpark pairs names with values as `zip` does; fields beyond the 
names keep theirs.
-        let fields = self.fields();
-        names.truncate(fields.len());
-        names.extend(fields[names.len()..].iter().cloned());
-        match &mut self.data {
-            RowData::Spark { row, .. } => *row = Row::new(names, 
row.values().to_vec()),
-            RowData::Python { fields, .. } => *fields = names,
-        }
-        self.named = true;
-        Ok(())
-    }
-
-    /// Field names, in order: a list attribute, as PySpark's `Row.__fields__` 
is. A row
-    /// without names has none.
-    #[getter]
-    fn __fields__(&self, py: Python<'_>) -> PyResult<Vec<String>> {
-        if !self.named {
-            return Err(Self::no_attribute(py, "__fields__"));
-        }
-        Ok(self.fields().to_vec())
-    }
-
-    /// As a dict of field name -> value (mirrors `Row.asDict`), with nested 
rows, also in
-    /// lists and dicts, as dicts if `recursive`.
-    #[pyo3(name = "asDict")]
-    #[pyo3(signature = (recursive=false))]
-    fn as_dict<'py>(&self, py: Python<'py>, recursive: bool) -> 
PyResult<Bound<'py, PyDict>> {
-        fn conv<'py>(obj: Bound<'py, PyAny>) -> PyResult<Bound<'py, PyAny>> {
-            let py = obj.py();
-            if let Ok(row) = obj.cast::<PyRow>() {
-                return Ok(row.borrow().as_dict(py, true)?.into_any());
-            }
-            if let Ok(list) = obj.cast::<PyList>() {
-                let items = 
list.iter().map(conv).collect::<PyResult<Vec<_>>>()?;
-                return Ok(PyList::new(py, items)?.into_any());
-            }
-            if let Ok(dict) = obj.cast::<PyDict>() {
-                let converted = PyDict::new(py);
-                for (k, v) in dict.iter() {
-                    converted.set_item(k, conv(v)?)?;
-                }
-                return Ok(converted.into_any());
-            }
-            Ok(obj)
-        }
-        if !self.named {
-            return Err(raise_pyspark(
-                py,
-                "PySparkTypeError",
-                "CANNOT_CONVERT_TYPE",
-                &[("from_type", "Row"), ("to_type", "dict")],
-            ));
-        }
-        let dict = PyDict::new(py);
-        for (name, value) in self.fields().iter().zip(self.values(py)?.iter()) 
{
-            dict.set_item(name, if recursive { conv(value)? } else { value })?;
-        }
-        Ok(dict)
-    }
-
-    /// A row class, such as `Row("name", "age")`, makes a row of `args` named 
by its values.
-    #[pyo3(signature = (*args))]
-    fn __call__(&self, py: Python<'_>, args: &Bound<'_, PyTuple>) -> 
PyResult<PyRow> {
-        if args.len() > self.fields().len() {
-            let (expected, actual) = (self.fields().len().to_string(), 
args.len().to_string());
-            return Err(raise_pyspark(
-                py,
-                "PySparkValueError",
-                "TOO_MANY_VALUES",
-                &[
-                    ("expected", expected.as_str()),
-                    ("item", "fields"),
-                    ("actual", actual.as_str()),
-                ],
-            ));
-        }
-        let fields = self
-            .values(py)?
-            .iter()
-            .take(args.len())
-            .map(|name| name.str().map(|name| name.to_string()))
-            .collect::<PyResult<Vec<_>>>()?;
-        Ok(PyRow::of_python(fields, args.clone(), true))
-    }
-
-    /// Pickles as PySpark's row does: with `_create_row` from the field names 
and values, or
-    /// as `Row(*values)` for a row without names.
-    fn __reduce__<'py>(slf: &Bound<'py, Self>) -> PyResult<Bound<'py, 
PyTuple>> {
-        let py = slf.py();
-        let row = slf.borrow();
-        let values = row.values(py)?;
-        if row.named {
-            let create_row = 
py.import("pyspark.sql.types")?.getattr("_create_row")?;
-            let args = (PyList::new(py, row.fields())?, 
values).into_pyobject(py)?;
-            return (create_row, args).into_pyobject(py);
-        }
-        (slf.get_type(), values).into_pyobject(py)
-    }
-
-    /// `Row(name='Alice', age=11)`, with the `repr()` of each value, or 
`<Row('a', 'b')>` for
-    /// a row without names, as PySpark prints a row.
-    fn __repr__(&self, py: Python<'_>) -> PyResult<String> {
-        let values = self.values(py)?;
-        let mut parts = Vec::with_capacity(values.len());
-        if self.named {
-            for (name, value) in self.fields().iter().zip(values.iter()) {
-                parts.push(format!("{}={}", name, value.repr()?));
-            }
-            return Ok(format!("Row({})", parts.join(", ")));
-        }
-        for value in values.iter() {
-            parts.push(value.repr()?.to_string());
-        }
-        Ok(format!("<Row({})>", parts.join(", ")))
-    }
-
-    /// Hashes as the tuple of its values does.
-    fn __hash__(&self, py: Python<'_>) -> PyResult<isize> {
-        self.values(py)?.hash()
-    }
-
-    /// Compares as the tuple of its values does, with another row or a tuple.
-    fn __richcmp__<'py>(
-        &self,
-        other: &Bound<'py, PyAny>,
-        op: CompareOp,
-    ) -> PyResult<Bound<'py, PyAny>> {
-        let py = other.py();
-        match Self::tuple_of(other)? {
-            Some(other) => self.values(py)?.rich_compare(other, op),
-            None => Ok(py.NotImplemented().into_bound(py)),
-        }
-    }
-
-    /// Concatenates as a tuple does, into a tuple.
-    fn __add__<'py>(&self, other: &Bound<'py, PyAny>) -> PyResult<Bound<'py, 
PyAny>> {
-        let py = other.py();
-        match Self::tuple_of(other)? {
-            Some(other) => self.values(py)?.as_any().add(other),
-            None => Ok(py.NotImplemented().into_bound(py)),
-        }
-    }
-
-    fn __radd__<'py>(&self, other: &Bound<'py, PyAny>) -> PyResult<Bound<'py, 
PyAny>> {
-        let py = other.py();
-        match Self::tuple_of(other)? {
-            Some(other) => other.as_any().add(self.values(py)?),
-            None => Ok(py.NotImplemented().into_bound(py)),
-        }
-    }
-
-    /// Repeats as a tuple does, into a tuple.
-    fn __mul__<'py>(&self, times: &Bound<'py, PyAny>) -> PyResult<Bound<'py, 
PyAny>> {
-        self.values(times.py())?.as_any().mul(times)
-    }
-
-    fn __rmul__<'py>(&self, times: &Bound<'py, PyAny>) -> PyResult<Bound<'py, 
PyAny>> {
-        self.__mul__(times)
-    }
-
-    /// Number of occurrences of `value` among the row's values (tuple.count).
-    fn count(&self, py: Python<'_>, value: &Bound<'_, PyAny>) -> 
PyResult<usize> {
-        self.values(py)?
-            .as_any()
-            .call_method1("count", (value,))?
-            .extract()
-    }
-
-    /// Index of the first occurrence of `value` among the row's values 
(tuple.index).
-    #[pyo3(signature = (value, *args))]
-    fn index(
-        &self,
-        py: Python<'_>,
-        value: &Bound<'_, PyAny>,
-        args: &Bound<'_, PyTuple>,
-    ) -> PyResult<usize> {
-        let mut call = vec![value.clone()];
-        call.extend(args.iter());
-        self.values(py)?
-            .as_any()
-            .call_method1("index", PyTuple::new(py, call)?)?
-            .extract()
-    }
-}
diff --git a/crates/pyspark-rs/src/session.rs b/crates/pyspark-rs/src/session.rs
index 6a42d5dc..ef75ad9e 100644
--- a/crates/pyspark-rs/src/session.rs
+++ b/crates/pyspark-rs/src/session.rs
@@ -717,8 +717,7 @@ impl PySparkSession {
         } else {
             // A VariantVal of PySpark has a `__dict__`, so that it is taken 
as a row, which
             // PySpark's `to_row` rejects, as `row_values` does.
-            !(first.is_instance_of::<crate::row::PyRow>()
-                || first.is_instance_of::<pyo3::types::PyTuple>()
+            !(first.is_instance_of::<pyo3::types::PyTuple>()
                 || first.is_instance_of::<PyList>()
                 || first.is_instance_of::<crate::values::PyVariantVal>()
                 || first.hasattr("__dict__")?)
@@ -3048,19 +3047,31 @@ pub(crate) fn py_to_value(obj: &Bound<'_, PyAny>) -> 
PyResult<Value> {
         return Ok(Value::List(items));
     }
     if let Ok(tuple) = obj.cast::<pyo3::types::PyTuple>() {
-        let mut items = Vec::with_capacity(tuple.len());
-        for item in tuple.iter() {
-            items.push(py_to_value(&item)?);
+        let mut items = tuple
+            .iter()
+            .map(|item| py_to_value(&item))
+            .collect::<PyResult<Vec<_>>>()?;
+        // Named Rows (tuple subclasses with `__fields__`) become structs on 
the generic
+        // object path, e.g. customized pandas object columns. Unnamed tuples 
stay arrays.
+        if !obj.is_exact_instance_of::<pyo3::types::PyTuple>() {
+            match obj.getattr(pyo3::intern!(obj.py(), "__fields__")) {
+                Ok(fields) => {
+                    let mut fields: Vec<String> = fields.extract()?;
+                    // Row factories may receive fewer values than names, and 
__fields__
+                    // is mutable. Pair them as schema inference does, without 
panicking
+                    // in the native Row constructor when their lengths differ.
+                    fields.truncate(items.len());
+                    items.truncate(fields.len());
+                    return Ok(Value::Struct(Row::new(fields, items).into()));
+                }
+                Err(error)
+                    if 
error.is_instance_of::<pyo3::exceptions::PyAttributeError>(obj.py()) => {}
+                Err(error) => return Err(error),
+            }
         }
         return Ok(Value::List(items));
     }
 
-    // pyspark Row -> struct: its named fields (values already core Values), 
so a nested
-    // Row inside createDataFrame data (e.g. Row(s=Row(x=1))) converts 
recursively.
-    if let Ok(pyrow) = obj.extract::<PyRef<crate::row::PyRow>>() {
-        return Ok(Value::Struct(pyrow.to_row(obj.py())?.into()));
-    }
-
     // dict -> map, its entries in order with keys and values of their own 
types
     if let Ok(dict) = obj.cast::<pyo3::types::PyDict>() {
         let mut map = Vec::with_capacity(dict.len());
diff --git a/crates/spark-connect/src/row.rs b/crates/spark-connect/src/row.rs
index 06edcadd..ec9f22c8 100644
--- a/crates/spark-connect/src/row.rs
+++ b/crates/spark-connect/src/row.rs
@@ -428,6 +428,17 @@ impl Row {
         self.fields.as_slice()
     }
 
+    /// The immutable field-name allocation shared by rows of an Arrow record 
batch.
+    /// Language bindings use its identity to preserve per-batch field-name 
sharing.
+    /// Locally constructed rows may own their names instead.
+    #[doc(hidden)]
+    pub fn shared_fields(&self) -> Option<&Arc<[String]>> {
+        match &self.fields {
+            RowFields::Shared(fields) => Some(fields),
+            RowFields::Owned(_) => None,
+        }
+    }
+
     /// Get values.
     pub fn values(&self) -> &[Value] {
         &self.values
@@ -493,6 +504,18 @@ mod tests {
         assert!(std::ptr::eq(row.fields(), cloned.fields()));
     }
 
+    #[test]
+    fn shared_fields_exposes_only_shared_name_storage() {
+        let fields: Arc<[String]> = vec!["id".to_string()].into();
+        let row = Row::with_shared_fields(Arc::clone(&fields), 
vec![Value::Long(1)]);
+        let cloned = row.clone();
+
+        assert!(Arc::ptr_eq(row.shared_fields().unwrap(), &fields));
+        assert!(Arc::ptr_eq(cloned.shared_fields().unwrap(), &fields));
+        let owned = Row::new(vec!["id".to_string()], vec![Value::Long(1)]);
+        assert!(owned.shared_fields().is_none());
+    }
+
     #[test]
     fn shared_field_names_preserve_order_duplicates_equality_and_debug() {
         let names = vec![
diff --git a/pyproject.toml b/pyproject.toml
index e6f77f83..0fb7b92f 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -44,18 +44,24 @@ Repository = "https://github.com/apache/spark-connect-rust";
 "Bug Tracker" = "https://issues.apache.org/jira/browse/SPARK";
 
 [tool.ruff]
-# Lint only our authored tooling/scripts. The python/pyspark tree is largely 
vendored
-# from Apache Spark (pandas-on-Spark, the drop-in skin) and must stay
-# byte-compatible with upstream, so it is intentionally excluded from our 
linter.
+# Lint our authored tooling/scripts and the embedded Row implementation.
+# The python/pyspark tree is largely vendored from Apache Spark 
(pandas-on-Spark
+# and the drop-in skin). It must stay byte-compatible with upstream and is 
excluded
+# from this linter.
 line-length = 100
 target-version = "py39"
 extend-exclude = ["python/pyspark", "target", ".claude"]
-include = ["scripts/**/*.py"]
+include = ["scripts/**/*.py", "crates/pyspark-rs/src/row.py"]
 
 [tool.ruff.lint]
 select = ["E", "F", "W", "I", "UP", "B"]
 ignore = ["E501"]  # line length handled by the formatter, not enforced hard
 
+[tool.ruff.lint.per-file-ignores]
+# Keep the upstream Row implementation's implicit exception context and percent
+# formatting so error behavior and source comparisons stay faithful to PySpark 
4.3.0.
+"crates/pyspark-rs/src/row.py" = ["B904", "UP031"]
+
 [tool.maturin]
 # Mixed Python/Rust layout: the pure-Python drop-in skin lives under 
python/pyspark,
 # and the PyO3 extension is placed inside it as pyspark._pyspark.
diff --git a/python/.coveragerc b/python/.coveragerc
index 284e3c6e..426e1c51 100644
--- a/python/.coveragerc
+++ b/python/.coveragerc
@@ -3,6 +3,9 @@
 # the Python glue only, excluding vendored third-party code (cloudpickle), the
 # test-support utilities, pandas-on-spark, generated wrappers, and the compiled
 # extension itself.
+# The embedded crates/pyspark-rs/src/row.py runs as pyspark/_row.py.
+# That path has no file and is explicitly omitted. Row tests exercise its 
behavior;
+# Ruff checks the embedded source.
 [run]
 source = pyspark
 branch = True
@@ -62,7 +65,10 @@ omit =
     */pyspark/sql/connect/logging.py
     */pyspark/sql/connect/_typing.py
     */pyspark/sql/connect/conversion.py
+    # The native extension and its embedded Row source, which has a synthetic 
filename.
     */pyspark/_pyspark*
+    */pyspark/_row.py
+    # Test helpers.
     */tests/*
 
 [report]
diff --git a/python/pyspark/sql/types.py b/python/pyspark/sql/types.py
index 33a92e30..09f1df76 100644
--- a/python/pyspark/sql/types.py
+++ b/python/pyspark/sql/types.py
@@ -21,10 +21,11 @@ try:
         # Spark's nanosecond timestamp types (no PySpark 4.3.0 class; not in 
__all__).
         TimestampNTZNanosType, TimestampLTZNanosType,
     )
-    # Row lives in the Rust core; upstream code imports it from 
pyspark.sql.types too.
+    # The extension exports the same tuple-backed Row as the public Python 
imports.
     from pyspark._pyspark import Row
 except ImportError:  # pragma: no cover - defensive fallback when the 
extension is absent
-    # Fallback: define minimal stubs for testing purposes
+    # Minimal DataType stubs for import-only tests. Row and runtime conversion
+    # helpers require the compiled extension.
     class DataType:
         """Base class for all data types."""
         def __init__(self):
@@ -614,12 +615,12 @@ def _udt_converter(dataType):
         fields = _udt_field_converters(dataType)
         if fields is None:
             return None
-        row = Row(*dataType.names)
+        field_names = dataType.names
 
         def convert_struct(value):
             if value is None:
                 return None
-            return row(*[v if f is None else f(v) for f, v in zip(fields, 
value)])
+            return _create_row(field_names, [v if f is None else f(v) for f, v 
in zip(fields, value)])
 
         return convert_struct
     return None
@@ -710,7 +711,6 @@ def _struct_to_internal(dataType, obj):
     if isinstance(obj, VariantVal):
         raise PySparkValueError("Rows cannot be of type VariantVal")
     names, fields = dataType.names, dataType.fields
-    # A Row is a tuple in PySpark.
     if dataType._needSerializeAnyField:
         # Only calling toInternal function for fields that need conversion
         need = dataType._needConversion
@@ -719,7 +719,7 @@ def _struct_to_internal(dataType, obj):
                 f.toInternal(obj.get(n)) if c else obj.get(n)
                 for n, f, c in zip(names, fields, need)
             )
-        elif isinstance(obj, (tuple, list, Row)):
+        elif isinstance(obj, (tuple, list)):
             return tuple(f.toInternal(v) if c else v for f, v, c in 
zip(fields, obj, need))
         elif hasattr(obj, "__dict__"):
             d = obj.__dict__
@@ -729,7 +729,7 @@ def _struct_to_internal(dataType, obj):
     else:
         if isinstance(obj, dict):
             return tuple(obj.get(n) for n in names)
-        elif isinstance(obj, (list, tuple, Row)):
+        elif isinstance(obj, (list, tuple)):
             return tuple(obj)
         elif hasattr(obj, "__dict__"):
             d = obj.__dict__
@@ -822,8 +822,7 @@ def _from_internal(dataType, obj):
 # The private helpers of PySpark's pyspark.sql.types, as of Apache Spark 
v4.3.0: the inference,
 # merging, conversion and verification of the types of Python objects, and 
their tables, which
 # PySpark Connect's modules, Apache Spark's tests and libraries import. They 
are PySpark's, as
-# the client's types share the classes and the attributes of PySpark's; but 
the client's Row is
-# not a tuple, so where PySpark takes a tuple of a Row, they take the client's 
Row too.
+# the client's types share the classes and attributes of PySpark's, including 
its tuple Row.
 import ctypes  # noqa: E402
 import datetime  # noqa: E402
 import decimal  # noqa: E402
@@ -1158,7 +1157,7 @@ def _infer_schema(
     if isinstance(row, dict):
         items = sorted(row.items())
 
-    elif isinstance(row, (tuple, list, Row)):
+    elif isinstance(row, (tuple, list)):
         if hasattr(row, "__fields__"):  # Row
             items = zip(row.__fields__, tuple(row))
         elif hasattr(row, "_fields"):  # namedtuple
@@ -1369,7 +1368,7 @@ def _create_converter(dataType: DataType) -> Callable:
         if obj is None:
             return None
 
-        if isinstance(obj, (tuple, list, Row)):
+        if isinstance(obj, (tuple, list)):
             if convert_fields:
                 return tuple(conv(v) for v, conv in zip(obj, converters))
             else:
@@ -1413,7 +1412,7 @@ _acceptable_types = {
     DayTimeIntervalType: (datetime.timedelta,),
     ArrayType: (list, tuple, array),
     MapType: (dict,),
-    StructType: (tuple, list, dict, Row),
+    StructType: (tuple, list, dict),
     GeometryType: (Geometry,),
     GeographyType: (Geography,),
     VariantType: (
@@ -1428,7 +1427,6 @@ _acceptable_types = {
         datetime.datetime,
         datetime.timedelta,
         tuple,
-        Row,
         list,
         dict,
         array,
@@ -1714,7 +1712,7 @@ def _make_type_verifier(
             if isinstance(obj, dict):
                 for f, verifier in verifiers:
                     verifier(obj.get(f))
-            elif isinstance(obj, (tuple, list, Row)):
+            elif isinstance(obj, (tuple, list)):
                 if len(obj) != len(verifiers):
                     if name is not None:
                         raise PySparkValueError(
diff --git a/python/tests/test_arrow_handoff.py 
b/python/tests/test_arrow_handoff.py
index 91422b7c..70f20f2c 100644
--- a/python/tests/test_arrow_handoff.py
+++ b/python/tests/test_arrow_handoff.py
@@ -249,6 +249,98 @@ def 
test_collect_reads_binary_default_with_arrow_only_schema(protocol):
     assert calls == {"execute": 1, "config": 1}
 
 
[email protected](
+    "action",
+    ["collect", "take", "tail", "first", "head", "head_many", "iterator", 
"prefetch"],
+)
+def test_native_result_actions_return_tuple_rows_recursively(protocol, action):
+    from pyspark import Row
+
+    child_type = pa.struct([pa.field("value", pa.int64())])
+    batch = pa.RecordBatch.from_arrays(
+        [
+            pa.array([1, 2], type=pa.int64()),
+            pa.array([{"value": 3}, None], type=child_type),
+            pa.array([[{"value": 4}, None], []], type=pa.list_(child_type)),
+            pa.array([[("child", {"value": 5})], []], 
type=pa.map_(pa.string(), child_type)),
+        ],
+        names=["id", "child", "children", "by_name"],
+    )
+    with _dataframe(protocol=protocol, batches=[batch]) as (frame, _):
+        if action in ("collect", "first", "head"):
+            result = getattr(frame, action)()
+            rows = result if action == "collect" else [result]
+        elif action in ("take", "tail"):
+            rows = getattr(frame, action)(2)
+        elif action == "head_many":
+            rows = frame.head(2)
+        else:
+            rows = list(frame.toLocalIterator(prefetchPartitions=action == 
"prefetch"))
+
+    assert rows and all(type(row) is Row and isinstance(row, tuple) for row in 
rows)
+    row = rows[0]
+    assert type(row.child) is type(row.children[0]) is 
type(row.by_name["child"]) is Row
+    assert tuple.__getitem__(row.child, 0) == 3
+    assert row.children[1] is None
+    assert row.asDict(recursive=True) == {
+        "id": 1,
+        "child": {"value": 3},
+        "children": [{"value": 4}, None],
+        "by_name": {"child": {"value": 5}},
+    }
+    if len(rows) == 2:
+        assert rows[1] == (2, None, [], {})
+
+
+def test_native_collect_rows_share_mutable_field_names(protocol):
+    batch = pa.RecordBatch.from_arrays([pa.array([1, 2])], names=["value"])
+    with _dataframe(protocol=protocol, batches=[batch.slice(0, 1), 
batch.slice(1, 1)]) as (frame, _):
+        rows = frame.collect()
+
+    assert rows[0].__fields__ is rows[1].__fields__
+    rows[0].__fields__[0] = "renamed"
+    assert rows[1].renamed == 2
+    assert rows[1].asDict() == {"renamed": 2}
+
+
[email protected]("prefetch", [False, True])
[email protected]("one_response", [False, True])
+def test_native_iterator_rows_share_field_names_only_within_each_arrow_batch(
+    protocol, prefetch, one_response
+):
+    batch = pa.RecordBatch.from_arrays([pa.array([1, 2, 3, 4])], 
names=["value"])
+    batches = [batch.slice(0, 2), batch.slice(2, 2)]
+    payloads = None
+    if one_response:
+        sink = pa.BufferOutputStream()
+        with pa.ipc.new_stream(sink, batch.schema) as writer:
+            for part in batches:
+                writer.write_batch(part)
+        payloads = [(batch.num_rows, sink.getvalue().to_pybytes())]
+    with _dataframe(protocol=protocol, batches=batches, payloads=payloads) as 
(frame, _):
+        rows = list(frame.toLocalIterator(prefetchPartitions=prefetch))
+
+    assert rows[0].__fields__ is rows[1].__fields__
+    assert rows[2].__fields__ is rows[3].__fields__
+    assert rows[0].__fields__ is not rows[2].__fields__
+    rows[0].__fields__[0] = "renamed"
+    assert rows[1].renamed == 2
+    assert rows[2].value == 3
+    assert rows[3].asDict() == {"value": 4}
+
+
+def test_native_iterator_ends_after_row_materialization_fails(protocol):
+    # The first timestamp is year 10000, outside Python datetime's range.
+    batch = pa.RecordBatch.from_arrays(
+        [pa.array([253402300800000000, 0], type=pa.timestamp("us"))], 
names=["value"]
+    )
+    with _dataframe(protocol=protocol, batches=[batch]) as (frame, _):
+        rows = frame.toLocalIterator()
+        with pytest.raises(OverflowError):
+            next(rows)
+        assert list(rows) == []
+
+
 def test_handoff_preserves_values_metadata_and_chunks(protocol):
     batch = _mixed_batch()
     batches = [batch.slice(1, 2), batch.slice(0, 1), batch.slice(3, 1)]
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index 86dd2d65..a85ed8b0 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -248,8 +248,7 @@ def test_spatial_and_collated_json_forms_parse():
 
 def test_private_type_helpers_are_those_of_pyspark():
     # As PySpark's pyspark.sql.types: the inference, merging, conversion and 
verification of the
-    # types of Python objects, and their tables. They take the client's Row as 
PySpark's take
-    # theirs, a tuple.
+    # types of Python objects and their tables, including tuple-based Rows.
     from array import array
 
     from pyspark.errors import PySparkTypeError, PySparkValueError
@@ -358,8 +357,7 @@ def _type_verifier_cases():
 
 def test_type_verifiers_take_and_reject_the_values_of_pyspark():
     # As Apache Spark's DataTypeVerificationTests: _make_type_verifier takes 
the values of each
-    # type, of the client's Row too, and raises the errors of PySpark for the 
others, with the
-    # names of the fields.
+    # type, including Rows, and raises the errors of PySpark for the others 
with field names.
     from pyspark.errors import PySparkTypeError, PySparkValueError
     from pyspark.sql.types import _make_type_verifier
 
@@ -383,7 +381,7 @@ def 
test_type_verifiers_take_and_reject_the_values_of_pyspark():
 
 def 
test_types_of_python_objects_are_inferred_merged_and_converted_as_pyspark():
     # As PySpark's _infer_type, _infer_schema, _merge_type and 
_create_converter, of the
-    # values of each type, also nested, of the client's Row too.
+    # values of each type, including nested Rows.
     import datetime
     import decimal
 
@@ -4308,6 +4306,243 @@ def test_row_behaves_as_a_tuple_of_its_values():
     assert pickle.loads(pickle.dumps(Row(1, 2))) == (1, 2)
 
 
+def test_row_is_the_public_tuple_subclass():
+    from pyspark import Row as root_row
+    from pyspark._pyspark import Row as extension_row
+    from pyspark.sql import Row
+    from pyspark.sql.types import Row as types_row
+
+    assert root_row is extension_row is Row is types_row
+    assert Row.__module__ == "pyspark.sql.types"
+    assert Row.__bases__ == (tuple,)
+    row = Row(a=1, child=Row(b=2))
+    assert isinstance(row, tuple)
+    assert tuple.__getitem__(row, 0) == 1
+    assert json.loads(json.dumps(row)) == [1, [2]]
+
+    class DerivedRow(Row):
+        def total(self):
+            return sum(self)
+
+    derived = DerivedRow(a=1, b=2)
+    assert type(derived) is DerivedRow
+    assert isinstance(derived, Row) and isinstance(derived, tuple)
+    assert derived.total() == 3 and derived.asDict() == {"a": 1, "b": 2}
+
+
+def test_row_field_names_keep_their_identity_and_mutations():
+    from pyspark.errors import PySparkKeyError
+    from pyspark.sql import Row
+    from pyspark.sql.types import _create_row
+
+    row = Row(a=1, b=2)
+    fields = row.__fields__
+    assert row.__fields__ is fields
+    fields[0] = "renamed"
+    assert row.renamed == 1 and "a" not in row
+    assert row.asDict() == {"renamed": 1, "b": 2}
+    assert repr(row) == "Row(renamed=1, b=2)"
+    fields.append("missing")
+    assert len(row) == 2 and tuple(row) == (1, 2)
+    assert "missing" in row
+    with pytest.raises(PySparkKeyError):
+        row["missing"]
+
+    shared = ["x", "y"]
+    left = _create_row(shared, (1, 2))
+    right = _create_row(shared, (3, 4))
+    assert left.__fields__ is right.__fields__ is shared
+    shared[1] = "z"
+    assert (left.z, right.z) == (2, 4)
+    row.__fields__ = shared
+    assert row.__fields__ is shared and row.z == 2
+
+    factory = Row("first", "second")
+    made = factory(1)
+    assert made.__fields__ is factory
+    assert made.asDict() == {"first": 1}
+    assert repr(made) == "Row(first=1)"
+
+
[email protected]("serializer", ["pickle", "cloudpickle"])
+def test_row_serialization_preserves_public_type_and_shared_names(serializer):
+    from pyspark import cloudpickle
+    from pyspark.sql import Row
+    from pyspark.sql.types import _create_row
+
+    codec = pickle if serializer == "pickle" else cloudpickle
+    fields = ["value"]
+    rows = [_create_row(fields, (Row(child=1),)), _create_row(fields, 
(Row(child=2),))]
+    restored = codec.loads(codec.dumps(rows))
+    assert all(type(row) is Row and type(row.value) is Row for row in restored)
+    assert restored[0].__fields__ is restored[1].__fields__
+    restored[0].__fields__[0] = "renamed"
+    assert restored[1].renamed == Row(child=2)
+    unnamed = codec.loads(codec.dumps(Row(1, 2)))
+    assert type(unnamed) is Row and isinstance(unnamed, tuple)
+    assert not hasattr(unnamed, "__fields__")
+    assert codec.loads(codec.dumps(Row)) is Row
+
+
+def _tuple_row_results(mock_connect, table, schema, method):
+    import pyarrow as pa
+
+    spark, service = mock_connect
+    sink = pa.BufferOutputStream()
+    with pa.ipc.new_stream(sink, table.schema) as writer:
+        writer.write_table(table, max_chunksize=2)
+    service.execute_payload = message_field(7, schema) + message_field(
+        2, varint_field(1, table.num_rows) + message_field(2, 
sink.getvalue().to_pybytes()))
+    frame = spark.range(table.num_rows)
+    return frame.collect() if method == "collect" else list(
+        frame.toLocalIterator(method == "prefetch"))
+
+
[email protected]("method", ["collect", "iterator", "prefetch"])
+def test_collected_tuple_rows_share_names_by_schema_path(mock_connect, method):
+    import pyarrow as pa
+    from _mock_connect_server import data_type_proto
+
+    # PySpark keeps top-level names per collection or iterator batch, but 
nested
+    # names belong to the schema and span batches. Equal sibling names are 
independent.
+    child = pa.struct([("v", pa.int64())])
+    values = [{"v": index} for index in range(4)]
+    table = pa.table({
+        "left": pa.array(values, child),
+        "right": pa.array(values, child),
+        "items": pa.array([[value, value] for value in values], 
pa.list_(child)),
+        "lookup": pa.array([[('a', value), ('b', value)] for value in values],
+                           pa.map_(pa.string(), child)),
+        "struct_map": pa.array([[(value, value)] for value in values], 
pa.map_(child, child)),
+    })
+    rows = _tuple_row_results(mock_connect, table, data_type_proto(ddl_json(
+        "left struct<v:bigint>, right struct<v:bigint>, "
+        "items array<struct<v:bigint>>, lookup map<string,struct<v:bigint>>, "
+        "struct_map map<struct<v:bigint>,struct<v:bigint>>")), method)
+    assert len(rows) == 4
+    for index, row in enumerate(rows):
+        assert type(row) is Row and isinstance(row, tuple)
+        assert type(row.left) is Row and row.left == (index,)
+        assert type(row.items[0]) is Row and row.items == [(index,), (index,)]
+        assert type(row.lookup['a']) is Row and row.lookup == {'a': (index,), 
'b': (index,)}
+        assert type(row.left.__fields__) is list
+        assert row.left.__fields__ is rows[0].left.__fields__
+        assert row.right.__fields__ is rows[0].right.__fields__
+        assert row.items[0].__fields__ is row.items[1].__fields__ is 
rows[0].items[0].__fields__
+        assert row.lookup['a'].__fields__ is row.lookup['b'].__fields__ is 
rows[0].lookup['a'].__fields__
+        key, value = next(iter(row.struct_map.items()))
+        assert type(key) is type(value) is Row and key == value == (index,)
+        first_key, first_value = next(iter(rows[0].struct_map.items()))
+        assert key.__fields__ is first_key.__fields__
+        assert value.__fields__ is first_value.__fields__
+        assert key.__fields__ is not value.__fields__
+    assert len({id(rows[0][index].__fields__) for index in (0, 1)}) == 2
+    assert rows[0].left.__fields__ is not rows[0].items[0].__fields__
+    assert rows[0].left.__fields__ is not rows[0].lookup['a'].__fields__
+
+    rows[0].left.__fields__[0] = "renamed"
+    rows[0].items[0].__fields__[0] = "array_value"
+    rows[0].lookup['a'].__fields__[0] = "map_value"
+    next(iter(rows[0].struct_map)).__fields__[0] = "map_key"
+    assert [row.left.renamed for row in rows] == list(range(4))
+    assert [row.right.v for row in rows] == list(range(4))
+    assert [row.items[1].array_value for row in rows] == list(range(4))
+    assert [row.lookup['b'].map_value for row in rows] == list(range(4))
+    assert [next(iter(row.struct_map)).map_key for row in rows] == 
list(range(4))
+
+    assert rows[0].__fields__ is rows[1].__fields__
+    assert rows[2].__fields__ is rows[3].__fields__
+    assert (rows[0].__fields__ is rows[2].__fields__) == (method == "collect")
+    rows[0].__fields__[0] = "renamed_left"
+    assert rows[1].renamed_left.renamed == 1
+    if method != "collect":
+        assert rows[2].left.renamed == 2
+
+
+class _TupleBinaryUDT(T.UserDefinedType):
+    @classmethod
+    def sqlType(cls):
+        return T.StructType([T.StructField("b", T.BinaryType())])
+
+    @classmethod
+    def module(cls):
+        return __name__
+
+    def deserialize(self, datum):
+        return datum, datum.__fields__
+
+
+def _tuple_row_type_proto(data_type):
+    from _mock_connect_server import data_type_proto
+
+    if isinstance(data_type, T.UserDefinedType):
+        value = data_type.jsonValue()
+        # DataType.udt=23: type=1, python_class=3, serialized_python_class=4, 
sql_type=5.
+        return message_field(23, message_field(1, "udt") + message_field(3, 
value["pyClass"])
+                             + message_field(4, value["serializedClass"])
+                             + message_field(5, 
_tuple_row_type_proto(data_type.sqlType())))
+    if isinstance(data_type, T.StructType):
+        return message_field(21, b"".join(message_field(
+            1, message_field(1, field.name) + message_field(2, 
_tuple_row_type_proto(field.dataType))
+            + varint_field(3, int(field.nullable))) for field in 
data_type.fields))
+    return data_type_proto(data_type.jsonValue())
+
+
[email protected]("method", ["collect", "iterator", "prefetch"])
+def test_tuple_rows_keep_configured_binary_and_udt_inputs(mock_connect, 
method):
+    import pyarrow as pa
+
+    _, service = mock_connect
+    service.confs = {"spark.sql.execution.pyspark.binaryAsBytes": "false"}
+    child = pa.struct([("b", pa.binary())])
+    converted = pa.struct([("b", pa.binary()), ("u", child)])
+    values = [{"b": bytes([index])} for index in range(4)]
+    table = pa.table({
+        "s": pa.array(values, child),
+        "converted": pa.array([{"b": value["b"], "u": value} for value in 
values], converted),
+    })
+    schema = T.StructType([
+        T.StructField("s", T.StructType([T.StructField("b", T.BinaryType())])),
+        T.StructField("converted", T.StructType([
+            T.StructField("b", T.BinaryType()), T.StructField("u", 
_TupleBinaryUDT()),
+        ])),
+    ])
+    rows = _tuple_row_results(mock_connect, table, 
_tuple_row_type_proto(schema), method)
+    assert len(rows) == 4
+    for index, row in enumerate(rows):
+        assert type(row) is Row and type(row.s) is Row and type(row.converted) 
is Row
+        assert type(row.s.b) is type(row.converted.b) is bytearray
+        assert row.s.b == row.converted.b == bytes([index])
+        udt_input, input_fields = row.converted.u
+        assert type(udt_input) is Row and isinstance(udt_input, tuple)
+        assert type(udt_input.b) is bytearray and udt_input.b == bytes([index])
+        assert input_fields is udt_input.__fields__ and type(input_fields) is 
list
+        assert row.converted.__fields__ is rows[0].converted.__fields__
+    assert type(rows[0].converted.__fields__) is list
+    rows[0].converted.__fields__[0] = "renamed_binary"
+    assert rows[-1].converted.renamed_binary == b"\x03"
+
+
[email protected]("spatial_type", [T.GeometryType, T.GeographyType])
+def 
test_tuple_struct_converter_keeps_spatial_payloads_and_mutable_names(spatial_type):
+    from pyspark.sql.types import _udt_field_converters
+
+    wkb = bytes.fromhex("0101000000000000000000f03f0000000000000040")
+    schema = T.StructType([T.StructField("s", T.StructType([
+        T.StructField("b", T.BinaryType()), T.StructField("g", 
spatial_type(4326)),
+    ]))])
+    convert = _udt_field_converters(schema)[0]
+    left = convert(Row(b=bytearray(b"x"), g=Row(wkb=wkb, srid=4326)))
+    right = convert(Row(b=bytearray(b"y"), g=Row(wkb=wkb, srid=4326)))
+    assert type(left) is Row and isinstance(left, tuple)
+    assert type(left.b) is bytearray and left.b == b"x"
+    assert type(left.g.wkb) is bytes and left.g.wkb == wkb
+    assert left.g == right.g
+    assert type(left.__fields__) is list and left.__fields__ is 
right.__fields__
+    left.__fields__[0] = "renamed_binary"
+    assert right.renamed_binary == b"y"
+
+
 def test_row_fields():
     from pyspark.errors import (
         PySparkAttributeError,
@@ -7710,9 +7945,9 @@ def 
test_create_dataframe_raises_recursion_error_for_a_value_that_contains_itsel
             _local_create_dataframe([(value,)])
 
 
-# py_to_value converts the cells of object-dtype pandas input, as well as the 
values of
-# fillna, pivot and Row conversion; local rows go through infer.convert 
instead, so the
-# "local" source is a control for values that the exact-number fast path must 
not change.
+# py_to_value converts object-dtype cells of customized pandas frames, plus 
fillna
+# and pivot values; local rows use infer.convert. The "local" source below is a
+# control for values that the exact-number fast path must not change.
 def _RowPathFrame(pd):
     """A subclass of pandas.DataFrame: createDataFrame converts a customized 
frame with its
     row path (py_to_value), where it converts a plain one with PySpark's Arrow 
conversion."""
@@ -7847,6 +8082,98 @@ def 
test_local_input_native_numbers_inside_nested_values(mock_connect):
     assert dict(actual["mapping"]) == {"x": 2**63 - 1}
 
 
[email protected]("source", ["local", "pandas"])
[email protected]("make, expected", [
+    (lambda: Row(a=1, b="x"), {"a": 1, "b": "x"}),
+    (lambda: Row("a", "b")(1, "x"), {"a": 1, "b": "x"}),
+    (lambda: Row("a", "b")(1), {"a": 1}),
+])
+def test_tuple_rows_remain_named_structs_in_native_input(mock_connect, source, 
make, expected):
+    # A customized pandas subclass retains the native row path. Plain pandas
+    # frames take the separate Arrow path checked below.
+    value = make()
+    actual = _native_number_input_table(
+        mock_connect, [(value,)], source=source
+    ).column(0)[0].as_py()
+    assert actual == expected
+
+
[email protected]("source", ["local", "pandas"])
+def test_unnamed_nested_row_inference_matches_pyspark(mock_connect, source):
+    import pyarrow as pa
+
+    # Local inference makes positional structs; the customized pandas row path
+    # keeps unnamed tuple values as arrays.
+    column = _native_number_input_table(
+        mock_connect, [(Row(1, 2),)], source=source
+    ).column(0)
+    if source == "local":
+        assert column.type == pa.struct([pa.field("_1", pa.int64()), 
pa.field("_2", pa.int64())])
+        assert column[0].as_py() == {"_1": 1, "_2": 2}
+    else:
+        assert pa.types.is_list(column.type) or 
pa.types.is_large_list(column.type)
+        assert column.type.value_type == pa.int64()
+        assert column[0].as_py() == [1, 2]
+
+
[email protected]("value", [Row(1, 2), Row(a=1, b=2)])
+def test_plain_pandas_tuple_rows_keep_arrow_array_inference(mock_connect, 
value):
+    import pandas as pd
+    import pyarrow as pa
+
+    # The default branch preserves this plain frame and its Arrow conversion 
route.
+    table = _native_number_input_table(
+        mock_connect, pd.DataFrame({"value": [value]}, dtype=object))
+    table.validate(full=True)
+    assert (table.num_rows, table.num_columns) == (1, 1)
+    # IPC keeps temporary names; the enclosing ToDF applies the public label.
+    assert table.column_names == ["_0"]
+    _, service = mock_connect
+    request = parse_message(service.execute_requests[-1])
+    relation = parse_message(parse_message(request[3][0])[1][0])
+    rename = parse_message(relation[18][0])
+    assert [name.decode() for name in rename[2]] == ["value"]
+    column = table.column(0)
+    assert pa.types.is_list(column.type) or pa.types.is_large_list(column.type)
+    assert column.type.value_type == pa.int64()
+    assert column[0].as_py() == [1, 2]
+
+
+def test_customized_pandas_exact_nested_tuple_rows_remain_arrays(mock_connect):
+    import pyarrow as pa
+
+    values = [
+        ((1, None, -3), (), None, (2**63 - 1, -(2**63))),
+        (),
+        None,
+        ((4,), (5, 6)),
+    ]
+    table = _native_number_input_table(
+        mock_connect, [(value,) for value in values], source="pandas")
+    table.validate(full=True)
+    assert (table.num_rows, table.num_columns) == (4, 1)
+    assert table.column_names == ["value"]
+    assert table.column(0).type == pa.list_(pa.list_(pa.int64()))
+    assert table.column(0).to_pylist() == [
+        [[1, None, -3], [], None, [2**63 - 1, -(2**63)]],
+        [],
+        None,
+        [[4], [5, 6]],
+    ]
+
+
+def test_pandas_tuple_field_lookup_propagates_errors(mock_connect):
+    class InvalidFields(tuple):
+        @property
+        def __fields__(self):
+            raise RuntimeError("field lookup failed")
+
+    with pytest.raises(RuntimeError, match="^field lookup failed$"):
+        _native_number_input_table(
+            mock_connect, [(InvalidFields((1,)),)], source="pandas"
+        )
+
+
 # Exact list/tuple conversion and pandas input IPC regressions.
 from datetime import date as container_input_date, datetime as 
container_input_datetime
 from decimal import Decimal as ContainerInputDecimal


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to