This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new e8f889d  [SPARK-61108][CONNECT][PYTHON] Name the nested fields of 
collected rows by the schema, and raise for calendar intervals in collect, in 
the Python client as PySpark does
e8f889d is described below

commit e8f889d2516d1cb30b9cda37649a11466369179c
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sat Nov 7 18:53:27 2026 +0900

    [SPARK-61108][CONNECT][PYTHON] Name the nested fields of collected rows by 
the schema, and raise for calendar intervals in collect, in the Python client 
as PySpark does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-61108](https://issues.apache.org/jira/browse/SPARK-61108): collected 
rows now name the fields of nested structs by the schema, and `collect` raises 
for calendar intervals, as PySpark Connect does.
    
    As PySpark Connect's `ArrowTableToRowsConversion`, which builds rows by the 
schema of the result:
    - The core decodes the Arrow data of a result by the schema that the server 
sends before it (`decode_arrow_batch_of`, `ArrowValueLayout::of` in 
`crates/spark-connect/src/dataframe.rs`).
    - So the fields of nested structs, also in arrays, maps and user-defined 
types, have the names of the schema. These keep the duplicate names that the 
server's Arrow data numbers as `a_0`, `a_1`.
    - This holds for `collect`, `first`, `head`, `take`, `tail` and 
`toLocalIterator`.
    
    As PySpark Connect's `collect`, which checks the Arrow schema of the result 
with `from_arrow_schema`, `collect`, `first`, `head`, `take` and `tail` raise 
`UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION` for a calendar interval: 
"month_day_nano_interval is not supported in conversion to Arrow". Its 
`toLocalIterator` does not check, and neither does the client's.
    
    ### Why are the changes needed?
    
    The client named nested fields by the Arrow data. For example, `SELECT 
named_struct('a', 1, 'a', 2) AS v` collected `Row(v=Row(a_0=1, a_1=2))` instead 
of `Row(v=Row(a=1, a=2))`, and `v.a` raised `ATTRIBUTE_NOT_SUPPORTED`. It also 
collected a calendar interval as a string such as "14 months 25 days 
18367500000000 ns", where PySpark raises.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. Collected rows have the field names of PySpark 4.3.0, and collecting a 
calendar interval raises its error.
    
    ### How was this patch tested?
    
    - A new live test checks the names, the values and `asDict` of nested 
structs of duplicate names, also from `toLocalIterator`, and the error of a 
calendar interval. It passes with official PySpark 4.3.0 and fails without this 
change.
    - A new offline test checks the names of rows decoded by the schema of the 
response, against a mock Connect server, and that `collect` raises for a 
calendar interval where `toLocalIterator` does not.
    - A new core test checks the names of nested structs, also in arrays, 
decoded with a schema, and without one or of another shape.
    - On a local server, 156 cases of `collect`, `first`, `head`, `take`, 
`tail` and `toLocalIterator` of a value of each type give the same results as 
official PySpark 4.3.0.
    - Still different: `toLocalIterator` of a calendar interval gives the 
client's string, where PySpark gives PyArrow's `MonthDayNano`. Returning that 
needs a new variant of the core's `Value`.
    - The lib tests of both crates (590 passed), the build of all tests, also 
with `wasm-udf`, the offline Python suites (2056 passed, 12 skipped), and the 
live drop-in and Arrow suites (160 passed) and live cases (10 passed) against a 
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #305 from HyukjinKwon/SPARK-61108-collect-nested-names.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/dataframe.rs    |  47 +++++++--
 crates/spark-connect/src/dataframe.rs | 175 +++++++++++++++++++++++++++++++---
 python/tests/test_dropin_e2e.py       |  17 ++++
 python/tests/test_dropin_offline.py   |  37 +++++++
 4 files changed, 255 insertions(+), 21 deletions(-)

diff --git a/crates/pyspark-rs/src/dataframe.rs 
b/crates/pyspark-rs/src/dataframe.rs
index 16e6fd1..597c843 100644
--- a/crates/pyspark-rs/src/dataframe.rs
+++ b/crates/pyspark-rs/src/dataframe.rs
@@ -3727,26 +3727,58 @@ fn udt_field_converters(
 /// Whether `data_type` holds a year-month interval, also nested in an array, 
a map, a struct or
 /// a user-defined type (`pyspark.sql.types._has_type`).
 fn has_year_month_interval(data_type: &spark_connect::types::DataType) -> bool 
{
+    has_type(data_type, &|t| {
+        matches!(t, spark_connect::types::DataType::YearMonthInterval { .. })
+    })
+}
+
+/// Whether `data_type` is of `is`, also nested in an array, a map, a struct 
or a user-defined
+/// type.
+fn has_type(
+    data_type: &spark_connect::types::DataType,
+    is: &dyn Fn(&spark_connect::types::DataType) -> bool,
+) -> bool {
     use spark_connect::types::DataType;
+    if is(data_type) {
+        return true;
+    }
     match data_type {
-        DataType::YearMonthInterval { .. } => true,
-        DataType::Struct { fields } => fields
-            .iter()
-            .any(|field| has_year_month_interval(&field.data_type)),
-        DataType::Array { element_type, .. } => 
has_year_month_interval(element_type),
+        DataType::Struct { fields } => fields.iter().any(|field| 
has_type(&field.data_type, is)),
+        DataType::Array { element_type, .. } => has_type(element_type, is),
         DataType::Map {
             key_type,
             value_type,
             ..
-        } => has_year_month_interval(key_type) || 
has_year_month_interval(value_type),
+        } => has_type(key_type, is) || has_type(value_type, is),
         DataType::Udt {
             sql_type: Some(sql_type),
             ..
-        } => has_year_month_interval(sql_type),
+        } => has_type(sql_type, is),
         _ => false,
     }
 }
 
+/// Raises as PySpark's `collect` does for a result of `schema` with a 
calendar interval: it
+/// checks the Arrow schema of the result with `from_arrow_schema`, which has 
no type of the
+/// Arrow `month_day_nano_interval` of a calendar interval. Its 
`toLocalIterator` does not.
+fn check_arrow_schema(
+    py: Python<'_>,
+    schema: Option<&spark_connect::types::DataType>,
+) -> PyResult<()> {
+    let calendar = |t: &spark_connect::types::DataType| {
+        matches!(t, spark_connect::types::DataType::CalendarInterval)
+    };
+    if schema.is_some_and(|schema| has_type(schema, &calendar)) {
+        return Err(crate::column::raise_pyspark(
+            py,
+            "PySparkTypeError",
+            "UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION",
+            &[("data_type", "month_day_nano_interval")],
+        ));
+    }
+    Ok(())
+}
+
 /// Raises as PySpark's Arrow conversion of the rows does for a result of 
`schema` with a
 /// year-month interval, which Spark Connect cannot collect: the error is 
raised even when the
 /// result has no rows.
@@ -3775,6 +3807,7 @@ fn rows_to_py(
     rows: Vec<spark_connect::row::Row>,
     schema: Option<&spark_connect::types::DataType>,
 ) -> PyResult<Vec<PyRow>> {
+    check_arrow_schema(py, schema)?;
     check_collectable(py, schema)?;
     let converters = udt_field_converters(py, schema)?;
     Ok(rows
diff --git a/crates/spark-connect/src/dataframe.rs 
b/crates/spark-connect/src/dataframe.rs
index 587635c..fc826d5 100644
--- a/crates/spark-connect/src/dataframe.rs
+++ b/crates/spark-connect/src/dataframe.rs
@@ -147,7 +147,8 @@ impl LocalRowIterator {
                         if let 
Some(proto::execute_plan_response::ResponseType::ArrowBatch(batch)) =
                             resp.response_type
                         {
-                            return Some(decode_arrow_batch(&batch));
+                            let schema = self.schema.lock().unwrap_or_else(|p| 
p.into_inner());
+                            return Some(decode_arrow_batch_of(&batch, 
schema.as_ref()));
                         }
                         // Metrics/progress response: keep pulling for a batch.
                     }
@@ -221,7 +222,11 @@ fn spawn_prefetch(
                     if let 
Some(proto::execute_plan_response::ResponseType::ArrowBatch(batch)) =
                         resp.response_type
                     {
-                        match decode_arrow_batch(&batch) {
+                        let decoded = {
+                            let schema = schema.lock().unwrap_or_else(|p| 
p.into_inner());
+                            decode_arrow_batch_of(&batch, schema.as_ref())
+                        };
+                        match decoded {
                             Ok(rows) => {
                                 // A send error means the consumer dropped the
                                 // iterator; stop fetching.
@@ -819,7 +824,7 @@ impl DataFrame {
             if let 
Some(proto::execute_plan_response::ResponseType::ArrowBatch(batch)) =
                 resp.response_type
             {
-                let batch_rows = decode_arrow_batch(&batch)?;
+                let batch_rows = decode_arrow_batch_of(&batch, 
schema.as_ref())?;
                 rows.extend(batch_rows);
             }
         }
@@ -2317,6 +2322,26 @@ fn random_seed() -> i64 {
 /// Decode an Arrow batch into rows.
 pub(crate) fn decode_arrow_batch(
     batch: &proto::execute_plan_response::ArrowBatch,
+) -> Result<Vec<Row>> {
+    decode_arrow_batch_of(batch, None)
+}
+
+/// The fields of `schema`, a struct of `columns` fields, for the columns of a 
batch: none of
+/// another schema, whose names the batch keeps.
+fn spark_fields(schema: Option<&DataType>, columns: usize) -> 
Option<&[crate::types::StructField]> {
+    match schema {
+        Some(DataType::Struct { fields }) if fields.len() == columns => 
Some(fields),
+        _ => None,
+    }
+}
+
+/// The rows of `batch`, of the names of `schema`, the schema of the result 
that the server sends
+/// before the rows, if any: as PySpark's `ArrowTableToRowsConversion` builds 
them by the schema,
+/// the fields of nested structs have the names of the schema, which keep the 
duplicate names
+/// that the Arrow data of the server numbers (`a_0`, `a_1`).
+pub(crate) fn decode_arrow_batch_of(
+    batch: &proto::execute_plan_response::ArrowBatch,
+    spark_schema: Option<&DataType>,
 ) -> Result<Vec<Row>> {
     use arrow::ipc::reader::StreamReader;
     use std::io::Cursor;
@@ -2342,15 +2367,22 @@ pub(crate) fn decode_arrow_batch(
         if num_rows == 0 {
             continue;
         }
-        let field_names = schema
-            .fields()
-            .iter()
-            .map(|field| field.name().clone())
-            .collect::<Arc<[String]>>();
+        let fields = spark_fields(spark_schema, record_batch.num_columns());
+        let field_names = match fields {
+            Some(fields) => fields.iter().map(|field| 
field.name.clone()).collect(),
+            None => schema
+                .fields()
+                .iter()
+                .map(|field| field.name().clone())
+                .collect::<Arc<[String]>>(),
+        };
         let layouts = record_batch
             .columns()
             .iter()
-            .map(|column| ArrowValueLayout::new(column.as_ref()))
+            .enumerate()
+            .map(|(i, column)| {
+                ArrowValueLayout::of(column.as_ref(), fields.map(|fields| 
&fields[i].data_type))
+            })
             .collect::<Vec<_>>();
 
         for row_idx in 0..num_rows {
@@ -2551,9 +2583,26 @@ enum ArrowValueDecoder<'a> {
 }
 
 impl<'a> ArrowValueLayout<'a> {
+    #[cfg(test)]
     fn new(array: &'a dyn arrow::array::Array) -> Self {
+        Self::of(array, None)
+    }
+
+    /// The layout of `array`, of the Spark type `data_type` if known: the 
fields of its structs,
+    /// also in arrays, maps and user-defined types, have the names of the 
type.
+    fn of(array: &'a dyn arrow::array::Array, data_type: Option<&DataType>) -> 
Self {
         use arrow::array::*;
 
+        // A user-defined type is in the Arrow data of its SQL type.
+        let mut data_type = data_type;
+        while let Some(DataType::Udt {
+            sql_type: Some(sql_type),
+            ..
+        }) = data_type
+        {
+            data_type = Some(sql_type);
+        }
+
         // Resolution never fails: null parents and empty containers must not
         // reject unsupported children that are never read.
         let any = array.as_any();
@@ -2628,9 +2677,13 @@ impl<'a> ArrowValueLayout<'a> {
         } else if let Some(array) = 
any.downcast_ref::<IntervalMonthDayNanoArray>() {
             ArrowValueDecoder::IntervalMonthDayNano(array)
         } else if let Some(array) = any.downcast_ref::<ListArray>() {
+            let element_type = match data_type {
+                Some(DataType::Array { element_type, .. }) => 
Some(element_type.as_ref()),
+                _ => None,
+            };
             ArrowValueDecoder::List {
                 array,
-                child: Box::new(Self::new(array.values().as_ref())),
+                child: Box::new(Self::of(array.values().as_ref(), 
element_type)),
             }
         } else if let Some(array) = any.downcast_ref::<StructArray>() {
             let fields = array.fields();
@@ -2640,12 +2693,27 @@ impl<'a> ArrowValueLayout<'a> {
                     .get("variant")
                     .is_some_and(|value| value == "true")
             });
+            let spark_fields = match data_type {
+                Some(DataType::Struct { fields: spark }) if spark.len() == 
fields.len() => {
+                    Some(spark)
+                }
+                _ => None,
+            };
             ArrowValueDecoder::Struct {
-                names: fields.iter().map(|field| 
field.name().clone()).collect(),
+                names: match spark_fields {
+                    Some(spark) => spark.iter().map(|field| 
field.name.clone()).collect(),
+                    None => fields.iter().map(|field| 
field.name().clone()).collect(),
+                },
                 children: array
                     .columns()
                     .iter()
-                    .map(|child| Self::new(child.as_ref()))
+                    .enumerate()
+                    .map(|(i, child)| {
+                        Self::of(
+                            child.as_ref(),
+                            spark_fields.map(|spark| &spark[i].data_type),
+                        )
+                    })
                     .collect(),
                 variant_fields: is_variant.then(|| {
                     (
@@ -2655,11 +2723,19 @@ impl<'a> ArrowValueLayout<'a> {
                 }),
             }
         } else if let Some(array) = any.downcast_ref::<MapArray>() {
+            let (key_type, value_type) = match data_type {
+                Some(DataType::Map {
+                    key_type,
+                    value_type,
+                    ..
+                }) => (Some(key_type.as_ref()), Some(value_type.as_ref())),
+                _ => (None, None),
+            };
             ArrowValueDecoder::Map {
                 array,
                 children: Box::new((
-                    Self::new(array.keys().as_ref()),
-                    Self::new(array.values().as_ref()),
+                    Self::of(array.keys().as_ref(), key_type),
+                    Self::of(array.values().as_ref(), value_type),
                 )),
             }
         } else {
@@ -2952,6 +3028,77 @@ mod nested_decode_tests {
         assert_eq!(rows[2].get(1), Some(&Value::String("é\0雪".into())));
     }
 
+    #[test]
+    fn nested_struct_fields_take_the_names_of_the_schema_of_the_result() {
+        // As PySpark's ArrowTableToRowsConversion, which builds the rows by 
the schema of the
+        // result: the fields of nested structs, also in arrays and maps, have 
the names of the
+        // schema, which keep the duplicate names that the server's Arrow data 
numbers.
+        use crate::types::{DataType as Spark, StructField as SparkField};
+        let field = |name: &str, data_type: Spark| SparkField {
+            name: name.to_string(),
+            data_type,
+            nullable: true,
+            metadata: Default::default(),
+        };
+        let pair = || {
+            StructArray::from(vec![
+                (
+                    Arc::new(Field::new("a_0", DataType::Int32, true)),
+                    Arc::new(Int32Array::from(vec![1])) as ArrayRef,
+                ),
+                (
+                    Arc::new(Field::new("a_1", DataType::Int32, true)),
+                    Arc::new(Int32Array::from(vec![2])) as ArrayRef,
+                ),
+            ])
+        };
+        let list = ListArray::new(
+            Arc::new(Field::new("element", pair().data_type().clone(), true)),
+            arrow::buffer::OffsetBuffer::from_lengths([1]),
+            Arc::new(pair()),
+            None,
+        );
+        let batch = encoded_batches(
+            vec![("v", Arc::new(pair()) as ArrayRef), ("w", Arc::new(list))],
+            1,
+            false,
+        );
+        let pair_type = || Spark::Struct {
+            fields: vec![field("a", Spark::Integer), field("a", 
Spark::Integer)],
+        };
+        let schema = Spark::Struct {
+            fields: vec![
+                field("v", pair_type()),
+                field(
+                    "w",
+                    Spark::Array {
+                        element_type: Box::new(pair_type()),
+                        contains_null: true,
+                    },
+                ),
+            ],
+        };
+        let rows = decode_arrow_batch_of(&batch, Some(&schema)).unwrap();
+        assert_eq!(rows[0].fields(), &["v", "w"]);
+        assert_eq!(as_struct(rows[0].get(0).unwrap()).fields(), &["a", "a"]);
+        let Some(Value::List(items)) = rows[0].get(1) else {
+            panic!("expected a list");
+        };
+        assert_eq!(as_struct(&items[0]).fields(), &["a", "a"]);
+        assert_eq!(
+            as_struct(&items[0]).values(),
+            &[Value::Integer(1), Value::Integer(2)]
+        );
+        // Without the schema, or of a schema of another shape, the names of 
the Arrow data.
+        let other = Spark::Struct {
+            fields: vec![field("v", Spark::Integer)],
+        };
+        for schema in [None, Some(&other)] {
+            let rows = decode_arrow_batch_of(&batch, schema).unwrap();
+            assert_eq!(as_struct(rows[0].get(0).unwrap()).fields(), &["a_0", 
"a_1"]);
+        }
+    }
+
     #[test]
     fn sliced_parents_decode_children_that_were_already_sliced() {
         let leaf = StructArray::from(vec![(
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index f3be9c7..d7dc597 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -1355,6 +1355,23 @@ def 
test_writer_v2_takes_its_arguments_as_pyspark(session):
         session.sql(f"DROP TABLE IF EXISTS {name}")
 
 
+def 
test_collected_rows_take_the_names_of_the_schema_of_the_result_as_pyspark(session):
+    # As PySpark Connect's: the fields of nested structs have the names of the 
schema, which keep
+    # duplicate names, and collect raises for a calendar interval.
+    from pyspark.errors import PySparkTypeError
+
+    df = session.sql(
+        "SELECT named_struct('a', 1, 'a', 2) AS v, array(named_struct('b', 1, 
'b', 'x')) AS w")
+    row = df.collect()[0]
+    assert row.v.__fields__ == ["a", "a"] and tuple(row.v) == (1, 2) and 
row.v.a == 1
+    assert row.w[0].__fields__ == ["b", "b"]
+    assert row.asDict(recursive=True) == {"v": {"a": 2}, "w": [{"b": "x"}]}
+    assert next(df.toLocalIterator()).v.__fields__ == ["a", "a"]
+    with pytest.raises(PySparkTypeError) as e:
+        session.sql("SELECT make_interval(1) AS v").collect()
+    assert e.value.getCondition() == 
"UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION"
+    assert e.value.getMessageParameters() == {"data_type": 
"month_day_nano_interval"}
+
 def test_collect_reads_a_result_across_stream_ends(session):
     # Takes several response streams when the server's senderMaxStreamSize is 
small.
     n = 300_000
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index f1488a7..1f546c0 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -663,6 +663,43 @@ def 
test_builder_options_are_set_on_the_session_as_pyspark(mock_connect):
     assert len(service.set_confs) == 1
 
 
+def 
test_collected_rows_take_the_names_of_the_schema_of_the_result_as_pyspark(mock_connect):
+    # As PySpark Connect's, which builds the rows by the schema of the result: 
the fields of
+    # nested structs have the names of the schema, which keep the duplicate 
names that the
+    # server's Arrow data numbers; collect raises for a calendar interval, 
whose Arrow type
+    # from_arrow_schema does not convert, while toLocalIterator does not check 
it.
+    import pyarrow as pa
+    from _mock_connect_server import data_type_proto
+    from pyspark.errors import PySparkTypeError
+
+    spark, service = mock_connect
+
+    def respond(schema, table):
+        sink = pa.BufferOutputStream()
+        with pa.ipc.new_stream(sink, table.schema) as writer:
+            writer.write_table(table)
+        # ExecutePlanResponse.schema=7 and arrow_batch=2: 
ArrowBatch.row_count=1, data=2.
+        service.execute_payload = message_field(7, schema) + message_field(
+            2, varint_field(1, table.num_rows) + message_field(2, 
sink.getvalue().to_pybytes()))
+
+    field = {"name": "a", "type": "integer", "nullable": True, "metadata": {}}
+    pair = {"type": "struct", "fields": [field, field]}
+    respond(data_type_proto({"type": "struct", "fields": [
+        {"name": "v", "type": pair, "nullable": True, "metadata": {}}]}),
+        pa.table({"v": pa.array([{"a_0": 1, "a_1": 2}])}))
+    for rows in (spark.range(1).collect(), 
list(spark.range(1).toLocalIterator())):
+        assert rows[0].v.__fields__ == ["a", "a"]
+        assert tuple(rows[0].v) == (1, 2)
+    # Struct.fields=1, StructField: name=1, data_type=2 
(DataType.calendar_interval=17).
+    respond(message_field(21, message_field(1, message_field(1, "v") + 
message_field(
+        2, message_field(17, b"")))),
+        pa.table({"v": pa.array([(1, 2, 3)], pa.month_day_nano_interval())}))
+    with pytest.raises(PySparkTypeError) as e:
+        spark.range(1).collect()
+    assert e.value.getCondition() == 
"UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION"
+    assert e.value.getMessageParameters() == {"data_type": 
"month_day_nano_interval"}
+    assert len(list(spark.range(1).toLocalIterator())) == 1
+
 def test_tags_are_those_of_each_thread_as_pyspark(mock_connect):
     # As PySpark's: the tags of addTag are those of the current thread, of a 
set of getTags,
     # which the operations that the thread starts carry; a new thread starts 
without tags.


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to