This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new bc67aac  [SPARK-60967][CONNECT][PYTHON] Deserialize collected values 
of user-defined types, and name them in dtypes, as PySpark does
bc67aac is described below

commit bc67aac335bc448be138e648f83a7137383ceea5
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Fri Nov 6 17:44:13 2026 +0900

    [SPARK-60967][CONNECT][PYTHON] Deserialize collected values of user-defined 
types, and name them in dtypes, as PySpark does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-60967](https://issues.apache.org/jira/browse/SPARK-60967): a 
collected value of a user-defined type is now the object that its UDT 
deserializes, as PySpark's `ArrowTableToRowsConversion` makes it. A value of a 
`VectorUDT` is a `DenseVector`, in arrays, maps and structs too.
    
    - `collect`, `first`, `head`, `take`, `tail` and `toLocalIterator` read the 
schema of the result that the server sends with the rows, 
`ExecutePlanResponse.schema`. Unlike the Arrow schema of the rows, it has the 
user-defined types. They read it through the new 
`DataFrame::collect_with_schema`, `first_with_schema`, `take_with_schema` and 
`LocalRowIterator::schema` of the Rust core.
    - For a result with a user-defined type, 
`pyspark.sql.types._udt_field_converters` gives a converter per field, which 
the rows apply to their values when they are first read. Other results are 
unchanged.
    - `dtypes`, and `simpleString()` of an array, map, struct or field of a 
user-defined type, name the type by the `simpleString()` of its Python class, 
such as `vector` and `array<vector>`.
    
    ### Why are the changes needed?
    
    With `df = spark.sql("SELECT array(1.5D, 2.0D) AS 
a").select(array_to_vector("a").alias("vec"))` of `pyspark.ml.functions`:
    
    | | PySpark 4.3.0 and now | Before |
    | --- | --- | --- |
    | `df.first().vec` | `DenseVector([1.5, 2.0])` | `Row(type=1, size=None, 
indices=None, values=[1.5, 2.0])` |
    | `df.dtypes` | `[('vec', 'vector')]` | `[('vec', 'udt')]` |
    | `ArrayType(VectorUDT()).simpleString()` | `array<vector>` | `array<udt>` |
    
    The field type of `df.schema` was already the `VectorUDT`.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes, collected values of user-defined types and their names are those of 
PySpark 4.3.0.
    
    ### How was this patch tested?
    
    - Against a local Spark 4.3.0 server, `dtypes`, `first`, `collect`, `take`, 
`head`, `tail` and `toLocalIterator` (with and without prefetching) of vectors 
give the same results as official PySpark 4.3.0. So do `NULL` vectors, vectors 
in arrays, structs and maps, `asDict(recursive=True)`, `printSchema()` and 
`schema.simpleString()`. Before this change, all but `printSchema()` differed.
    - A new live test checks these. A new offline test checks the names of 
nested user-defined types. Both also pass with official PySpark 4.3.0, and fail 
without this change. A new offline test checks the converters.
    - The lib tests and every integration test, e2e included, against the local 
server (652 passed, and 123 of the transport crate), the build of all tests 
with `wasm-udf`, the offline Python suites (880 passed, 9 skipped), and the 
live drop-in and Arrow suites (82 passed) and live cases (10 passed). In the 
changed lines, clippy reports only the `result_large_err` of the `py.detach` 
closures, which every action of the Python `DataFrame` has.
    
    `toPandas()` of a user-defined type, and of timestamps and maps, is 
[SPARK-61000](https://issues.apache.org/jira/browse/SPARK-61000).
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #199 from HyukjinKwon/SPARK-60967-udt-values.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/dataframe.rs    | 113 ++++++++++++++++++++++++----------
 crates/pyspark-rs/src/row.rs          |  35 ++++++++++-
 crates/pyspark-rs/src/types.rs        | 102 +++++++++++++++++++++++++-----
 crates/spark-connect/src/dataframe.rs |  62 ++++++++++++++++++-
 python/pyspark/sql/types.py           |  47 ++++++++++++++
 python/tests/test_dropin_e2e.py       |  26 ++++++++
 python/tests/test_dropin_offline.py   |  54 ++++++++++++++++
 7 files changed, 388 insertions(+), 51 deletions(-)

diff --git a/crates/pyspark-rs/src/dataframe.rs 
b/crates/pyspark-rs/src/dataframe.rs
index d2a1e11..54264e9 100644
--- a/crates/pyspark-rs/src/dataframe.rs
+++ b/crates/pyspark-rs/src/dataframe.rs
@@ -10,7 +10,7 @@ use crate::column::PyColumn;
 use crate::errors::ResultExt;
 use crate::functions::to_column;
 use crate::group::PyGroupedData;
-use crate::row::{value_to_py, PyRow};
+use crate::row::{value_to_py, FieldConverters, PyRow};
 use crate::streaming::PyDataStreamWriter;
 use crate::types::PyDataType;
 
@@ -1234,8 +1234,10 @@ impl PyDataFrame {
     /// Collect all rows into memory.
     fn collect(&self, py: Python<'_>) -> PyResult<Vec<PyRow>> {
         // Release the GIL across the blocking RPC so other Python threads run.
-        let rows = py.detach(|| self.dataframe.collect()).to_pyerr()?;
-        Ok(rows.into_iter().map(PyRow::new).collect())
+        let (rows, schema) = py
+            .detach(|| self.dataframe.collect_with_schema())
+            .to_pyerr()?;
+        rows_to_py(py, rows, schema.as_ref())
     }
 
     /// Return an iterator over rows without materializing the entire result 
in memory.
@@ -1404,8 +1406,10 @@ impl PyDataFrame {
 
     /// Get the first row.
     fn first(&self, py: Python<'_>) -> PyResult<Option<PyRow>> {
-        let row = py.detach(|| self.dataframe.first()).to_pyerr()?;
-        Ok(row.map(PyRow::new))
+        let (row, schema) = py
+            .detach(|| self.dataframe.first_with_schema())
+            .to_pyerr()?;
+        Ok(rows_to_py(py, row.into_iter().collect(), schema.as_ref())?.pop())
     }
 
     /// Alias for first.
@@ -1414,25 +1418,23 @@ impl PyDataFrame {
         use pyo3::IntoPyObjectExt;
         match n {
             // head() -> the first Row (or None), matching reference pyspark.
-            None => {
-                let row = py.detach(|| self.dataframe.first()).to_pyerr()?;
-                row.map(PyRow::new).into_py_any(py)
-            }
+            None => self.first(py)?.into_py_any(py),
             // head(n) -> a list of the first n Rows.
             Some(k) => {
-                let rows = py.detach(|| 
self.dataframe.limit(k).collect()).to_pyerr()?;
-                rows.into_iter()
-                    .map(PyRow::new)
-                    .collect::<Vec<_>>()
-                    .into_py_any(py)
+                let (rows, schema) = py
+                    .detach(|| self.dataframe.limit(k).collect_with_schema())
+                    .to_pyerr()?;
+                rows_to_py(py, rows, schema.as_ref())?.into_py_any(py)
             }
         }
     }
 
     /// Get the first n rows.
     fn take(&self, py: Python<'_>, num: usize) -> PyResult<Vec<PyRow>> {
-        let rows = py.detach(|| self.dataframe.take(num)).to_pyerr()?;
-        Ok(rows.into_iter().map(PyRow::new).collect())
+        let (rows, schema) = py
+            .detach(|| self.dataframe.take_with_schema(num))
+            .to_pyerr()?;
+        rows_to_py(py, rows, schema.as_ref())
     }
 
     /// Column names (a property, mirroring `DataFrame.columns`).
@@ -1443,10 +1445,10 @@ impl PyDataFrame {
 
     /// Get the last `num` rows as a list of Rows (eager, like pyspark `tail`).
     fn tail(&self, py: Python<'_>, num: i32) -> PyResult<Vec<PyRow>> {
-        let rows = py
-            .detach(|| self.dataframe.tail(num).collect())
+        let (rows, schema) = py
+            .detach(|| self.dataframe.tail(num).collect_with_schema())
             .to_pyerr()?;
-        Ok(rows.into_iter().map(PyRow::new).collect())
+        rows_to_py(py, rows, schema.as_ref())
     }
 
     /// Sample a fraction of rows. Mirrors 
`DataFrame.sample(withReplacement=None,
@@ -1525,10 +1527,15 @@ impl PyDataFrame {
         else {
             unreachable!("schema() validates struct responses")
         };
-        Ok(fields
+        // As PySpark, a user-defined type is named by its simpleString(), 
such as `vector`,
+        // which only its Python class knows.
+        fields
             .into_iter()
-            .map(|field| (field.name, field.data_type.simple_string()))
-            .collect())
+            .map(|field| {
+                let name = crate::types::simple_string_with_udts(py, 
&field.data_type)?;
+                Ok((field.name, name))
+            })
+            .collect()
     }
 
     /// Additional set operation: unionAll (alias for union).
@@ -2750,17 +2757,57 @@ fn build_side_effect_udf(
     ))
 }
 
+/// The converters of the fields of a result of `schema` that deserialize its 
values of
+/// user-defined types, as PySpark does when it collects them
+/// (`pyspark.sql.types._udt_field_converters`); None without a user-defined 
type.
+fn udt_field_converters(
+    py: Python<'_>,
+    schema: Option<&spark_connect::types::DataType>,
+) -> PyResult<Option<FieldConverters>> {
+    let Some(schema) = schema.filter(|schema| crate::types::has_udt(schema)) 
else {
+        return Ok(None);
+    };
+    let schema = crate::types::data_type_to_py(py, schema)?;
+    let converters = py
+        .import("pyspark.sql.types")?
+        .call_method1("_udt_field_converters", (schema,))?;
+    if converters.is_none() {
+        return Ok(None);
+    }
+    Ok(Some(std::sync::Arc::new(converters.extract()?)))
+}
+
+/// The collected `rows` of a result of `schema`, whose values of user-defined 
types are
+/// deserialized when first read.
+fn rows_to_py(
+    py: Python<'_>,
+    rows: Vec<spark_connect::row::Row>,
+    schema: Option<&spark_connect::types::DataType>,
+) -> PyResult<Vec<PyRow>> {
+    let converters = udt_field_converters(py, schema)?;
+    Ok(rows
+        .into_iter()
+        .map(|row| PyRow::with_converters(row, converters.clone()))
+        .collect())
+}
+
 /// Python wrapper for LocalRowIterator yielding Row objects lazily.
 ///
 /// Implements `__iter__` and `__next__` to make it a proper Python iterator.
 #[pyclass(name = "LocalRowIterator", module = "pyspark.sql.dataframe")]
 pub struct PyLocalRowIterator {
     iterator: LocalRowIterator,
+    /// The converters of the fields of user-defined types of the rows, once 
the schema of the
+    /// result is known: with the first row.
+    converters: Option<Option<FieldConverters>>,
 }
 
 impl PyLocalRowIterator {
     pub fn new(iterator: LocalRowIterator) -> Self {
-        PyLocalRowIterator { iterator }
+        PyLocalRowIterator {
+            iterator,
+            converters: None,
+        }
     }
 }
 
@@ -2774,15 +2821,19 @@ impl PyLocalRowIterator {
     /// Fetch the next row from the iterator.
     ///
     /// Releases the GIL during the blocking fetch to allow other threads to 
run.
-    fn __next__(&mut self, py: Python<'_>) -> Option<PyRow> {
-        loop {
-            // Release the GIL while fetching from the Rust iterator (which 
may block on network I/O).
-            let next_result = py.detach(|| self.iterator.next());
-            match next_result {
-                Some(Ok(row)) => return Some(PyRow::new(row)),
-                Some(Err(_e)) => return None, // On error, stop iterating
-                None => return None,
+    fn __next__(&mut self, py: Python<'_>) -> PyResult<Option<PyRow>> {
+        // Release the GIL while fetching from the Rust iterator (which may 
block on network I/O).
+        match py.detach(|| self.iterator.next()) {
+            Some(Ok(row)) => {
+                if self.converters.is_none() {
+                    let schema = self.iterator.schema();
+                    self.converters = Some(udt_field_converters(py, 
schema.as_ref())?);
+                }
+                let converters = self.converters.clone().flatten();
+                Ok(Some(PyRow::with_converters(row, converters)))
             }
+            Some(Err(_e)) => Ok(None), // On error, stop iterating
+            None => Ok(None),
         }
     }
 }
diff --git a/crates/pyspark-rs/src/row.rs b/crates/pyspark-rs/src/row.rs
index 2dc524e..66b2956 100644
--- a/crates/pyspark-rs/src/row.rs
+++ b/crates/pyspark-rs/src/row.rs
@@ -20,6 +20,11 @@ pub struct PyRow {
     named: bool,
 }
 
+/// The Python functions that turn the values of the fields of collected rows 
whose types have
+/// user-defined types into what PySpark gives for them, by deserializing 
those values; None
+/// for the other fields.
+pub(crate) type FieldConverters = std::sync::Arc<Vec<Option<Py<PyAny>>>>;
+
 enum RowData {
     /// A row that the server returned, whose Spark values become Python 
values when first
     /// read, and stay those objects, as a PySpark row is a tuple of them: so 
the row equals
@@ -27,6 +32,8 @@ enum RowData {
     Spark {
         row: Row,
         values: PyOnceLock<Py<PyTuple>>,
+        /// The converters of its fields, when one has a user-defined type.
+        converters: Option<FieldConverters>,
     },
     /// A row made in Python, which holds its Python values as they are, as 
PySpark's does.
     Python {
@@ -37,10 +44,17 @@ enum RowData {
 
 impl PyRow {
     pub fn new(row: Row) -> Self {
+        Self::with_converters(row, None)
+    }
+
+    /// A row that the server returned, whose values of user-defined types 
`converters`
+    /// deserializes when they are first read.
+    pub(crate) fn with_converters(row: Row, converters: 
Option<FieldConverters>) -> Self {
         PyRow {
             data: RowData::Spark {
                 row,
                 values: PyOnceLock::new(),
+                converters,
             },
             named: true,
         }
@@ -67,12 +81,29 @@ impl PyRow {
     /// The values, as the tuple that a PySpark row is.
     fn values<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyTuple>> {
         match &self.data {
-            RowData::Spark { row, values } => {
+            RowData::Spark {
+                row,
+                values,
+                converters,
+            } => {
                 let values = values.get_or_try_init(py, || -> 
PyResult<Py<PyTuple>> {
+                    let convert = |i: usize| {
+                        converters
+                            .as_ref()
+                            .and_then(|c| c.get(i))
+                            .and_then(Option::as_ref)
+                    };
                     let values = row
                         .values()
                         .iter()
-                        .map(|v| value_to_py(py, v))
+                        .enumerate()
+                        .map(|(i, v)| {
+                            let value = value_to_py(py, v)?;
+                            match convert(i) {
+                                Some(converter) => 
converter.bind(py).call1((value,)),
+                                None => Ok(value),
+                            }
+                        })
                         .collect::<PyResult<Vec<_>>>()?;
                     Ok(PyTuple::new(py, values)?.unbind())
                 })?;
diff --git a/crates/pyspark-rs/src/types.rs b/crates/pyspark-rs/src/types.rs
index 7e03242..0c44a6c 100644
--- a/crates/pyspark-rs/src/types.rs
+++ b/crates/pyspark-rs/src/types.rs
@@ -212,8 +212,17 @@ impl PyDataType {
         Ok(format!("{}()", slf.get_type().name()?))
     }
 
-    fn simpleString(&self) -> String {
-        self.inner.simple_string()
+    /// As pyspark's `simpleString()`. An array, map or struct of a 
user-defined type names it
+    /// by the `simpleString()` of its Python class, such as `array<vector>`.
+    fn simpleString(&self, py: Python<'_>) -> PyResult<String> {
+        match &self.inner {
+            DataType::Array { .. } | DataType::Map { .. } | DataType::Struct { 
.. }
+                if has_udt(&self.inner) =>
+            {
+                simple_string_with_udts(py, &self.inner)
+            }
+            _ => Ok(self.inner.simple_string()),
+        }
     }
 
     // Equality/hashing are defined once here and inherited by every concrete 
type class
@@ -1168,8 +1177,11 @@ impl PyArrayType {
         ))
     }
     #[pyo3(name = "simpleString")]
-    fn simple_string(&self) -> String {
-        format!("array<{}>", self.element_type.simple_string())
+    fn simple_string(&self, py: Python<'_>) -> PyResult<String> {
+        Ok(format!(
+            "array<{}>",
+            simple_string_with_udts(py, &self.element_type)?
+        ))
     }
     #[pyo3(name = "typeName")]
     fn type_name(&self) -> &'static str {
@@ -1310,12 +1322,12 @@ impl PyMapType {
         ))
     }
     #[pyo3(name = "simpleString")]
-    fn simple_string(&self) -> String {
-        format!(
+    fn simple_string(&self, py: Python<'_>) -> PyResult<String> {
+        Ok(format!(
             "map<{},{}>",
-            self.key_type.simple_string(),
-            self.value_type.simple_string()
-        )
+            simple_string_with_udts(py, &self.key_type)?,
+            simple_string_with_udts(py, &self.value_type)?
+        ))
     }
     #[pyo3(name = "typeName")]
     fn type_name(&self) -> &'static str {
@@ -1362,8 +1374,12 @@ impl PyStructField {
     // itself (a StructField is not a DataType, so these do not go through
     // py_to_data_type). ---
     #[pyo3(name = "simpleString")]
-    fn __sf_simple_string(&self) -> String {
-        self.field.simple_string()
+    fn __sf_simple_string(&self, py: Python<'_>) -> PyResult<String> {
+        Ok(format!(
+            "{}:{}",
+            self.field.name,
+            simple_string_with_udts(py, &self.field.data_type)?
+        ))
     }
     #[pyo3(name = "typeName")]
     fn __sf_type_name(&self) -> PyResult<String> {
@@ -1745,11 +1761,13 @@ impl PyStructType {
     }
 
     #[pyo3(name = "simpleString")]
-    fn simple_string(&self) -> String {
-        DataType::Struct {
-            fields: self.fields.clone(),
-        }
-        .simple_string()
+    fn simple_string(&self, py: Python<'_>) -> PyResult<String> {
+        simple_string_with_udts(
+            py,
+            &DataType::Struct {
+                fields: self.fields.clone(),
+            },
+        )
     }
 
     /// `StructType.typeName()` == "struct".
@@ -2593,6 +2611,58 @@ pub fn parse_datatype_json(py: Python<'_>, json: &str) 
-> PyResult<Py<PyAny>> {
 /// `StructType`/`IntegerType`/... with the correct MRO — the inverse of
 /// [`py_to_data_type`]. Used by `df.schema`, `StructField.dataType`, and 
`StructType`
 /// field access so schema introspection yields the proper classes (not a bare 
`DataType`).
+/// Whether `data_type` has a user-defined type.
+pub(crate) fn has_udt(data_type: &DataType) -> bool {
+    match data_type {
+        DataType::Udt { .. } => true,
+        DataType::Array { element_type, .. } => has_udt(element_type),
+        DataType::Map {
+            key_type,
+            value_type,
+            ..
+        } => has_udt(key_type) || has_udt(value_type),
+        DataType::Struct { fields } => fields.iter().any(|field| 
has_udt(&field.data_type)),
+        _ => false,
+    }
+}
+
+/// The `simpleString()` of `data_type` as PySpark's: of the Python class of 
each of its
+/// user-defined types, such as `vector` of a `VectorUDT`, which the Rust core 
does not know.
+pub(crate) fn simple_string_with_udts(py: Python<'_>, data_type: &DataType) -> 
PyResult<String> {
+    Ok(match data_type {
+        DataType::Udt { .. } => data_type_to_py(py, data_type)?
+            .bind(py)
+            .call_method0("simpleString")?
+            .extract()?,
+        DataType::Array { element_type, .. } => {
+            format!("array<{}>", simple_string_with_udts(py, element_type)?)
+        }
+        DataType::Map {
+            key_type,
+            value_type,
+            ..
+        } => format!(
+            "map<{},{}>",
+            simple_string_with_udts(py, key_type)?,
+            simple_string_with_udts(py, value_type)?
+        ),
+        DataType::Struct { fields } => {
+            let fields = fields
+                .iter()
+                .map(|field| {
+                    Ok(format!(
+                        "{}:{}",
+                        field.name,
+                        simple_string_with_udts(py, &field.data_type)?
+                    ))
+                })
+                .collect::<PyResult<Vec<_>>>()?;
+            format!("struct<{}>", fields.join(","))
+        }
+        _ => data_type.simple_string(),
+    })
+}
+
 pub(crate) fn data_type_to_py(py: Python<'_>, dt: &DataType) -> 
PyResult<Py<PyAny>> {
     macro_rules! obj {
         ($init:expr) => {
diff --git a/crates/spark-connect/src/dataframe.rs 
b/crates/spark-connect/src/dataframe.rs
index 0013770..dc69cb9 100644
--- a/crates/spark-connect/src/dataframe.rs
+++ b/crates/spark-connect/src/dataframe.rs
@@ -51,6 +51,21 @@ pub struct LocalRowIterator {
     source: RowSource,
     /// Set once the batch source is exhausted or has errored.
     done: bool,
+    /// The schema of the result that the server sends before the rows
+    /// (`ExecutePlanResponse.schema`), once received.
+    schema: SchemaSlot,
+}
+
+/// The schema of a result as the server sends it, shared with a prefetch task.
+type SchemaSlot = Arc<Mutex<Option<DataType>>>;
+
+/// Keeps the schema of `resp`, if it has one, in `slot`.
+fn keep_schema(resp: &mut proto::ExecutePlanResponse, slot: &SchemaSlot) -> 
Result<()> {
+    if let Some(data_type) = resp.schema.take() {
+        let schema = DataType::from_proto(&data_type)?;
+        *slot.lock().unwrap_or_else(|poisoned| poisoned.into_inner()) = 
Some(schema);
+    }
+    Ok(())
 }
 
 /// The source of successive Arrow batches for a [`LocalRowIterator`].
@@ -78,9 +93,10 @@ impl LocalRowIterator {
         stream: ReattachableResponseStream,
         prefetch_partitions: bool,
     ) -> Self {
+        let schema = SchemaSlot::default();
         let source = if prefetch_partitions {
             RowSource::Prefetch {
-                rx: spawn_prefetch(session, stream),
+                rx: spawn_prefetch(session, stream, Arc::clone(&schema)),
             }
         } else {
             RowSource::OnDemand {
@@ -94,9 +110,20 @@ impl LocalRowIterator {
             current_rows: vec![].into_iter(),
             source,
             done: false,
+            schema,
         }
     }
 
+    /// The schema of the result, as the server sends it before the rows, once 
received: so
+    /// after the first row. Unlike the Arrow schema of the rows, it has the 
user-defined
+    /// types of the columns.
+    pub fn schema(&self) -> Option<DataType> {
+        self.schema
+            .lock()
+            .unwrap_or_else(|poisoned| poisoned.into_inner())
+            .clone()
+    }
+
     /// Fetch the next batch of rows, or `None` once the source is exhausted.
     ///
     /// A batch may legitimately decode to zero rows (e.g. a metrics-only
@@ -112,6 +139,9 @@ impl LocalRowIterator {
                 match block_on(stream.message()) {
                     Ok(Some(mut resp)) => {
                         capture_execution(&mut resp, execution_info, session);
+                        if let Err(e) = keep_schema(&mut resp, &self.schema) {
+                            return Some(Err(e));
+                        }
                         if let 
Some(proto::execute_plan_response::ResponseType::ArrowBatch(batch)) =
                             resp.response_type
                         {
@@ -173,6 +203,7 @@ impl Iterator for LocalRowIterator {
 fn spawn_prefetch(
     session: SparkSession,
     mut stream: ReattachableResponseStream,
+    schema: SchemaSlot,
 ) -> tokio::sync::mpsc::Receiver<Result<Vec<Row>>> {
     let (tx, rx) = tokio::sync::mpsc::channel::<Result<Vec<Row>>>(1);
     get_runtime().spawn(async move {
@@ -181,6 +212,10 @@ fn spawn_prefetch(
             match stream.message().await {
                 Ok(Some(mut resp)) => {
                     capture_execution(&mut resp, &mut execution_info, 
&session);
+                    if let Err(e) = keep_schema(&mut resp, &schema) {
+                        let _ = tx.send(Err(e)).await;
+                        break;
+                    }
                     if let 
Some(proto::execute_plan_response::ResponseType::ArrowBatch(batch)) =
                         resp.response_type
                     {
@@ -714,10 +749,18 @@ impl DataFrame {
 
     /// Collect all rows into memory.
     pub fn collect(&self) -> Result<Vec<Row>> {
+        self.collect_with_schema().map(|(rows, _)| rows)
+    }
+
+    /// Collect all rows into memory, with the schema of the result if the 
server sends it
+    /// (`ExecutePlanResponse.schema`), as the reference client's `_to_table` 
returns it.
+    /// Unlike the Arrow schema of the rows, it has the user-defined types of 
the columns.
+    pub fn collect_with_schema(&self) -> Result<(Vec<Row>, Option<DataType>)> {
         let request = self.build_execute_request()?;
         let mut stream = 
block_on(self.session.client().execute_plan_reattachable(request))?;
 
         let mut rows = vec![];
+        let mut schema = None;
         let mut info = ExecutionInfo::default();
         loop {
             let resp = block_on(stream.message())?;
@@ -725,6 +768,9 @@ impl DataFrame {
                 break;
             };
             capture_execution(&mut resp, &mut info, &self.session);
+            if let Some(data_type) = resp.schema.take() {
+                schema = Some(DataType::from_proto(&data_type)?);
+            }
             if let 
Some(proto::execute_plan_response::ResponseType::ArrowBatch(batch)) =
                 resp.response_type
             {
@@ -734,7 +780,7 @@ impl DataFrame {
         }
         self.record_execution(info);
 
-        Ok(rows)
+        Ok((rows, schema))
     }
 
     /// Return an iterator that lazily streams rows from the server.
@@ -914,6 +960,12 @@ impl DataFrame {
         self.limit(1).collect().map(|rows| rows.into_iter().next())
     }
 
+    /// The first row, with the schema of the result, as 
[`DataFrame::collect_with_schema`].
+    pub fn first_with_schema(&self) -> Result<(Option<Row>, Option<DataType>)> 
{
+        let (rows, schema) = self.limit(1).collect_with_schema()?;
+        Ok((rows.into_iter().next(), schema))
+    }
+
     /// Alias for first().
     pub fn head(&self) -> Result<Option<Row>> {
         self.first()
@@ -924,6 +976,12 @@ impl DataFrame {
         self.limit(n as i32).collect()
     }
 
+    /// The first `n` rows, with the schema of the result, as
+    /// [`DataFrame::collect_with_schema`].
+    pub fn take_with_schema(&self, n: usize) -> Result<(Vec<Row>, 
Option<DataType>)> {
+        self.limit(n as i32).collect_with_schema()
+    }
+
     /// Check if the DataFrame is empty.
     pub fn is_empty(&self) -> Result<bool> {
         self.limit(1).count().map(|c| c == 0)
diff --git a/python/pyspark/sql/types.py b/python/pyspark/sql/types.py
index 174e04e..43abd15 100644
--- a/python/pyspark/sql/types.py
+++ b/python/pyspark/sql/types.py
@@ -392,3 +392,50 @@ def _create_row(fields, values):
     row = Row(*values)
     row.__fields__ = fields
     return row
+
+
+def _udt_converter(dataType):
+    """The function that turns a collected value of ``dataType`` into what 
PySpark gives for it,
+    deserializing the values of its user-defined types as PySpark's
+    ``ArrowTableToRowsConversion`` does, or None for a type without a 
user-defined type."""
+    if isinstance(dataType, UserDefinedType):
+        inner = _udt_converter(dataType.sqlType())
+
+        def convert_udt(value):
+            if value is None:
+                return None
+            return dataType.deserialize(value if inner is None else 
inner(value))
+
+        return convert_udt
+    if isinstance(dataType, ArrayType):
+        element = _udt_converter(dataType.elementType)
+        if element is None:
+            return None
+        return lambda value: None if value is None else [element(v) for v in 
value]
+    if isinstance(dataType, MapType):
+        key, val = _udt_converter(dataType.keyType), 
_udt_converter(dataType.valueType)
+        if key is None and val is None:
+            return None
+        key = key or (lambda v: v)
+        val = val or (lambda v: v)
+        return lambda value: None if value is None else {key(k): val(v) for k, 
v in value.items()}
+    if isinstance(dataType, StructType):
+        fields = _udt_field_converters(dataType)
+        if fields is None:
+            return None
+        row = Row(*dataType.names)
+
+        def convert_struct(value):
+            if value is None:
+                return None
+            return row(*[v if f is None else f(v) for f, v in zip(fields, 
value)])
+
+        return convert_struct
+    return None
+
+
+def _udt_field_converters(schema):
+    """The converters of the fields of ``schema`` (see ``_udt_converter``), or 
None when no
+    field has a user-defined type. The client applies them to the rows that it 
collects."""
+    converters = [_udt_converter(field.dataType) for field in schema.fields]
+    return None if all(c is None for c in converters) else converters
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index b694b18..570b9a4 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -887,6 +887,32 @@ def test_ml_vector_functions_as_pyspark(session):
     assert tuple(row.first()) == ([1.5, 2.0], [1.5, 2.0])
 
 
+def 
test_collected_values_of_user_defined_types_are_deserialized_as_pyspark(session):
+    # As official PySpark 4.3.0: a collected value of a UDT is its 
deserialized object, and
+    # dtypes names the UDT by its simpleString(), in every way that rows are 
collected.
+    from pyspark.ml.functions import array_to_vector
+    from pyspark.ml.linalg import DenseVector
+
+    df = session.sql(
+        "SELECT array(1.5D, 2.0D) AS a, 1 AS k UNION ALL SELECT array(3.0D), 2"
+    ).select(F.when(F.col("k") == 1, array_to_vector("a")).alias("vec"), 
"k").orderBy("k")
+    vector = DenseVector([1.5, 2.0])
+    rows = [Row(vec=vector, k=1), Row(vec=None, k=2)]
+    assert df.dtypes == [("vec", "vector"), ("k", "int")]
+    assert df.collect() == rows and df.first() == rows[0] and df.head() == 
rows[0]
+    assert df.take(1) == rows[:1] and df.head(1) == rows[:1] and df.tail(1) == 
rows[1:]
+    assert list(df.toLocalIterator()) == rows and 
list(df.toLocalIterator(True)) == rows
+    assert type(df.first().vec) is DenseVector
+    nested = df.limit(1).select(
+        F.array("vec", "vec").alias("arr"),
+        F.struct("vec", F.lit(1).alias("n")).alias("st"),
+        F.create_map(F.lit("x"), "vec").alias("m"),
+    )
+    assert nested.dtypes == [
+        ("arr", "array<vector>"), ("st", "struct<vec:vector,n:int>"), ("m", 
"map<string,vector>")]
+    assert nested.first() == Row(arr=[vector, vector], st=Row(vec=vector, 
n=1), m={"x": vector})
+
+
 def test_create_dataframe_merges_the_types_of_every_row_as_pyspark(session):
     import array
     import decimal
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index 3320ef2..755c2fb 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -735,6 +735,60 @@ def 
test_decimal_literal_is_of_the_default_decimal_type_as_pyspark(mock_connect)
     assert decimal(wide[1][0]) == {1: [b"12345678901.25"], 2: [10], 3: [0]}
 
 
+def test_user_defined_types_are_named_by_their_classes_as_pyspark():
+    # As PySpark's simpleString(): a user-defined type in an array, map or 
struct is named
+    # by its class, as VectorUDT's `vector`, and `udt` by default.
+    from pyspark.ml.linalg import VectorUDT
+
+    point = _DayPointUDT()
+    assert T.ArrayType(VectorUDT()).simpleString() == "array<vector>"
+    assert T.MapType(T.StringType(), point).simpleString() == "map<string,udt>"
+    struct = T.StructType([T.StructField("v", VectorUDT()), T.StructField("p", 
point)])
+    assert struct.simpleString() == "struct<v:vector,p:udt>"
+    assert struct.fields[0].simpleString() == "v:vector"
+
+
+class _DayPointUDT(T.UserDefinedType):
+    @classmethod
+    def sqlType(cls):
+        return T.ArrayType(T.DoubleType())
+
+    @classmethod
+    def module(cls):
+        return __name__
+
+    def serialize(self, obj):
+        return obj
+
+    def deserialize(self, datum):
+        return tuple(datum)
+
+
+def test_collected_values_of_user_defined_types_are_deserialized_as_pyspark():
+    # The converters that the client applies to collected rows: the values of 
user-defined
+    # types, in arrays, maps and structs too, are deserialized by their UDTs, 
as PySpark's
+    # ArrowTableToRowsConversion does; fields of other types need none.
+    from pyspark.ml.linalg import DenseVector, VectorUDT
+    from pyspark.sql.types import _udt_field_converters
+
+    vector = Row(type=1, size=None, indices=None, values=[1.5, 2.0])
+    schema = T.StructType([
+        T.StructField("v", VectorUDT()),
+        T.StructField("k", T.IntegerType()),
+        T.StructField("arr", T.ArrayType(VectorUDT())),
+        T.StructField("m", T.MapType(T.StringType(), _DayPointUDT())),
+        T.StructField("st", T.StructType([T.StructField("v", VectorUDT())])),
+    ])
+    v, k, arr, m, st = _udt_field_converters(schema)
+    assert v(vector) == DenseVector([1.5, 2.0]) and v(None) is None
+    assert k is None
+    assert arr([vector, None]) == [DenseVector([1.5, 2.0]), None]
+    assert m({"x": [1.0, 2.0]}) == {"x": (1.0, 2.0)}
+    converted = st(Row(v=vector))
+    assert converted == Row(v=DenseVector([1.5, 2.0])) and converted.v == 
DenseVector([1.5, 2.0])
+    assert _udt_field_converters(T.StructType([T.StructField("k", 
T.LongType())])) is None
+
+
 def test_dropin_module_imports():
     # The thin re-export shims import cleanly.
     import pyspark.sql.dataframe  # noqa: F401


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to