This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new bc67aac [SPARK-60967][CONNECT][PYTHON] Deserialize collected values
of user-defined types, and name them in dtypes, as PySpark does
bc67aac is described below
commit bc67aac335bc448be138e648f83a7137383ceea5
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Fri Nov 6 17:44:13 2026 +0900
[SPARK-60967][CONNECT][PYTHON] Deserialize collected values of user-defined
types, and name them in dtypes, as PySpark does
### What changes were proposed in this pull request?
[SPARK-60967](https://issues.apache.org/jira/browse/SPARK-60967): a
collected value of a user-defined type is now the object that its UDT
deserializes, as PySpark's `ArrowTableToRowsConversion` makes it. A value of a
`VectorUDT` is a `DenseVector`, in arrays, maps and structs too.
- `collect`, `first`, `head`, `take`, `tail` and `toLocalIterator` read the
schema of the result that the server sends with the rows,
`ExecutePlanResponse.schema`. Unlike the Arrow schema of the rows, it has the
user-defined types. They read it through the new
`DataFrame::collect_with_schema`, `first_with_schema`, `take_with_schema` and
`LocalRowIterator::schema` of the Rust core.
- For a result with a user-defined type,
`pyspark.sql.types._udt_field_converters` gives a converter per field, which
the rows apply to their values when they are first read. Other results are
unchanged.
- `dtypes`, and `simpleString()` of an array, map, struct or field of a
user-defined type, name the type by the `simpleString()` of its Python class,
such as `vector` and `array<vector>`.
### Why are the changes needed?
With `df = spark.sql("SELECT array(1.5D, 2.0D) AS
a").select(array_to_vector("a").alias("vec"))` of `pyspark.ml.functions`:
| | PySpark 4.3.0 and now | Before |
| --- | --- | --- |
| `df.first().vec` | `DenseVector([1.5, 2.0])` | `Row(type=1, size=None,
indices=None, values=[1.5, 2.0])` |
| `df.dtypes` | `[('vec', 'vector')]` | `[('vec', 'udt')]` |
| `ArrayType(VectorUDT()).simpleString()` | `array<vector>` | `array<udt>` |
The field type of `df.schema` was already the `VectorUDT`.
### Does this PR introduce _any_ user-facing change?
Yes, collected values of user-defined types and their names are those of
PySpark 4.3.0.
### How was this patch tested?
- Against a local Spark 4.3.0 server, `dtypes`, `first`, `collect`, `take`,
`head`, `tail` and `toLocalIterator` (with and without prefetching) of vectors
give the same results as official PySpark 4.3.0. So do `NULL` vectors, vectors
in arrays, structs and maps, `asDict(recursive=True)`, `printSchema()` and
`schema.simpleString()`. Before this change, all but `printSchema()` differed.
- A new live test checks these. A new offline test checks the names of
nested user-defined types. Both also pass with official PySpark 4.3.0, and fail
without this change. A new offline test checks the converters.
- The lib tests and every integration test, e2e included, against the local
server (652 passed, and 123 of the transport crate), the build of all tests
with `wasm-udf`, the offline Python suites (880 passed, 9 skipped), and the
live drop-in and Arrow suites (82 passed) and live cases (10 passed). In the
changed lines, clippy reports only the `result_large_err` of the `py.detach`
closures, which every action of the Python `DataFrame` has.
`toPandas()` of a user-defined type, and of timestamps and maps, is
[SPARK-61000](https://issues.apache.org/jira/browse/SPARK-61000).
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #199 from HyukjinKwon/SPARK-60967-udt-values.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/dataframe.rs | 113 ++++++++++++++++++++++++----------
crates/pyspark-rs/src/row.rs | 35 ++++++++++-
crates/pyspark-rs/src/types.rs | 102 +++++++++++++++++++++++++-----
crates/spark-connect/src/dataframe.rs | 62 ++++++++++++++++++-
python/pyspark/sql/types.py | 47 ++++++++++++++
python/tests/test_dropin_e2e.py | 26 ++++++++
python/tests/test_dropin_offline.py | 54 ++++++++++++++++
7 files changed, 388 insertions(+), 51 deletions(-)
diff --git a/crates/pyspark-rs/src/dataframe.rs
b/crates/pyspark-rs/src/dataframe.rs
index d2a1e11..54264e9 100644
--- a/crates/pyspark-rs/src/dataframe.rs
+++ b/crates/pyspark-rs/src/dataframe.rs
@@ -10,7 +10,7 @@ use crate::column::PyColumn;
use crate::errors::ResultExt;
use crate::functions::to_column;
use crate::group::PyGroupedData;
-use crate::row::{value_to_py, PyRow};
+use crate::row::{value_to_py, FieldConverters, PyRow};
use crate::streaming::PyDataStreamWriter;
use crate::types::PyDataType;
@@ -1234,8 +1234,10 @@ impl PyDataFrame {
/// Collect all rows into memory.
fn collect(&self, py: Python<'_>) -> PyResult<Vec<PyRow>> {
// Release the GIL across the blocking RPC so other Python threads run.
- let rows = py.detach(|| self.dataframe.collect()).to_pyerr()?;
- Ok(rows.into_iter().map(PyRow::new).collect())
+ let (rows, schema) = py
+ .detach(|| self.dataframe.collect_with_schema())
+ .to_pyerr()?;
+ rows_to_py(py, rows, schema.as_ref())
}
/// Return an iterator over rows without materializing the entire result
in memory.
@@ -1404,8 +1406,10 @@ impl PyDataFrame {
/// Get the first row.
fn first(&self, py: Python<'_>) -> PyResult<Option<PyRow>> {
- let row = py.detach(|| self.dataframe.first()).to_pyerr()?;
- Ok(row.map(PyRow::new))
+ let (row, schema) = py
+ .detach(|| self.dataframe.first_with_schema())
+ .to_pyerr()?;
+ Ok(rows_to_py(py, row.into_iter().collect(), schema.as_ref())?.pop())
}
/// Alias for first.
@@ -1414,25 +1418,23 @@ impl PyDataFrame {
use pyo3::IntoPyObjectExt;
match n {
// head() -> the first Row (or None), matching reference pyspark.
- None => {
- let row = py.detach(|| self.dataframe.first()).to_pyerr()?;
- row.map(PyRow::new).into_py_any(py)
- }
+ None => self.first(py)?.into_py_any(py),
// head(n) -> a list of the first n Rows.
Some(k) => {
- let rows = py.detach(||
self.dataframe.limit(k).collect()).to_pyerr()?;
- rows.into_iter()
- .map(PyRow::new)
- .collect::<Vec<_>>()
- .into_py_any(py)
+ let (rows, schema) = py
+ .detach(|| self.dataframe.limit(k).collect_with_schema())
+ .to_pyerr()?;
+ rows_to_py(py, rows, schema.as_ref())?.into_py_any(py)
}
}
}
/// Get the first n rows.
fn take(&self, py: Python<'_>, num: usize) -> PyResult<Vec<PyRow>> {
- let rows = py.detach(|| self.dataframe.take(num)).to_pyerr()?;
- Ok(rows.into_iter().map(PyRow::new).collect())
+ let (rows, schema) = py
+ .detach(|| self.dataframe.take_with_schema(num))
+ .to_pyerr()?;
+ rows_to_py(py, rows, schema.as_ref())
}
/// Column names (a property, mirroring `DataFrame.columns`).
@@ -1443,10 +1445,10 @@ impl PyDataFrame {
/// Get the last `num` rows as a list of Rows (eager, like pyspark `tail`).
fn tail(&self, py: Python<'_>, num: i32) -> PyResult<Vec<PyRow>> {
- let rows = py
- .detach(|| self.dataframe.tail(num).collect())
+ let (rows, schema) = py
+ .detach(|| self.dataframe.tail(num).collect_with_schema())
.to_pyerr()?;
- Ok(rows.into_iter().map(PyRow::new).collect())
+ rows_to_py(py, rows, schema.as_ref())
}
/// Sample a fraction of rows. Mirrors
`DataFrame.sample(withReplacement=None,
@@ -1525,10 +1527,15 @@ impl PyDataFrame {
else {
unreachable!("schema() validates struct responses")
};
- Ok(fields
+ // As PySpark, a user-defined type is named by its simpleString(),
such as `vector`,
+ // which only its Python class knows.
+ fields
.into_iter()
- .map(|field| (field.name, field.data_type.simple_string()))
- .collect())
+ .map(|field| {
+ let name = crate::types::simple_string_with_udts(py,
&field.data_type)?;
+ Ok((field.name, name))
+ })
+ .collect()
}
/// Additional set operation: unionAll (alias for union).
@@ -2750,17 +2757,57 @@ fn build_side_effect_udf(
))
}
+/// The converters of the fields of a result of `schema` that deserialize its
values of
+/// user-defined types, as PySpark does when it collects them
+/// (`pyspark.sql.types._udt_field_converters`); None without a user-defined
type.
+fn udt_field_converters(
+ py: Python<'_>,
+ schema: Option<&spark_connect::types::DataType>,
+) -> PyResult<Option<FieldConverters>> {
+ let Some(schema) = schema.filter(|schema| crate::types::has_udt(schema))
else {
+ return Ok(None);
+ };
+ let schema = crate::types::data_type_to_py(py, schema)?;
+ let converters = py
+ .import("pyspark.sql.types")?
+ .call_method1("_udt_field_converters", (schema,))?;
+ if converters.is_none() {
+ return Ok(None);
+ }
+ Ok(Some(std::sync::Arc::new(converters.extract()?)))
+}
+
+/// The collected `rows` of a result of `schema`, whose values of user-defined
types are
+/// deserialized when first read.
+fn rows_to_py(
+ py: Python<'_>,
+ rows: Vec<spark_connect::row::Row>,
+ schema: Option<&spark_connect::types::DataType>,
+) -> PyResult<Vec<PyRow>> {
+ let converters = udt_field_converters(py, schema)?;
+ Ok(rows
+ .into_iter()
+ .map(|row| PyRow::with_converters(row, converters.clone()))
+ .collect())
+}
+
/// Python wrapper for LocalRowIterator yielding Row objects lazily.
///
/// Implements `__iter__` and `__next__` to make it a proper Python iterator.
#[pyclass(name = "LocalRowIterator", module = "pyspark.sql.dataframe")]
pub struct PyLocalRowIterator {
iterator: LocalRowIterator,
+ /// The converters of the fields of user-defined types of the rows, once
the schema of the
+ /// result is known: with the first row.
+ converters: Option<Option<FieldConverters>>,
}
impl PyLocalRowIterator {
pub fn new(iterator: LocalRowIterator) -> Self {
- PyLocalRowIterator { iterator }
+ PyLocalRowIterator {
+ iterator,
+ converters: None,
+ }
}
}
@@ -2774,15 +2821,19 @@ impl PyLocalRowIterator {
/// Fetch the next row from the iterator.
///
/// Releases the GIL during the blocking fetch to allow other threads to
run.
- fn __next__(&mut self, py: Python<'_>) -> Option<PyRow> {
- loop {
- // Release the GIL while fetching from the Rust iterator (which
may block on network I/O).
- let next_result = py.detach(|| self.iterator.next());
- match next_result {
- Some(Ok(row)) => return Some(PyRow::new(row)),
- Some(Err(_e)) => return None, // On error, stop iterating
- None => return None,
+ fn __next__(&mut self, py: Python<'_>) -> PyResult<Option<PyRow>> {
+ // Release the GIL while fetching from the Rust iterator (which may
block on network I/O).
+ match py.detach(|| self.iterator.next()) {
+ Some(Ok(row)) => {
+ if self.converters.is_none() {
+ let schema = self.iterator.schema();
+ self.converters = Some(udt_field_converters(py,
schema.as_ref())?);
+ }
+ let converters = self.converters.clone().flatten();
+ Ok(Some(PyRow::with_converters(row, converters)))
}
+ Some(Err(_e)) => Ok(None), // On error, stop iterating
+ None => Ok(None),
}
}
}
diff --git a/crates/pyspark-rs/src/row.rs b/crates/pyspark-rs/src/row.rs
index 2dc524e..66b2956 100644
--- a/crates/pyspark-rs/src/row.rs
+++ b/crates/pyspark-rs/src/row.rs
@@ -20,6 +20,11 @@ pub struct PyRow {
named: bool,
}
+/// The Python functions that turn the values of the fields of collected rows
whose types have
+/// user-defined types into what PySpark gives for them, by deserializing
those values; None
+/// for the other fields.
+pub(crate) type FieldConverters = std::sync::Arc<Vec<Option<Py<PyAny>>>>;
+
enum RowData {
/// A row that the server returned, whose Spark values become Python
values when first
/// read, and stay those objects, as a PySpark row is a tuple of them: so
the row equals
@@ -27,6 +32,8 @@ enum RowData {
Spark {
row: Row,
values: PyOnceLock<Py<PyTuple>>,
+ /// The converters of its fields, when one has a user-defined type.
+ converters: Option<FieldConverters>,
},
/// A row made in Python, which holds its Python values as they are, as
PySpark's does.
Python {
@@ -37,10 +44,17 @@ enum RowData {
impl PyRow {
pub fn new(row: Row) -> Self {
+ Self::with_converters(row, None)
+ }
+
+ /// A row that the server returned, whose values of user-defined types
`converters`
+ /// deserializes when they are first read.
+ pub(crate) fn with_converters(row: Row, converters:
Option<FieldConverters>) -> Self {
PyRow {
data: RowData::Spark {
row,
values: PyOnceLock::new(),
+ converters,
},
named: true,
}
@@ -67,12 +81,29 @@ impl PyRow {
/// The values, as the tuple that a PySpark row is.
fn values<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyTuple>> {
match &self.data {
- RowData::Spark { row, values } => {
+ RowData::Spark {
+ row,
+ values,
+ converters,
+ } => {
let values = values.get_or_try_init(py, || ->
PyResult<Py<PyTuple>> {
+ let convert = |i: usize| {
+ converters
+ .as_ref()
+ .and_then(|c| c.get(i))
+ .and_then(Option::as_ref)
+ };
let values = row
.values()
.iter()
- .map(|v| value_to_py(py, v))
+ .enumerate()
+ .map(|(i, v)| {
+ let value = value_to_py(py, v)?;
+ match convert(i) {
+ Some(converter) =>
converter.bind(py).call1((value,)),
+ None => Ok(value),
+ }
+ })
.collect::<PyResult<Vec<_>>>()?;
Ok(PyTuple::new(py, values)?.unbind())
})?;
diff --git a/crates/pyspark-rs/src/types.rs b/crates/pyspark-rs/src/types.rs
index 7e03242..0c44a6c 100644
--- a/crates/pyspark-rs/src/types.rs
+++ b/crates/pyspark-rs/src/types.rs
@@ -212,8 +212,17 @@ impl PyDataType {
Ok(format!("{}()", slf.get_type().name()?))
}
- fn simpleString(&self) -> String {
- self.inner.simple_string()
+ /// As pyspark's `simpleString()`. An array, map or struct of a
user-defined type names it
+ /// by the `simpleString()` of its Python class, such as `array<vector>`.
+ fn simpleString(&self, py: Python<'_>) -> PyResult<String> {
+ match &self.inner {
+ DataType::Array { .. } | DataType::Map { .. } | DataType::Struct {
.. }
+ if has_udt(&self.inner) =>
+ {
+ simple_string_with_udts(py, &self.inner)
+ }
+ _ => Ok(self.inner.simple_string()),
+ }
}
// Equality/hashing are defined once here and inherited by every concrete
type class
@@ -1168,8 +1177,11 @@ impl PyArrayType {
))
}
#[pyo3(name = "simpleString")]
- fn simple_string(&self) -> String {
- format!("array<{}>", self.element_type.simple_string())
+ fn simple_string(&self, py: Python<'_>) -> PyResult<String> {
+ Ok(format!(
+ "array<{}>",
+ simple_string_with_udts(py, &self.element_type)?
+ ))
}
#[pyo3(name = "typeName")]
fn type_name(&self) -> &'static str {
@@ -1310,12 +1322,12 @@ impl PyMapType {
))
}
#[pyo3(name = "simpleString")]
- fn simple_string(&self) -> String {
- format!(
+ fn simple_string(&self, py: Python<'_>) -> PyResult<String> {
+ Ok(format!(
"map<{},{}>",
- self.key_type.simple_string(),
- self.value_type.simple_string()
- )
+ simple_string_with_udts(py, &self.key_type)?,
+ simple_string_with_udts(py, &self.value_type)?
+ ))
}
#[pyo3(name = "typeName")]
fn type_name(&self) -> &'static str {
@@ -1362,8 +1374,12 @@ impl PyStructField {
// itself (a StructField is not a DataType, so these do not go through
// py_to_data_type). ---
#[pyo3(name = "simpleString")]
- fn __sf_simple_string(&self) -> String {
- self.field.simple_string()
+ fn __sf_simple_string(&self, py: Python<'_>) -> PyResult<String> {
+ Ok(format!(
+ "{}:{}",
+ self.field.name,
+ simple_string_with_udts(py, &self.field.data_type)?
+ ))
}
#[pyo3(name = "typeName")]
fn __sf_type_name(&self) -> PyResult<String> {
@@ -1745,11 +1761,13 @@ impl PyStructType {
}
#[pyo3(name = "simpleString")]
- fn simple_string(&self) -> String {
- DataType::Struct {
- fields: self.fields.clone(),
- }
- .simple_string()
+ fn simple_string(&self, py: Python<'_>) -> PyResult<String> {
+ simple_string_with_udts(
+ py,
+ &DataType::Struct {
+ fields: self.fields.clone(),
+ },
+ )
}
/// `StructType.typeName()` == "struct".
@@ -2593,6 +2611,58 @@ pub fn parse_datatype_json(py: Python<'_>, json: &str)
-> PyResult<Py<PyAny>> {
/// `StructType`/`IntegerType`/... with the correct MRO — the inverse of
/// [`py_to_data_type`]. Used by `df.schema`, `StructField.dataType`, and
`StructType`
/// field access so schema introspection yields the proper classes (not a bare
`DataType`).
+/// Whether `data_type` has a user-defined type.
+pub(crate) fn has_udt(data_type: &DataType) -> bool {
+ match data_type {
+ DataType::Udt { .. } => true,
+ DataType::Array { element_type, .. } => has_udt(element_type),
+ DataType::Map {
+ key_type,
+ value_type,
+ ..
+ } => has_udt(key_type) || has_udt(value_type),
+ DataType::Struct { fields } => fields.iter().any(|field|
has_udt(&field.data_type)),
+ _ => false,
+ }
+}
+
+/// The `simpleString()` of `data_type` as PySpark's: of the Python class of
each of its
+/// user-defined types, such as `vector` of a `VectorUDT`, which the Rust core
does not know.
+pub(crate) fn simple_string_with_udts(py: Python<'_>, data_type: &DataType) ->
PyResult<String> {
+ Ok(match data_type {
+ DataType::Udt { .. } => data_type_to_py(py, data_type)?
+ .bind(py)
+ .call_method0("simpleString")?
+ .extract()?,
+ DataType::Array { element_type, .. } => {
+ format!("array<{}>", simple_string_with_udts(py, element_type)?)
+ }
+ DataType::Map {
+ key_type,
+ value_type,
+ ..
+ } => format!(
+ "map<{},{}>",
+ simple_string_with_udts(py, key_type)?,
+ simple_string_with_udts(py, value_type)?
+ ),
+ DataType::Struct { fields } => {
+ let fields = fields
+ .iter()
+ .map(|field| {
+ Ok(format!(
+ "{}:{}",
+ field.name,
+ simple_string_with_udts(py, &field.data_type)?
+ ))
+ })
+ .collect::<PyResult<Vec<_>>>()?;
+ format!("struct<{}>", fields.join(","))
+ }
+ _ => data_type.simple_string(),
+ })
+}
+
pub(crate) fn data_type_to_py(py: Python<'_>, dt: &DataType) ->
PyResult<Py<PyAny>> {
macro_rules! obj {
($init:expr) => {
diff --git a/crates/spark-connect/src/dataframe.rs
b/crates/spark-connect/src/dataframe.rs
index 0013770..dc69cb9 100644
--- a/crates/spark-connect/src/dataframe.rs
+++ b/crates/spark-connect/src/dataframe.rs
@@ -51,6 +51,21 @@ pub struct LocalRowIterator {
source: RowSource,
/// Set once the batch source is exhausted or has errored.
done: bool,
+ /// The schema of the result that the server sends before the rows
+ /// (`ExecutePlanResponse.schema`), once received.
+ schema: SchemaSlot,
+}
+
+/// The schema of a result as the server sends it, shared with a prefetch task.
+type SchemaSlot = Arc<Mutex<Option<DataType>>>;
+
+/// Keeps the schema of `resp`, if it has one, in `slot`.
+fn keep_schema(resp: &mut proto::ExecutePlanResponse, slot: &SchemaSlot) ->
Result<()> {
+ if let Some(data_type) = resp.schema.take() {
+ let schema = DataType::from_proto(&data_type)?;
+ *slot.lock().unwrap_or_else(|poisoned| poisoned.into_inner()) =
Some(schema);
+ }
+ Ok(())
}
/// The source of successive Arrow batches for a [`LocalRowIterator`].
@@ -78,9 +93,10 @@ impl LocalRowIterator {
stream: ReattachableResponseStream,
prefetch_partitions: bool,
) -> Self {
+ let schema = SchemaSlot::default();
let source = if prefetch_partitions {
RowSource::Prefetch {
- rx: spawn_prefetch(session, stream),
+ rx: spawn_prefetch(session, stream, Arc::clone(&schema)),
}
} else {
RowSource::OnDemand {
@@ -94,9 +110,20 @@ impl LocalRowIterator {
current_rows: vec![].into_iter(),
source,
done: false,
+ schema,
}
}
+ /// The schema of the result, as the server sends it before the rows, once
received: so
+ /// after the first row. Unlike the Arrow schema of the rows, it has the
user-defined
+ /// types of the columns.
+ pub fn schema(&self) -> Option<DataType> {
+ self.schema
+ .lock()
+ .unwrap_or_else(|poisoned| poisoned.into_inner())
+ .clone()
+ }
+
/// Fetch the next batch of rows, or `None` once the source is exhausted.
///
/// A batch may legitimately decode to zero rows (e.g. a metrics-only
@@ -112,6 +139,9 @@ impl LocalRowIterator {
match block_on(stream.message()) {
Ok(Some(mut resp)) => {
capture_execution(&mut resp, execution_info, session);
+ if let Err(e) = keep_schema(&mut resp, &self.schema) {
+ return Some(Err(e));
+ }
if let
Some(proto::execute_plan_response::ResponseType::ArrowBatch(batch)) =
resp.response_type
{
@@ -173,6 +203,7 @@ impl Iterator for LocalRowIterator {
fn spawn_prefetch(
session: SparkSession,
mut stream: ReattachableResponseStream,
+ schema: SchemaSlot,
) -> tokio::sync::mpsc::Receiver<Result<Vec<Row>>> {
let (tx, rx) = tokio::sync::mpsc::channel::<Result<Vec<Row>>>(1);
get_runtime().spawn(async move {
@@ -181,6 +212,10 @@ fn spawn_prefetch(
match stream.message().await {
Ok(Some(mut resp)) => {
capture_execution(&mut resp, &mut execution_info,
&session);
+ if let Err(e) = keep_schema(&mut resp, &schema) {
+ let _ = tx.send(Err(e)).await;
+ break;
+ }
if let
Some(proto::execute_plan_response::ResponseType::ArrowBatch(batch)) =
resp.response_type
{
@@ -714,10 +749,18 @@ impl DataFrame {
/// Collect all rows into memory.
pub fn collect(&self) -> Result<Vec<Row>> {
+ self.collect_with_schema().map(|(rows, _)| rows)
+ }
+
+ /// Collect all rows into memory, with the schema of the result if the
server sends it
+ /// (`ExecutePlanResponse.schema`), as the reference client's `_to_table`
returns it.
+ /// Unlike the Arrow schema of the rows, it has the user-defined types of
the columns.
+ pub fn collect_with_schema(&self) -> Result<(Vec<Row>, Option<DataType>)> {
let request = self.build_execute_request()?;
let mut stream =
block_on(self.session.client().execute_plan_reattachable(request))?;
let mut rows = vec![];
+ let mut schema = None;
let mut info = ExecutionInfo::default();
loop {
let resp = block_on(stream.message())?;
@@ -725,6 +768,9 @@ impl DataFrame {
break;
};
capture_execution(&mut resp, &mut info, &self.session);
+ if let Some(data_type) = resp.schema.take() {
+ schema = Some(DataType::from_proto(&data_type)?);
+ }
if let
Some(proto::execute_plan_response::ResponseType::ArrowBatch(batch)) =
resp.response_type
{
@@ -734,7 +780,7 @@ impl DataFrame {
}
self.record_execution(info);
- Ok(rows)
+ Ok((rows, schema))
}
/// Return an iterator that lazily streams rows from the server.
@@ -914,6 +960,12 @@ impl DataFrame {
self.limit(1).collect().map(|rows| rows.into_iter().next())
}
+ /// The first row, with the schema of the result, as
[`DataFrame::collect_with_schema`].
+ pub fn first_with_schema(&self) -> Result<(Option<Row>, Option<DataType>)>
{
+ let (rows, schema) = self.limit(1).collect_with_schema()?;
+ Ok((rows.into_iter().next(), schema))
+ }
+
/// Alias for first().
pub fn head(&self) -> Result<Option<Row>> {
self.first()
@@ -924,6 +976,12 @@ impl DataFrame {
self.limit(n as i32).collect()
}
+ /// The first `n` rows, with the schema of the result, as
+ /// [`DataFrame::collect_with_schema`].
+ pub fn take_with_schema(&self, n: usize) -> Result<(Vec<Row>,
Option<DataType>)> {
+ self.limit(n as i32).collect_with_schema()
+ }
+
/// Check if the DataFrame is empty.
pub fn is_empty(&self) -> Result<bool> {
self.limit(1).count().map(|c| c == 0)
diff --git a/python/pyspark/sql/types.py b/python/pyspark/sql/types.py
index 174e04e..43abd15 100644
--- a/python/pyspark/sql/types.py
+++ b/python/pyspark/sql/types.py
@@ -392,3 +392,50 @@ def _create_row(fields, values):
row = Row(*values)
row.__fields__ = fields
return row
+
+
+def _udt_converter(dataType):
+ """The function that turns a collected value of ``dataType`` into what
PySpark gives for it,
+ deserializing the values of its user-defined types as PySpark's
+ ``ArrowTableToRowsConversion`` does, or None for a type without a
user-defined type."""
+ if isinstance(dataType, UserDefinedType):
+ inner = _udt_converter(dataType.sqlType())
+
+ def convert_udt(value):
+ if value is None:
+ return None
+ return dataType.deserialize(value if inner is None else
inner(value))
+
+ return convert_udt
+ if isinstance(dataType, ArrayType):
+ element = _udt_converter(dataType.elementType)
+ if element is None:
+ return None
+ return lambda value: None if value is None else [element(v) for v in
value]
+ if isinstance(dataType, MapType):
+ key, val = _udt_converter(dataType.keyType),
_udt_converter(dataType.valueType)
+ if key is None and val is None:
+ return None
+ key = key or (lambda v: v)
+ val = val or (lambda v: v)
+ return lambda value: None if value is None else {key(k): val(v) for k,
v in value.items()}
+ if isinstance(dataType, StructType):
+ fields = _udt_field_converters(dataType)
+ if fields is None:
+ return None
+ row = Row(*dataType.names)
+
+ def convert_struct(value):
+ if value is None:
+ return None
+ return row(*[v if f is None else f(v) for f, v in zip(fields,
value)])
+
+ return convert_struct
+ return None
+
+
+def _udt_field_converters(schema):
+ """The converters of the fields of ``schema`` (see ``_udt_converter``), or
None when no
+ field has a user-defined type. The client applies them to the rows that it
collects."""
+ converters = [_udt_converter(field.dataType) for field in schema.fields]
+ return None if all(c is None for c in converters) else converters
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index b694b18..570b9a4 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -887,6 +887,32 @@ def test_ml_vector_functions_as_pyspark(session):
assert tuple(row.first()) == ([1.5, 2.0], [1.5, 2.0])
+def
test_collected_values_of_user_defined_types_are_deserialized_as_pyspark(session):
+ # As official PySpark 4.3.0: a collected value of a UDT is its
deserialized object, and
+ # dtypes names the UDT by its simpleString(), in every way that rows are
collected.
+ from pyspark.ml.functions import array_to_vector
+ from pyspark.ml.linalg import DenseVector
+
+ df = session.sql(
+ "SELECT array(1.5D, 2.0D) AS a, 1 AS k UNION ALL SELECT array(3.0D), 2"
+ ).select(F.when(F.col("k") == 1, array_to_vector("a")).alias("vec"),
"k").orderBy("k")
+ vector = DenseVector([1.5, 2.0])
+ rows = [Row(vec=vector, k=1), Row(vec=None, k=2)]
+ assert df.dtypes == [("vec", "vector"), ("k", "int")]
+ assert df.collect() == rows and df.first() == rows[0] and df.head() ==
rows[0]
+ assert df.take(1) == rows[:1] and df.head(1) == rows[:1] and df.tail(1) ==
rows[1:]
+ assert list(df.toLocalIterator()) == rows and
list(df.toLocalIterator(True)) == rows
+ assert type(df.first().vec) is DenseVector
+ nested = df.limit(1).select(
+ F.array("vec", "vec").alias("arr"),
+ F.struct("vec", F.lit(1).alias("n")).alias("st"),
+ F.create_map(F.lit("x"), "vec").alias("m"),
+ )
+ assert nested.dtypes == [
+ ("arr", "array<vector>"), ("st", "struct<vec:vector,n:int>"), ("m",
"map<string,vector>")]
+ assert nested.first() == Row(arr=[vector, vector], st=Row(vec=vector,
n=1), m={"x": vector})
+
+
def test_create_dataframe_merges_the_types_of_every_row_as_pyspark(session):
import array
import decimal
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index 3320ef2..755c2fb 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -735,6 +735,60 @@ def
test_decimal_literal_is_of_the_default_decimal_type_as_pyspark(mock_connect)
assert decimal(wide[1][0]) == {1: [b"12345678901.25"], 2: [10], 3: [0]}
+def test_user_defined_types_are_named_by_their_classes_as_pyspark():
+ # As PySpark's simpleString(): a user-defined type in an array, map or
struct is named
+ # by its class, as VectorUDT's `vector`, and `udt` by default.
+ from pyspark.ml.linalg import VectorUDT
+
+ point = _DayPointUDT()
+ assert T.ArrayType(VectorUDT()).simpleString() == "array<vector>"
+ assert T.MapType(T.StringType(), point).simpleString() == "map<string,udt>"
+ struct = T.StructType([T.StructField("v", VectorUDT()), T.StructField("p",
point)])
+ assert struct.simpleString() == "struct<v:vector,p:udt>"
+ assert struct.fields[0].simpleString() == "v:vector"
+
+
+class _DayPointUDT(T.UserDefinedType):
+ @classmethod
+ def sqlType(cls):
+ return T.ArrayType(T.DoubleType())
+
+ @classmethod
+ def module(cls):
+ return __name__
+
+ def serialize(self, obj):
+ return obj
+
+ def deserialize(self, datum):
+ return tuple(datum)
+
+
+def test_collected_values_of_user_defined_types_are_deserialized_as_pyspark():
+ # The converters that the client applies to collected rows: the values of
user-defined
+ # types, in arrays, maps and structs too, are deserialized by their UDTs,
as PySpark's
+ # ArrowTableToRowsConversion does; fields of other types need none.
+ from pyspark.ml.linalg import DenseVector, VectorUDT
+ from pyspark.sql.types import _udt_field_converters
+
+ vector = Row(type=1, size=None, indices=None, values=[1.5, 2.0])
+ schema = T.StructType([
+ T.StructField("v", VectorUDT()),
+ T.StructField("k", T.IntegerType()),
+ T.StructField("arr", T.ArrayType(VectorUDT())),
+ T.StructField("m", T.MapType(T.StringType(), _DayPointUDT())),
+ T.StructField("st", T.StructType([T.StructField("v", VectorUDT())])),
+ ])
+ v, k, arr, m, st = _udt_field_converters(schema)
+ assert v(vector) == DenseVector([1.5, 2.0]) and v(None) is None
+ assert k is None
+ assert arr([vector, None]) == [DenseVector([1.5, 2.0]), None]
+ assert m({"x": [1.0, 2.0]}) == {"x": (1.0, 2.0)}
+ converted = st(Row(v=vector))
+ assert converted == Row(v=DenseVector([1.5, 2.0])) and converted.v ==
DenseVector([1.5, 2.0])
+ assert _udt_field_converters(T.StructType([T.StructField("k",
T.LongType())])) is None
+
+
def test_dropin_module_imports():
# The thin re-export shims import cleanly.
import pyspark.sql.dataframe # noqa: F401
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]