This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new e8f889d [SPARK-61108][CONNECT][PYTHON] Name the nested fields of
collected rows by the schema, and raise for calendar intervals in collect, in
the Python client as PySpark does
e8f889d is described below
commit e8f889d2516d1cb30b9cda37649a11466369179c
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sat Nov 7 18:53:27 2026 +0900
[SPARK-61108][CONNECT][PYTHON] Name the nested fields of collected rows by
the schema, and raise for calendar intervals in collect, in the Python client
as PySpark does
### What changes were proposed in this pull request?
[SPARK-61108](https://issues.apache.org/jira/browse/SPARK-61108): collected
rows now name the fields of nested structs by the schema, and `collect` raises
for calendar intervals, as PySpark Connect does.
As PySpark Connect's `ArrowTableToRowsConversion`, which builds rows by the
schema of the result:
- The core decodes the Arrow data of a result by the schema that the server
sends before it (`decode_arrow_batch_of`, `ArrowValueLayout::of` in
`crates/spark-connect/src/dataframe.rs`).
- So the fields of nested structs, also in arrays, maps and user-defined
types, have the names of the schema. These keep the duplicate names that the
server's Arrow data numbers as `a_0`, `a_1`.
- This holds for `collect`, `first`, `head`, `take`, `tail` and
`toLocalIterator`.
As PySpark Connect's `collect`, which checks the Arrow schema of the result
with `from_arrow_schema`, `collect`, `first`, `head`, `take` and `tail` raise
`UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION` for a calendar interval:
"month_day_nano_interval is not supported in conversion to Arrow". Its
`toLocalIterator` does not check, and neither does the client's.
### Why are the changes needed?
The client named nested fields by the Arrow data. For example, `SELECT
named_struct('a', 1, 'a', 2) AS v` collected `Row(v=Row(a_0=1, a_1=2))` instead
of `Row(v=Row(a=1, a=2))`, and `v.a` raised `ATTRIBUTE_NOT_SUPPORTED`. It also
collected a calendar interval as a string such as "14 months 25 days
18367500000000 ns", where PySpark raises.
### Does this PR introduce _any_ user-facing change?
Yes. Collected rows have the field names of PySpark 4.3.0, and collecting a
calendar interval raises its error.
### How was this patch tested?
- A new live test checks the names, the values and `asDict` of nested
structs of duplicate names, also from `toLocalIterator`, and the error of a
calendar interval. It passes with official PySpark 4.3.0 and fails without this
change.
- A new offline test checks the names of rows decoded by the schema of the
response, against a mock Connect server, and that `collect` raises for a
calendar interval where `toLocalIterator` does not.
- A new core test checks the names of nested structs, also in arrays,
decoded with a schema, and without one or of another shape.
- On a local server, 156 cases of `collect`, `first`, `head`, `take`,
`tail` and `toLocalIterator` of a value of each type give the same results as
official PySpark 4.3.0.
- Still different: `toLocalIterator` of a calendar interval gives the
client's string, where PySpark gives PyArrow's `MonthDayNano`. Returning that
needs a new variant of the core's `Value`.
- The lib tests of both crates (590 passed), the build of all tests, also
with `wasm-udf`, the offline Python suites (2056 passed, 12 skipped), and the
live drop-in and Arrow suites (160 passed) and live cases (10 passed) against a
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #305 from HyukjinKwon/SPARK-61108-collect-nested-names.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/dataframe.rs | 47 +++++++--
crates/spark-connect/src/dataframe.rs | 175 +++++++++++++++++++++++++++++++---
python/tests/test_dropin_e2e.py | 17 ++++
python/tests/test_dropin_offline.py | 37 +++++++
4 files changed, 255 insertions(+), 21 deletions(-)
diff --git a/crates/pyspark-rs/src/dataframe.rs
b/crates/pyspark-rs/src/dataframe.rs
index 16e6fd1..597c843 100644
--- a/crates/pyspark-rs/src/dataframe.rs
+++ b/crates/pyspark-rs/src/dataframe.rs
@@ -3727,26 +3727,58 @@ fn udt_field_converters(
/// Whether `data_type` holds a year-month interval, also nested in an array,
a map, a struct or
/// a user-defined type (`pyspark.sql.types._has_type`).
fn has_year_month_interval(data_type: &spark_connect::types::DataType) -> bool
{
+ has_type(data_type, &|t| {
+ matches!(t, spark_connect::types::DataType::YearMonthInterval { .. })
+ })
+}
+
+/// Whether `data_type` is of `is`, also nested in an array, a map, a struct
or a user-defined
+/// type.
+fn has_type(
+ data_type: &spark_connect::types::DataType,
+ is: &dyn Fn(&spark_connect::types::DataType) -> bool,
+) -> bool {
use spark_connect::types::DataType;
+ if is(data_type) {
+ return true;
+ }
match data_type {
- DataType::YearMonthInterval { .. } => true,
- DataType::Struct { fields } => fields
- .iter()
- .any(|field| has_year_month_interval(&field.data_type)),
- DataType::Array { element_type, .. } =>
has_year_month_interval(element_type),
+ DataType::Struct { fields } => fields.iter().any(|field|
has_type(&field.data_type, is)),
+ DataType::Array { element_type, .. } => has_type(element_type, is),
DataType::Map {
key_type,
value_type,
..
- } => has_year_month_interval(key_type) ||
has_year_month_interval(value_type),
+ } => has_type(key_type, is) || has_type(value_type, is),
DataType::Udt {
sql_type: Some(sql_type),
..
- } => has_year_month_interval(sql_type),
+ } => has_type(sql_type, is),
_ => false,
}
}
+/// Raises as PySpark's `collect` does for a result of `schema` with a
calendar interval: it
+/// checks the Arrow schema of the result with `from_arrow_schema`, which has
no type of the
+/// Arrow `month_day_nano_interval` of a calendar interval. Its
`toLocalIterator` does not.
+fn check_arrow_schema(
+ py: Python<'_>,
+ schema: Option<&spark_connect::types::DataType>,
+) -> PyResult<()> {
+ let calendar = |t: &spark_connect::types::DataType| {
+ matches!(t, spark_connect::types::DataType::CalendarInterval)
+ };
+ if schema.is_some_and(|schema| has_type(schema, &calendar)) {
+ return Err(crate::column::raise_pyspark(
+ py,
+ "PySparkTypeError",
+ "UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION",
+ &[("data_type", "month_day_nano_interval")],
+ ));
+ }
+ Ok(())
+}
+
/// Raises as PySpark's Arrow conversion of the rows does for a result of
`schema` with a
/// year-month interval, which Spark Connect cannot collect: the error is
raised even when the
/// result has no rows.
@@ -3775,6 +3807,7 @@ fn rows_to_py(
rows: Vec<spark_connect::row::Row>,
schema: Option<&spark_connect::types::DataType>,
) -> PyResult<Vec<PyRow>> {
+ check_arrow_schema(py, schema)?;
check_collectable(py, schema)?;
let converters = udt_field_converters(py, schema)?;
Ok(rows
diff --git a/crates/spark-connect/src/dataframe.rs
b/crates/spark-connect/src/dataframe.rs
index 587635c..fc826d5 100644
--- a/crates/spark-connect/src/dataframe.rs
+++ b/crates/spark-connect/src/dataframe.rs
@@ -147,7 +147,8 @@ impl LocalRowIterator {
if let
Some(proto::execute_plan_response::ResponseType::ArrowBatch(batch)) =
resp.response_type
{
- return Some(decode_arrow_batch(&batch));
+ let schema = self.schema.lock().unwrap_or_else(|p|
p.into_inner());
+ return Some(decode_arrow_batch_of(&batch,
schema.as_ref()));
}
// Metrics/progress response: keep pulling for a batch.
}
@@ -221,7 +222,11 @@ fn spawn_prefetch(
if let
Some(proto::execute_plan_response::ResponseType::ArrowBatch(batch)) =
resp.response_type
{
- match decode_arrow_batch(&batch) {
+ let decoded = {
+ let schema = schema.lock().unwrap_or_else(|p|
p.into_inner());
+ decode_arrow_batch_of(&batch, schema.as_ref())
+ };
+ match decoded {
Ok(rows) => {
// A send error means the consumer dropped the
// iterator; stop fetching.
@@ -819,7 +824,7 @@ impl DataFrame {
if let
Some(proto::execute_plan_response::ResponseType::ArrowBatch(batch)) =
resp.response_type
{
- let batch_rows = decode_arrow_batch(&batch)?;
+ let batch_rows = decode_arrow_batch_of(&batch,
schema.as_ref())?;
rows.extend(batch_rows);
}
}
@@ -2317,6 +2322,26 @@ fn random_seed() -> i64 {
/// Decode an Arrow batch into rows.
pub(crate) fn decode_arrow_batch(
batch: &proto::execute_plan_response::ArrowBatch,
+) -> Result<Vec<Row>> {
+ decode_arrow_batch_of(batch, None)
+}
+
+/// The fields of `schema`, a struct of `columns` fields, for the columns of a
batch: none of
+/// another schema, whose names the batch keeps.
+fn spark_fields(schema: Option<&DataType>, columns: usize) ->
Option<&[crate::types::StructField]> {
+ match schema {
+ Some(DataType::Struct { fields }) if fields.len() == columns =>
Some(fields),
+ _ => None,
+ }
+}
+
+/// The rows of `batch`, of the names of `schema`, the schema of the result
that the server sends
+/// before the rows, if any: as PySpark's `ArrowTableToRowsConversion` builds
them by the schema,
+/// the fields of nested structs have the names of the schema, which keep the
duplicate names
+/// that the Arrow data of the server numbers (`a_0`, `a_1`).
+pub(crate) fn decode_arrow_batch_of(
+ batch: &proto::execute_plan_response::ArrowBatch,
+ spark_schema: Option<&DataType>,
) -> Result<Vec<Row>> {
use arrow::ipc::reader::StreamReader;
use std::io::Cursor;
@@ -2342,15 +2367,22 @@ pub(crate) fn decode_arrow_batch(
if num_rows == 0 {
continue;
}
- let field_names = schema
- .fields()
- .iter()
- .map(|field| field.name().clone())
- .collect::<Arc<[String]>>();
+ let fields = spark_fields(spark_schema, record_batch.num_columns());
+ let field_names = match fields {
+ Some(fields) => fields.iter().map(|field|
field.name.clone()).collect(),
+ None => schema
+ .fields()
+ .iter()
+ .map(|field| field.name().clone())
+ .collect::<Arc<[String]>>(),
+ };
let layouts = record_batch
.columns()
.iter()
- .map(|column| ArrowValueLayout::new(column.as_ref()))
+ .enumerate()
+ .map(|(i, column)| {
+ ArrowValueLayout::of(column.as_ref(), fields.map(|fields|
&fields[i].data_type))
+ })
.collect::<Vec<_>>();
for row_idx in 0..num_rows {
@@ -2551,9 +2583,26 @@ enum ArrowValueDecoder<'a> {
}
impl<'a> ArrowValueLayout<'a> {
+ #[cfg(test)]
fn new(array: &'a dyn arrow::array::Array) -> Self {
+ Self::of(array, None)
+ }
+
+ /// The layout of `array`, of the Spark type `data_type` if known: the
fields of its structs,
+ /// also in arrays, maps and user-defined types, have the names of the
type.
+ fn of(array: &'a dyn arrow::array::Array, data_type: Option<&DataType>) ->
Self {
use arrow::array::*;
+ // A user-defined type is in the Arrow data of its SQL type.
+ let mut data_type = data_type;
+ while let Some(DataType::Udt {
+ sql_type: Some(sql_type),
+ ..
+ }) = data_type
+ {
+ data_type = Some(sql_type);
+ }
+
// Resolution never fails: null parents and empty containers must not
// reject unsupported children that are never read.
let any = array.as_any();
@@ -2628,9 +2677,13 @@ impl<'a> ArrowValueLayout<'a> {
} else if let Some(array) =
any.downcast_ref::<IntervalMonthDayNanoArray>() {
ArrowValueDecoder::IntervalMonthDayNano(array)
} else if let Some(array) = any.downcast_ref::<ListArray>() {
+ let element_type = match data_type {
+ Some(DataType::Array { element_type, .. }) =>
Some(element_type.as_ref()),
+ _ => None,
+ };
ArrowValueDecoder::List {
array,
- child: Box::new(Self::new(array.values().as_ref())),
+ child: Box::new(Self::of(array.values().as_ref(),
element_type)),
}
} else if let Some(array) = any.downcast_ref::<StructArray>() {
let fields = array.fields();
@@ -2640,12 +2693,27 @@ impl<'a> ArrowValueLayout<'a> {
.get("variant")
.is_some_and(|value| value == "true")
});
+ let spark_fields = match data_type {
+ Some(DataType::Struct { fields: spark }) if spark.len() ==
fields.len() => {
+ Some(spark)
+ }
+ _ => None,
+ };
ArrowValueDecoder::Struct {
- names: fields.iter().map(|field|
field.name().clone()).collect(),
+ names: match spark_fields {
+ Some(spark) => spark.iter().map(|field|
field.name.clone()).collect(),
+ None => fields.iter().map(|field|
field.name().clone()).collect(),
+ },
children: array
.columns()
.iter()
- .map(|child| Self::new(child.as_ref()))
+ .enumerate()
+ .map(|(i, child)| {
+ Self::of(
+ child.as_ref(),
+ spark_fields.map(|spark| &spark[i].data_type),
+ )
+ })
.collect(),
variant_fields: is_variant.then(|| {
(
@@ -2655,11 +2723,19 @@ impl<'a> ArrowValueLayout<'a> {
}),
}
} else if let Some(array) = any.downcast_ref::<MapArray>() {
+ let (key_type, value_type) = match data_type {
+ Some(DataType::Map {
+ key_type,
+ value_type,
+ ..
+ }) => (Some(key_type.as_ref()), Some(value_type.as_ref())),
+ _ => (None, None),
+ };
ArrowValueDecoder::Map {
array,
children: Box::new((
- Self::new(array.keys().as_ref()),
- Self::new(array.values().as_ref()),
+ Self::of(array.keys().as_ref(), key_type),
+ Self::of(array.values().as_ref(), value_type),
)),
}
} else {
@@ -2952,6 +3028,77 @@ mod nested_decode_tests {
assert_eq!(rows[2].get(1), Some(&Value::String("é\0雪".into())));
}
+ #[test]
+ fn nested_struct_fields_take_the_names_of_the_schema_of_the_result() {
+ // As PySpark's ArrowTableToRowsConversion, which builds the rows by
the schema of the
+ // result: the fields of nested structs, also in arrays and maps, have
the names of the
+ // schema, which keep the duplicate names that the server's Arrow data
numbers.
+ use crate::types::{DataType as Spark, StructField as SparkField};
+ let field = |name: &str, data_type: Spark| SparkField {
+ name: name.to_string(),
+ data_type,
+ nullable: true,
+ metadata: Default::default(),
+ };
+ let pair = || {
+ StructArray::from(vec![
+ (
+ Arc::new(Field::new("a_0", DataType::Int32, true)),
+ Arc::new(Int32Array::from(vec![1])) as ArrayRef,
+ ),
+ (
+ Arc::new(Field::new("a_1", DataType::Int32, true)),
+ Arc::new(Int32Array::from(vec![2])) as ArrayRef,
+ ),
+ ])
+ };
+ let list = ListArray::new(
+ Arc::new(Field::new("element", pair().data_type().clone(), true)),
+ arrow::buffer::OffsetBuffer::from_lengths([1]),
+ Arc::new(pair()),
+ None,
+ );
+ let batch = encoded_batches(
+ vec![("v", Arc::new(pair()) as ArrayRef), ("w", Arc::new(list))],
+ 1,
+ false,
+ );
+ let pair_type = || Spark::Struct {
+ fields: vec![field("a", Spark::Integer), field("a",
Spark::Integer)],
+ };
+ let schema = Spark::Struct {
+ fields: vec![
+ field("v", pair_type()),
+ field(
+ "w",
+ Spark::Array {
+ element_type: Box::new(pair_type()),
+ contains_null: true,
+ },
+ ),
+ ],
+ };
+ let rows = decode_arrow_batch_of(&batch, Some(&schema)).unwrap();
+ assert_eq!(rows[0].fields(), &["v", "w"]);
+ assert_eq!(as_struct(rows[0].get(0).unwrap()).fields(), &["a", "a"]);
+ let Some(Value::List(items)) = rows[0].get(1) else {
+ panic!("expected a list");
+ };
+ assert_eq!(as_struct(&items[0]).fields(), &["a", "a"]);
+ assert_eq!(
+ as_struct(&items[0]).values(),
+ &[Value::Integer(1), Value::Integer(2)]
+ );
+ // Without the schema, or of a schema of another shape, the names of
the Arrow data.
+ let other = Spark::Struct {
+ fields: vec![field("v", Spark::Integer)],
+ };
+ for schema in [None, Some(&other)] {
+ let rows = decode_arrow_batch_of(&batch, schema).unwrap();
+ assert_eq!(as_struct(rows[0].get(0).unwrap()).fields(), &["a_0",
"a_1"]);
+ }
+ }
+
#[test]
fn sliced_parents_decode_children_that_were_already_sliced() {
let leaf = StructArray::from(vec![(
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index f3be9c7..d7dc597 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -1355,6 +1355,23 @@ def
test_writer_v2_takes_its_arguments_as_pyspark(session):
session.sql(f"DROP TABLE IF EXISTS {name}")
+def
test_collected_rows_take_the_names_of_the_schema_of_the_result_as_pyspark(session):
+ # As PySpark Connect's: the fields of nested structs have the names of the
schema, which keep
+ # duplicate names, and collect raises for a calendar interval.
+ from pyspark.errors import PySparkTypeError
+
+ df = session.sql(
+ "SELECT named_struct('a', 1, 'a', 2) AS v, array(named_struct('b', 1,
'b', 'x')) AS w")
+ row = df.collect()[0]
+ assert row.v.__fields__ == ["a", "a"] and tuple(row.v) == (1, 2) and
row.v.a == 1
+ assert row.w[0].__fields__ == ["b", "b"]
+ assert row.asDict(recursive=True) == {"v": {"a": 2}, "w": [{"b": "x"}]}
+ assert next(df.toLocalIterator()).v.__fields__ == ["a", "a"]
+ with pytest.raises(PySparkTypeError) as e:
+ session.sql("SELECT make_interval(1) AS v").collect()
+ assert e.value.getCondition() ==
"UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION"
+ assert e.value.getMessageParameters() == {"data_type":
"month_day_nano_interval"}
+
def test_collect_reads_a_result_across_stream_ends(session):
# Takes several response streams when the server's senderMaxStreamSize is
small.
n = 300_000
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index f1488a7..1f546c0 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -663,6 +663,43 @@ def
test_builder_options_are_set_on_the_session_as_pyspark(mock_connect):
assert len(service.set_confs) == 1
+def
test_collected_rows_take_the_names_of_the_schema_of_the_result_as_pyspark(mock_connect):
+ # As PySpark Connect's, which builds the rows by the schema of the result:
the fields of
+ # nested structs have the names of the schema, which keep the duplicate
names that the
+ # server's Arrow data numbers; collect raises for a calendar interval,
whose Arrow type
+ # from_arrow_schema does not convert, while toLocalIterator does not check
it.
+ import pyarrow as pa
+ from _mock_connect_server import data_type_proto
+ from pyspark.errors import PySparkTypeError
+
+ spark, service = mock_connect
+
+ def respond(schema, table):
+ sink = pa.BufferOutputStream()
+ with pa.ipc.new_stream(sink, table.schema) as writer:
+ writer.write_table(table)
+ # ExecutePlanResponse.schema=7 and arrow_batch=2:
ArrowBatch.row_count=1, data=2.
+ service.execute_payload = message_field(7, schema) + message_field(
+ 2, varint_field(1, table.num_rows) + message_field(2,
sink.getvalue().to_pybytes()))
+
+ field = {"name": "a", "type": "integer", "nullable": True, "metadata": {}}
+ pair = {"type": "struct", "fields": [field, field]}
+ respond(data_type_proto({"type": "struct", "fields": [
+ {"name": "v", "type": pair, "nullable": True, "metadata": {}}]}),
+ pa.table({"v": pa.array([{"a_0": 1, "a_1": 2}])}))
+ for rows in (spark.range(1).collect(),
list(spark.range(1).toLocalIterator())):
+ assert rows[0].v.__fields__ == ["a", "a"]
+ assert tuple(rows[0].v) == (1, 2)
+ # Struct.fields=1, StructField: name=1, data_type=2
(DataType.calendar_interval=17).
+ respond(message_field(21, message_field(1, message_field(1, "v") +
message_field(
+ 2, message_field(17, b"")))),
+ pa.table({"v": pa.array([(1, 2, 3)], pa.month_day_nano_interval())}))
+ with pytest.raises(PySparkTypeError) as e:
+ spark.range(1).collect()
+ assert e.value.getCondition() ==
"UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION"
+ assert e.value.getMessageParameters() == {"data_type":
"month_day_nano_interval"}
+ assert len(list(spark.range(1).toLocalIterator())) == 1
+
def test_tags_are_those_of_each_thread_as_pyspark(mock_connect):
# As PySpark's: the tags of addTag are those of the current thread, of a
set of getTags,
# which the operations that the thread starts carry; a new thread starts
without tags.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]