This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 07c62825 [SPARK-61212][CONNECT][RUST] Normalize Arrow map child names
in local input
07c62825 is described below
commit 07c62825d7f35a4d9373898f330681151e206f15
Author: DB Tsai <[email protected]>
AuthorDate: Mon Nov 9 15:51:08 2026 +0900
[SPARK-61212][CONNECT][RUST] Normalize Arrow map child names in local input
### What changes were proposed in this pull request?
Normalize Arrow map storage fields to `entries`, `key`, and `value` before
serializing local input or uploading chunked artifacts. Checked reconstruction
preserves existing value, offset, and validity buffers, field metadata,
nullability, ordering flags, and slices. Inputs that need no schema changes
reuse their batches.
This runs at the shared Arrow-import boundary, including Rust batch/row
input and Python Arrow-table input. The internal IPC bridge also applies it
when decoding larger input for chunking; raw `LogicalPlan::LocalRelation` byte
constructors remain transparent.
### Why are the changes needed?
[SPARK-61212](https://issues.apache.org/jira/browse/SPARK-61212): Arrow
Rust's default `MapBuilder` names its children `keys` and `values`. Spark Java
looks them up as `key` and `value`, so a nonempty local map can fail with an
`ArrowColumnVector.MapAccessor` null-pointer exception. The mismatch also
appears in nested maps and empty IPC schemas.
### Does this PR introduce _any_ user-facing change?
Yes. Default and custom-named Arrow map inputs now use Spark-compatible
storage names. Reconstructed columns are checked against their schema;
incompatible child types or union ordering fail before configuration or upload
RPCs. IPC serialization is still required for transport.
### How was this patch tested?
- 509 Rust library tests passed, including new nested/sliced map,
buffer-sharing, schema-mismatch, and safe malformed-union regressions.
- Four live Rust tests passed against Spark 4.2: the existing array-literal
tests and new map tests covering inline input, multiple chunk artifacts,
nested/null/empty/Unicode values, and zero rows. Both new map tests fail on
unpatched base `925dc876f63f1f779fe176f62575a0aa96cc6886`.
- All integration tests and the `wasm-udf` test configuration compile in
release mode; formatting and scoped library/new-test Clippy pass.
- Full-workspace Clippy has existing approximate-constant test errors and
denied PyO3 warnings, reproduced on the unpatched base. The patch introduces
none of those errors; all-target checking with advisory lint severity passes.
The last documentation-only edits do not change the tested implementation.
No Python API-parity or performance claim is made.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex
This pull request and its description were written by OpenAI Codex.
Closes #396 from dbtsai/codex/normalize-arrow-map-input.
Lead-authored-by: DB Tsai <[email protected]>
Co-authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
.github/workflows/build_python_connect.yml | 1 +
crates/spark-connect/src/arrow_input.rs | 859 +++++++++++++++++++++++++-
crates/spark-connect/src/session.rs | 6 +
crates/spark-connect/tests/arrow_map_input.rs | 197 ++++++
4 files changed, 1054 insertions(+), 9 deletions(-)
diff --git a/.github/workflows/build_python_connect.yml
b/.github/workflows/build_python_connect.yml
index 8c875361..e6f8815b 100644
--- a/.github/workflows/build_python_connect.yml
+++ b/.github/workflows/build_python_connect.yml
@@ -392,6 +392,7 @@ jobs:
cargo test -p apache-spark-connect \
--test e2e_functions --test e2e_columns --test e2e_dataframe \
--test e2e_coverage --test e2e_linear_svc --test
e2e_linear_regression --test e2e_integration \
+ --test arrow_map_input \
-- --test-threads=1
- name: Rust WASM UDF e2e test (against the live server)
diff --git a/crates/spark-connect/src/arrow_input.rs
b/crates/spark-connect/src/arrow_input.rs
index 5bb2f007..31751353 100644
--- a/crates/spark-connect/src/arrow_input.rs
+++ b/crates/spark-connect/src/arrow_input.rs
@@ -1,12 +1,15 @@
//! Arrow local relations and session-scoped chunked uploads.
-use arrow::datatypes::SchemaRef;
+use arrow::array::{make_array, ArrayData};
+use arrow::datatypes::{DataType as ArrowType, FieldRef, Schema, SchemaRef};
use arrow::ipc::writer::StreamWriter;
-use arrow::record_batch::RecordBatch;
+use arrow::record_batch::{RecordBatch, RecordBatchOptions};
use spark_connect_core::client::SparkConnectClient;
use spark_connect_core::error::{Result, SparkError, SparkErrorKind};
use spark_connect_core::runtime::block_on;
+use std::borrow::Cow;
use std::collections::HashMap;
+use std::sync::Arc;
use crate::plan::LogicalPlan;
use crate::types::DataType;
@@ -120,6 +123,184 @@ fn validate(batches: &[RecordBatch]) -> Result<()> {
Ok(())
}
+fn normalize_map_field(field: &FieldRef) -> Result<FieldRef> {
+ let data_type = normalize_map_type(field.data_type())?;
+ if &data_type == field.data_type() {
+ return Ok(Arc::clone(field));
+ }
+ Ok(Arc::new(field.as_ref().clone().with_data_type(data_type)))
+}
+
+fn normalize_map_type(data_type: &ArrowType) -> Result<ArrowType> {
+ use ArrowType as A;
+ Ok(match data_type {
+ A::Map(entries, sorted) => {
+ let entries = normalize_map_field(entries)?;
+ let A::Struct(fields) = entries.data_type() else {
+ return Err(SparkError::connect_msg(
+ "Arrow map entries must be a struct",
+ ));
+ };
+ if fields.len() != 2 {
+ return Err(SparkError::connect_msg(
+ "Arrow maps must have key and value children",
+ ));
+ }
+ let fields = fields
+ .iter()
+ .zip(["key", "value"])
+ .map(|(field, name)|
Arc::new(field.as_ref().clone().with_name(name)))
+ .collect();
+ A::Map(
+ Arc::new(
+ entries
+ .as_ref()
+ .clone()
+ .with_name("entries")
+ .with_data_type(A::Struct(fields)),
+ ),
+ *sorted,
+ )
+ }
+ A::Struct(fields) => A::Struct(
+ fields
+ .iter()
+ .map(normalize_map_field)
+ .collect::<Result<Vec<_>>>()?
+ .into(),
+ ),
+ A::List(field) => A::List(normalize_map_field(field)?),
+ A::LargeList(field) => A::LargeList(normalize_map_field(field)?),
+ A::ListView(field) => A::ListView(normalize_map_field(field)?),
+ A::LargeListView(field) =>
A::LargeListView(normalize_map_field(field)?),
+ A::FixedSizeList(field, size) =>
A::FixedSizeList(normalize_map_field(field)?, *size),
+ A::Union(fields, mode) => {
+ let fields = fields
+ .iter()
+ .map(|(id, field)| Ok((id, normalize_map_field(field)?)))
+ .collect::<Result<Vec<_>>>()?
+ .into_iter()
+ .collect();
+ A::Union(fields, *mode)
+ }
+ A::Dictionary(key, value) => A::Dictionary(
+ Box::new(normalize_map_type(key)?),
+ Box::new(normalize_map_type(value)?),
+ ),
+ A::RunEndEncoded(ends, values) => {
+ A::RunEndEncoded(normalize_map_field(ends)?,
normalize_map_field(values)?)
+ }
+ other => other.clone(),
+ })
+}
+
+fn child_types(data_type: &ArrowType) -> Vec<&ArrowType> {
+ use ArrowType as A;
+ match data_type {
+ A::Map(field, _)
+ | A::List(field)
+ | A::LargeList(field)
+ | A::ListView(field)
+ | A::LargeListView(field)
+ | A::FixedSizeList(field, _) => vec![field.data_type()],
+ A::Struct(fields) => fields.iter().map(|field|
field.data_type()).collect(),
+ A::Union(fields, _) => fields.iter().map(|(_, field)|
field.data_type()).collect(),
+ A::Dictionary(_, value) => vec![value.as_ref()],
+ A::RunEndEncoded(ends, values) => vec![ends.data_type(),
values.data_type()],
+ _ => vec![],
+ }
+}
+
+/// Reconcile schema field names and metadata only through checked
construction.
+/// Arrow's name-insensitive equality also accepts reordered union fields,
whose
+/// children cannot be relabeled by position, so reject those before recursion.
+fn with_schema_type(data: ArrayData, data_type: &ArrowType) ->
Result<ArrayData> {
+ if data.data_type() == data_type {
+ return Ok(data);
+ }
+ let mismatch = || SparkError::connect_msg("Arrow column type does not
match its schema field");
+ if !data.data_type().equals_datatype(data_type) {
+ return Err(mismatch());
+ }
+ if let (ArrowType::Union(before, _), ArrowType::Union(after, _)) =
(data.data_type(), data_type)
+ {
+ if !before
+ .iter()
+ .map(|(id, _)| id)
+ .eq(after.iter().map(|(id, _)| id))
+ {
+ return Err(mismatch());
+ }
+ }
+ let child_types = child_types(data_type);
+ if child_types.len() != data.child_data().len() {
+ return Err(mismatch());
+ }
+ let children = data
+ .child_data()
+ .iter()
+ .zip(child_types)
+ .map(|(child, data_type)| with_schema_type(child.clone(), data_type))
+ .collect::<Result<Vec<_>>>()?;
+ // Safe Arrow union constructors can retain inconsistent declared child
+ // types. Keep validation even when only changing map field names.
+ data.into_builder()
+ .data_type(data_type.clone())
+ .child_data(children)
+ .build()
+ .map_err(arrow_error)
+}
+
+/// Arrow permits arbitrary map child names, but Spark's ArrowColumnVector
looks
+/// up map keys and values by name. Rewrite only those fields, including nested
+/// maps, while sharing the original value, offset, and validity buffers.
+/// When a map schema changes, checked reconstruction also reconciles column
+/// names and metadata with that schema, rejecting incompatible union order.
+fn normalize_map_names(batches: &[RecordBatch]) -> Result<Cow<'_,
[RecordBatch]>> {
+ // Do not let canonicalization conceal a mismatch in the supplied schemas.
+ validate(batches)?;
+ let Some(first) = batches.first() else {
+ return Ok(Cow::Borrowed(batches));
+ };
+ let original = first.schema();
+ let fields = original
+ .fields()
+ .iter()
+ .map(normalize_map_field)
+ .collect::<Result<Vec<_>>>()?;
+ if fields.as_slice() == original.fields().as_ref() {
+ return Ok(Cow::Borrowed(batches));
+ }
+ let schema = Arc::new(Schema::new_with_metadata(
+ fields,
+ original.metadata().clone(),
+ ));
+ batches
+ .iter()
+ .map(|batch| {
+ let columns = batch
+ .columns()
+ .iter()
+ .zip(schema.fields())
+ .map(|(array, field)| {
+ if array.data_type() == field.data_type() {
+ Ok(Arc::clone(array))
+ } else {
+ with_schema_type(array.to_data(),
field.data_type()).map(make_array)
+ }
+ })
+ .collect::<Result<Vec<_>>>()?;
+ RecordBatch::try_new_with_options(
+ Arc::clone(&schema),
+ columns,
+
&RecordBatchOptions::new().with_row_count(Some(batch.num_rows())),
+ )
+ .map_err(arrow_error)
+ })
+ .collect::<Result<Vec<_>>>()
+ .map(Cow::Owned)
+}
+
pub(crate) fn batch_data_size(batch: &RecordBatch) -> usize {
batch
.columns()
@@ -137,7 +318,11 @@ pub(crate) fn inline_relation(
batches: &[RecordBatch],
schema: Option<DataType>,
) -> Result<LogicalPlan> {
- validate(batches)?;
+ let batches = normalize_map_names(batches)?;
+ write_inline_relation(&batches, schema)
+}
+
+fn write_inline_relation(batches: &[RecordBatch], schema: Option<DataType>) ->
Result<LogicalPlan> {
let data = if let Some(first) = batches.first() {
let mut writer = StreamWriter::try_new(Vec::new(),
&first.schema()).map_err(arrow_error)?;
for batch in batches {
@@ -156,11 +341,12 @@ pub(crate) fn create_relation(
schema: Option<DataType>,
options: Option<&ArrowInputOptions>,
) -> Result<LogicalPlan> {
- // Validate before any RPC, and preserve the offline empty-input path.
- validate(&batches)?;
+ // Preserve the offline empty-input path. Both paths validate the original
+ // schemas and normalize map names exactly once, before any RPC.
if batches.iter().all(|batch| batch.num_rows() == 0) {
return inline_relation(&batches, schema);
}
+ let batches = normalize_map_names(&batches)?;
let fetched;
let options = match options {
Some(options) => options,
@@ -172,7 +358,7 @@ pub(crate) fn create_relation(
// Count the logical slices rather than their potentially much larger
backing buffers.
let bytes: usize = batches.iter().map(batch_data_size).sum();
if bytes < options.cache_threshold {
- return inline_relation(&batches, schema);
+ return write_inline_relation(&batches, schema);
}
let options = &options.upload;
@@ -221,6 +407,7 @@ fn new_writer(schema: &SchemaRef) ->
Result<StreamWriter<Vec<u8>>> {
/// Match PySpark's pre-write estimate: stream schema plus each batch's data
size.
/// The row setting caps individual record batches, not the complete IPC
stream.
+/// Callers supply batches whose schemas and map field names are already
prepared.
fn serialize_chunks(
batches: &[RecordBatch],
options: &Options,
@@ -262,8 +449,13 @@ fn serialize_chunks(
#[cfg(test)]
mod tests {
use super::*;
- use arrow::array::Int64Array;
- use arrow::datatypes::{DataType as ArrowType, Field, Schema};
+ use arrow::array::builder::Int64Builder;
+ use arrow::array::{
+ Array, ArrayRef, DictionaryArray, FixedSizeListArray, Int32Array,
Int64Array, Int8Array,
+ ListArray, MapArray, MapBuilder, RunArray, StructArray, UnionArray,
+ };
+ use arrow::buffer::{NullBuffer, OffsetBuffer};
+ use arrow::datatypes::{DataType as ArrowType, Field, Int32Type, Int8Type,
Schema};
use arrow::ipc::reader::StreamReader;
use std::io::Cursor;
use std::ops::Range;
@@ -274,6 +466,654 @@ mod tests {
RecordBatch::try_new(schema,
vec![Arc::new(Int64Array::from_iter_values(values))]).unwrap()
}
+ fn map(rows: usize) -> MapArray {
+ let mut builder = MapBuilder::new(None, Int64Builder::new(),
Int64Builder::new());
+ for row in 0..rows {
+ if row % 3 == 1 {
+ builder.append(false).unwrap();
+ } else {
+ if row % 3 == 0 {
+ builder.keys().append_value(row as i64);
+ builder.values().append_value(row as i64 + 10);
+ builder.keys().append_value(row as i64 + 100);
+ builder.values().append_null();
+ }
+ builder.append(true).unwrap();
+ }
+ }
+ builder.finish()
+ }
+
+ fn map_batch(rows: usize) -> RecordBatch {
+ let map: ArrayRef = Arc::new(map(rows));
+ RecordBatch::try_new(
+ Arc::new(Schema::new(vec![Field::new(
+ "mapping",
+ map.data_type().clone(),
+ true,
+ )])),
+ vec![map],
+ )
+ .unwrap()
+ }
+
+ fn map_fields(data_type: &ArrowType) -> (&FieldRef,
&arrow::datatypes::Fields, bool) {
+ let ArrowType::Map(entries, sorted) = data_type else {
+ panic!("expected map");
+ };
+ let ArrowType::Struct(fields) = entries.data_type() else {
+ panic!("expected entries struct");
+ };
+ (entries, fields, *sorted)
+ }
+
+ fn assert_canonical_map(data_type: &ArrowType) {
+ let (entries, fields, _) = map_fields(data_type);
+ assert_eq!(entries.name(), "entries");
+ assert_eq!(fields[0].name(), "key");
+ assert_eq!(fields[1].name(), "value");
+ }
+
+ fn assert_shared_buffers(before: &ArrayData, after: &ArrayData) {
+ assert_eq!(
+ (before.len(), before.offset()),
+ (after.len(), after.offset())
+ );
+ assert_eq!(before.buffers().len(), after.buffers().len());
+ for (before, after) in before.buffers().iter().zip(after.buffers()) {
+ assert!(before.ptr_eq(after), "normalization copied an Arrow
buffer");
+ }
+ match (before.nulls(), after.nulls()) {
+ (Some(before), Some(after)) =>
assert!(before.inner().ptr_eq(after.inner())),
+ (None, None) => {}
+ _ => panic!("normalization changed the validity bitmap"),
+ }
+ assert_eq!(before.child_data().len(), after.child_data().len());
+ for (before, after) in
before.child_data().iter().zip(after.child_data()) {
+ assert_shared_buffers(before, after);
+ }
+ }
+
+ fn normalize_array(array: ArrayRef) -> ArrayRef {
+ let batch = RecordBatch::try_new(
+ Arc::new(Schema::new(vec![Field::new(
+ "user_column",
+ array.data_type().clone(),
+ true,
+ )])),
+ vec![Arc::clone(&array)],
+ )
+ .unwrap();
+ let normalized =
normalize_map_names(std::slice::from_ref(&batch)).unwrap();
+ let result = Arc::clone(normalized[0].column(0));
+ assert_shared_buffers(&array.to_data(), &result.to_data());
+ result.to_data().validate_full().unwrap();
+ result
+ }
+
+ fn assert_map_values_unchanged(before: &ArrayRef, after: &ArrayRef) {
+ let before = before.as_any().downcast_ref::<MapArray>().unwrap();
+ let after = after.as_any().downcast_ref::<MapArray>().unwrap();
+ assert_canonical_map(after.data_type());
+ assert_eq!(before.len(), after.len());
+ assert_eq!(before.offsets(), after.offsets());
+ assert_eq!(before.nulls(), after.nulls());
+ assert_eq!(before.keys().to_data(), after.keys().to_data());
+ assert_eq!(before.values().to_data(), after.values().to_data());
+ }
+
+ #[test]
+ fn map_names_are_canonical_in_inline_and_chunked_ipc() {
+ let batch = map_batch(5).slice(1, 4);
+ let expected = normalize_map_names(std::slice::from_ref(&batch))
+ .unwrap()
+ .into_owned()
+ .remove(0);
+ let before =
batch.column(0).as_any().downcast_ref::<MapArray>().unwrap();
+ let after = expected
+ .column(0)
+ .as_any()
+ .downcast_ref::<MapArray>()
+ .unwrap();
+ assert_canonical_map(after.data_type());
+ assert_eq!(before.keys().to_data(), after.keys().to_data());
+ assert_eq!(before.values().to_data(), after.values().to_data());
+ assert_eq!(before.offsets(), after.offsets());
+
+ let LogicalPlan::LocalRelation {
+ data: Some(data), ..
+ } = inline_relation(std::slice::from_ref(&batch), None).unwrap()
+ else {
+ panic!("expected inline data");
+ };
+ let mut reader = StreamReader::try_new(Cursor::new(data),
None).unwrap();
+ assert_canonical_map(reader.schema().field(0).data_type());
+ assert_eq!(reader.next().unwrap().unwrap(), expected);
+ assert!(reader.next().is_none());
+
+ let options = Options {
+ size_limit: usize::MAX,
+ chunk_rows: 2,
+ chunk_bytes: 1,
+ upload_batch_bytes: usize::MAX,
+ };
+ let mut streams = Vec::new();
+ serialize_chunks(std::slice::from_ref(&expected), &options, &mut
|data| {
+ streams.push(data);
+ Ok(())
+ })
+ .unwrap();
+ assert_eq!(streams.len(), 2);
+ let mut offset = 0;
+ for data in streams {
+ let reader = StreamReader::try_new(Cursor::new(data),
None).unwrap();
+ assert_canonical_map(reader.schema().field(0).data_type());
+ for batch in reader {
+ let batch = batch.unwrap();
+ assert_eq!(batch, expected.slice(offset, batch.num_rows()));
+ offset += batch.num_rows();
+ }
+ }
+ assert_eq!(offset, expected.num_rows());
+ }
+
+ #[test]
+ fn nested_sliced_maps_share_buffers_and_preserve_field_attributes() {
+ let tagged = |name: &str, data_type, nullable| {
+ Field::new(name, data_type, nullable)
+ .with_metadata(HashMap::from([("tag".into(), name.into())]))
+ };
+ let nested: ArrayRef = Arc::new(map(4));
+ let entries = StructArray::new(
+ vec![
+ tagged("keys", ArrowType::Int64, false),
+ tagged("values", nested.data_type().clone(), true),
+ ]
+ .into(),
+ vec![Arc::new(Int64Array::from(vec![1, 2, 3, 4])), nested],
+ None,
+ );
+ let outer: ArrayRef = Arc::new(MapArray::new(
+ Arc::new(tagged("pairs", entries.data_type().clone(), false)),
+ OffsetBuffer::new(vec![0, 1, 2, 3, 4].into()),
+ entries,
+ Some(NullBuffer::from(vec![true, false, true, true])),
+ true,
+ ));
+ let list: ArrayRef = Arc::new(ListArray::new(
+ Arc::new(tagged("user_items", outer.data_type().clone(), true)),
+ OffsetBuffer::new(vec![0, 1, 3, 4].into()),
+ outer,
+ Some(NullBuffer::from(vec![true, false, true])),
+ ));
+ let record: ArrayRef = Arc::new(StructArray::new(
+ vec![tagged("user_lists", list.data_type().clone(), true)].into(),
+ vec![list],
+ Some(NullBuffer::from(vec![true, false, true])),
+ ));
+ let schema = Schema::new_with_metadata(
+ vec![
+ tagged("user_record", record.data_type().clone(), true),
+ tagged("untouched", ArrowType::Int64, false),
+ ],
+ HashMap::from([("schema_tag".into(), "preserved".into())]),
+ );
+ let batch = RecordBatch::try_new(
+ Arc::new(schema),
+ vec![record, Arc::new(Int64Array::from(vec![11, 22, 33]))],
+ )
+ .unwrap()
+ .slice(1, 2);
+ let normalized = normalize_map_names(std::slice::from_ref(&batch))
+ .unwrap()
+ .into_owned()
+ .remove(0);
+ assert_eq!(normalized.num_rows(), 2);
+ assert_eq!(batch.schema().metadata(), normalized.schema().metadata());
+ assert!(Arc::ptr_eq(batch.column(1), normalized.column(1)));
+ for (before, after) in
batch.columns().iter().zip(normalized.columns()) {
+ assert_shared_buffers(&before.to_data(), &after.to_data());
+ after.to_data().validate_full().unwrap();
+ }
+ let original = batch.schema();
+ let result = normalized.schema();
+ let mut before = original.field(0);
+ let mut after = result.field(0);
+ for expected_name in ["user_record", "user_lists", "user_items"] {
+ assert_eq!(after.name(), expected_name);
+ assert_eq!(before.metadata(), after.metadata());
+ assert_eq!(before.is_nullable(), after.is_nullable());
+ if expected_name != "user_items" {
+ before = match before.data_type() {
+ ArrowType::Struct(fields) => fields[0].as_ref(),
+ ArrowType::List(field) => field.as_ref(),
+ _ => panic!("expected user container"),
+ };
+ after = match after.data_type() {
+ ArrowType::Struct(fields) => fields[0].as_ref(),
+ ArrowType::List(field) => field.as_ref(),
+ _ => panic!("expected user container"),
+ };
+ }
+ }
+ let (old_entries, old_fields, old_sorted) =
map_fields(before.data_type());
+ let (entries, fields, sorted) = map_fields(after.data_type());
+ assert_canonical_map(after.data_type());
+ assert!(old_sorted && sorted);
+ assert_eq!(old_entries.metadata(), entries.metadata());
+ assert_eq!(old_entries.is_nullable(), entries.is_nullable());
+ for (before, after) in old_fields.iter().zip(fields) {
+ assert_eq!(before.metadata(), after.metadata());
+ assert_eq!(before.is_nullable(), after.is_nullable());
+ }
+ assert_canonical_map(fields[1].data_type());
+ assert!(!map_fields(fields[1].data_type()).2);
+ }
+
+ #[test]
+ fn original_map_schema_mismatches_are_rejected_before_normalization() {
+ fn assert_schema_error<T>(result: Result<T>) {
+ let Err(error) = result else {
+ panic!("expected the original batch-schema mismatch");
+ };
+ assert_eq!(error.message(), "Arrow batch schemas must match");
+ }
+ let session = crate::SparkSession::builder()
+ .remote("sc://127.0.0.1:1")
+ .get_or_create()
+ .unwrap();
+ for rows in [0, 1] {
+ let raw = map_batch(rows);
+ let canonical = normalize_map_names(std::slice::from_ref(&raw))
+ .unwrap()
+ .into_owned()
+ .remove(0);
+ let batches = vec![raw, canonical];
+ assert_schema_error(normalize_map_names(&batches));
+ assert_schema_error(inline_relation(&batches, None));
+ assert_schema_error(session.create_dataframe_from_arrow(batches,
None));
+ }
+ }
+
+ #[test]
+ fn reordered_union_schema_fields_are_rejected_before_relabeling_children()
{
+ let maps: ArrayRef = Arc::new(map(1));
+ let fields = [
+ (0, Arc::new(Field::new("small", ArrowType::Int32, false))),
+ (1, Arc::new(Field::new("large", ArrowType::Int64, false))),
+ (
+ 2,
+ Arc::new(Field::new("maps", maps.data_type().clone(), true)),
+ ),
+ ];
+ let array: ArrayRef = Arc::new(
+ UnionArray::try_new(
+ fields.clone().into_iter().collect(),
+ vec![0, 1, 2].into(),
+ Some(vec![0, 0, 0].into()),
+ vec![
+ Arc::new(Int32Array::from(vec![42])),
+ Arc::new(Int64Array::from(vec![84])),
+ maps,
+ ],
+ )
+ .unwrap(),
+ );
+ array.to_data().validate_full().unwrap();
+ let schema_type = ArrowType::Union(
+ [fields[1].clone(), fields[0].clone(), fields[2].clone()]
+ .into_iter()
+ .collect(),
+ arrow::datatypes::UnionMode::Dense,
+ );
+ assert_ne!(array.data_type(), &schema_type);
+ assert!(array.data_type().equals_datatype(&schema_type));
+ let batch = RecordBatch::try_new_with_options(
+ Arc::new(Schema::new(vec![Field::new(
+ "user_union",
+ schema_type,
+ true,
+ )])),
+ vec![array],
+ &RecordBatchOptions::new().with_match_field_names(false),
+ )
+ .unwrap();
+ let session = crate::SparkSession::builder()
+ .remote("sc://127.0.0.1:1")
+ .get_or_create()
+ .unwrap();
+ for result in [
+ normalize_map_names(std::slice::from_ref(&batch)).map(|_| ()),
+ inline_relation(std::slice::from_ref(&batch), None).map(|_| ()),
+ session
+ .create_dataframe_from_arrow(vec![batch], None)
+ .map(|_| ()),
+ ] {
+ let Err(error) = result else {
+ panic!("reordered union fields must not relabel physical
children");
+ };
+ assert_eq!(
+ error.message(),
+ "Arrow column type does not match its schema field"
+ );
+ }
+ }
+
+ #[test]
+ fn inconsistent_union_child_types_are_rejected_without_reinterpretation() {
+ let session = crate::SparkSession::builder()
+ .remote("sc://127.0.0.1:1")
+ .get_or_create()
+ .unwrap();
+ // The second fixture's Int32 buffer has enough bytes for one Int64;
+ // checking buffer sizes alone would not prove retyping was rejected.
+ for values in [vec![42], vec![42, 43]] {
+ let maps: ArrayRef = Arc::new(map(1));
+ let array: ArrayRef = Arc::new(
+ UnionArray::try_new(
+ [
+ (0, Arc::new(Field::new("wide", ArrowType::Int64,
false))),
+ (
+ 1,
+ Arc::new(Field::new("maps",
maps.data_type().clone(), true)),
+ ),
+ ]
+ .into_iter()
+ .collect(),
+ vec![0, 1].into(),
+ Some(vec![0, 0].into()),
+ vec![Arc::new(Int32Array::from(values)), maps],
+ )
+ .unwrap(),
+ );
+ // Arrow 58.3 accepts this through its safe constructor despite the
+ // declared Int64 field containing an Int32 child.
+ assert!(array.to_data().validate_full().is_err());
+ let batch = RecordBatch::try_new(
+ Arc::new(Schema::new(vec![Field::new(
+ "user_union",
+ array.data_type().clone(),
+ true,
+ )])),
+ vec![array],
+ )
+ .unwrap();
+ for result in [
+ normalize_map_names(std::slice::from_ref(&batch)).map(|_| ()),
+ inline_relation(std::slice::from_ref(&batch), None).map(|_|
()),
+ session
+ .create_dataframe_from_arrow(vec![batch], None)
+ .map(|_| ()),
+ ] {
+ let Err(error) = result else {
+ panic!("declared union types must not reinterpret physical
children");
+ };
+ assert_eq!(
+ error.message(),
+ "Arrow column type does not match its schema field"
+ );
+ }
+ }
+ }
+
+ #[test]
+ fn schema_field_renames_around_maps_use_checked_reconstruction() {
+ let maps: ArrayRef = Arc::new(map(3));
+ let array: ArrayRef = Arc::new(StructArray::new(
+ vec![Field::new("array_maps", maps.data_type().clone(),
true)].into(),
+ vec![Arc::clone(&maps)],
+ None,
+ ));
+ let field = Field::new("schema_maps", maps.data_type().clone(), true)
+ .with_metadata(HashMap::from([("schema_tag".into(),
"preserved".into())]));
+ let batch = RecordBatch::try_new_with_options(
+ Arc::new(Schema::new(vec![Field::new(
+ "user_record",
+ ArrowType::Struct(vec![field].into()),
+ true,
+ )])),
+ vec![Arc::clone(&array)],
+ &RecordBatchOptions::new().with_match_field_names(false),
+ )
+ .unwrap();
+ let normalized =
normalize_map_names(std::slice::from_ref(&batch)).unwrap();
+ let after = normalized[0].column(0);
+ assert_shared_buffers(&array.to_data(), &after.to_data());
+ after.to_data().validate_full().unwrap();
+ let ArrowType::Struct(fields) = after.data_type() else {
+ panic!("expected struct around map");
+ };
+ assert_eq!(fields[0].name(), "schema_maps");
+ assert_eq!(fields[0].metadata().get("schema_tag").unwrap(),
"preserved");
+ let after = after.as_any().downcast_ref::<StructArray>().unwrap();
+ assert_map_values_unchanged(&maps, after.column(0));
+ let LogicalPlan::LocalRelation {
+ data: Some(data), ..
+ } = inline_relation(std::slice::from_ref(&batch), None).unwrap()
+ else {
+ panic!("expected inline relation");
+ };
+ let mut reader = StreamReader::try_new(Cursor::new(data),
None).unwrap();
+ let decoded = reader.next().unwrap().unwrap();
+ decoded.column(0).to_data().validate_full().unwrap();
+ assert_eq!(decoded.schema(), normalized[0].schema());
+ }
+
+ #[test]
+ fn public_zero_row_arrow_input_canonicalizes_maps_without_a_config_rpc() {
+ let session = crate::SparkSession::builder()
+ .remote("sc://127.0.0.1:1")
+ .get_or_create()
+ .unwrap();
+ let dataframe = session
+ .create_dataframe_from_arrow(vec![map_batch(0)], None)
+ .unwrap();
+ let Some(spark_connect_proto::relation::RelType::LocalRelation(local))
=
+ dataframe.to_relation().rel_type
+ else {
+ panic!("expected zero-row inline relation");
+ };
+ assert!(local.schema.is_none());
+ let reader = StreamReader::try_new(Cursor::new(local.data.unwrap()),
None).unwrap();
+ assert_canonical_map(reader.schema().field(0).data_type());
+ let batches = reader.collect::<std::result::Result<Vec<_>,
_>>().unwrap();
+ assert_eq!(batches.len(), 1);
+ assert_eq!(batches[0].num_rows(), 0);
+ }
+
+ #[test]
+ fn
batches_without_maps_preserve_schema_columns_and_zero_column_row_counts() {
+ let batch = batch(0..2);
+ let batches = std::slice::from_ref(&batch);
+ let normalized = normalize_map_names(batches).unwrap();
+ assert!(matches!(normalized, Cow::Borrowed(_)));
+ assert!(std::ptr::eq(normalized.as_ptr(), batches.as_ptr()));
+ let normalized = &normalized[0];
+ assert!(Arc::ptr_eq(&batch.schema(), &normalized.schema()));
+ assert!(Arc::ptr_eq(batch.column(0), normalized.column(0)));
+ let no_columns = RecordBatch::try_new_with_options(
+ Arc::new(Schema::empty()),
+ vec![],
+ &RecordBatchOptions::new().with_row_count(Some(7)),
+ )
+ .unwrap();
+ assert_eq!(normalize_map_names(&[no_columns]).unwrap()[0].num_rows(),
7);
+ }
+
+ #[test]
+ fn
canonical_schema_with_default_map_column_writes_a_canonical_ipc_header() {
+ let original = map_batch(3);
+ let field =
normalize_map_field(&original.schema().fields()[0]).unwrap();
+ let batch = RecordBatch::try_new_with_options(
+ Arc::new(Schema::new(vec![field])),
+ original.columns().to_vec(),
+ &RecordBatchOptions::new().with_match_field_names(false),
+ )
+ .unwrap();
+ let prepared =
normalize_map_names(std::slice::from_ref(&batch)).unwrap();
+ assert!(matches!(prepared, Cow::Borrowed(_)));
+ let LogicalPlan::LocalRelation {
+ data: Some(data), ..
+ } = inline_relation(std::slice::from_ref(&batch), None).unwrap()
+ else {
+ panic!("expected inline relation");
+ };
+ let mut reader = StreamReader::try_new(Cursor::new(data),
None).unwrap();
+ assert_canonical_map(reader.schema().field(0).data_type());
+ let decoded = reader.next().unwrap().unwrap();
+ assert_canonical_map(decoded.schema().field(0).data_type());
+ decoded.column(0).to_data().validate_full().unwrap();
+ assert_map_values_unchanged(original.column(0), decoded.column(0));
+ assert!(reader.next().is_none());
+ }
+
+ #[test]
+ fn maps_are_normalized_inside_every_arrow_container() {
+ fn containers(map: ArrowType) -> Vec<ArrowType> {
+ let field = Arc::new(Field::new("user_item", map.clone(), true));
+ vec![
+ ArrowType::LargeList(Arc::clone(&field)),
+ ArrowType::FixedSizeList(Arc::clone(&field), 2),
+ ArrowType::ListView(Arc::clone(&field)),
+ ArrowType::LargeListView(Arc::clone(&field)),
+ ArrowType::Dictionary(Box::new(ArrowType::Int8),
Box::new(map)),
+ ArrowType::Union(
+ [(5, Arc::clone(&field))].into_iter().collect(),
+ arrow::datatypes::UnionMode::Dense,
+ ),
+ ArrowType::RunEndEncoded(
+ Arc::new(Field::new("user_ends", ArrowType::Int32, false)),
+ field,
+ ),
+ ]
+ }
+ let raw = map(0).data_type().clone();
+ let canonical = ArrowType::Map(
+ Arc::new(Field::new(
+ "entries",
+ ArrowType::Struct(
+ vec![
+ Field::new("key", ArrowType::Int64, false),
+ Field::new("value", ArrowType::Int64, true),
+ ]
+ .into(),
+ ),
+ false,
+ )),
+ false,
+ );
+ for (before, expected) in
containers(raw).into_iter().zip(containers(canonical)) {
+ let batch =
RecordBatch::new_empty(Arc::new(Schema::new(vec![Field::new(
+ "user_column",
+ before,
+ true,
+ )])));
+ let normalized = normalize_map_names(&[batch])
+ .unwrap()
+ .into_owned()
+ .remove(0);
+ assert_eq!(normalized.schema().field(0).data_type(), &expected);
+ normalized.column(0).to_data().validate_full().unwrap();
+ }
+ }
+
+ #[test]
+ fn sliced_fixed_size_lists_keep_map_values_and_parent_validity() {
+ let values: ArrayRef = Arc::new(map(8));
+ let before = FixedSizeListArray::new(
+ Arc::new(Field::new("user_item", values.data_type().clone(),
true)),
+ 2,
+ values,
+ Some(NullBuffer::from(vec![true, false, true, true])),
+ )
+ .slice(1, 2);
+ let normalized = normalize_array(Arc::new(before.clone()));
+ let after = normalized
+ .as_any()
+ .downcast_ref::<FixedSizeListArray>()
+ .unwrap();
+ assert_eq!(after.len(), 2);
+ assert_eq!(after.value_length(), 2);
+ assert_eq!(before.nulls(), after.nulls());
+ assert_eq!(before.value_offset(1), after.value_offset(1));
+ assert_map_values_unchanged(before.values(), after.values());
+ }
+
+ #[test]
+ fn sliced_dictionaries_keep_keys_and_unsliced_map_values() {
+ let before = DictionaryArray::<Int8Type>::try_new(
+ Int8Array::from(vec![Some(3), Some(0), None, Some(2), Some(3)]),
+ Arc::new(map(4)),
+ )
+ .unwrap()
+ .slice(1, 3);
+ let normalized = normalize_array(Arc::new(before.clone()));
+ let after = normalized
+ .as_any()
+ .downcast_ref::<DictionaryArray<Int8Type>>()
+ .unwrap();
+ assert_eq!(before.keys().to_data(), after.keys().to_data());
+ assert_eq!(
+ after.keys_iter().collect::<Vec<_>>(),
+ vec![Some(0), None, Some(2)]
+ );
+ assert_eq!(after.values().len(), 4);
+ assert_map_values_unchanged(before.values(), after.values());
+ }
+
+ #[test]
+ fn sliced_dense_unions_keep_nonsequential_map_offsets_and_type_ids() {
+ let values: ArrayRef = Arc::new(map(4));
+ let before = UnionArray::try_new(
+ [(
+ 5,
+ Arc::new(Field::new("user_maps", values.data_type().clone(),
true)),
+ )]
+ .into_iter()
+ .collect(),
+ vec![5, 5, 5, 5, 5].into(),
+ Some(vec![3, 0, 2, 1, 3].into()),
+ vec![values],
+ )
+ .unwrap()
+ .slice(1, 3);
+ let normalized = normalize_array(Arc::new(before.clone()));
+ let after = normalized.as_any().downcast_ref::<UnionArray>().unwrap();
+ assert_eq!(before.type_ids(), after.type_ids());
+ assert_eq!(after.type_ids().as_ref(), &[5, 5, 5]);
+ assert_eq!(before.offsets(), after.offsets());
+ assert_eq!(
+ (0..after.len())
+ .map(|i| after.value_offset(i))
+ .collect::<Vec<_>>(),
+ vec![0, 2, 1]
+ );
+ assert_map_values_unchanged(before.child(5), after.child(5));
+ }
+
+ #[test]
+ fn sliced_run_arrays_keep_physical_map_runs_and_logical_offsets() {
+ let before = RunArray::<Int32Type>::try_new(&Int32Array::from(vec![2,
5, 6, 9]), &map(4))
+ .unwrap()
+ .slice(1, 6);
+ let normalized = normalize_array(Arc::new(before.clone()));
+ let after = normalized
+ .as_any()
+ .downcast_ref::<RunArray<Int32Type>>()
+ .unwrap();
+ assert_eq!(before.offset(), after.offset());
+ assert_eq!(after.offset(), 1);
+ assert_eq!(before.run_ends().values(), after.run_ends().values());
+ assert_eq!(after.run_ends().values(), &[2, 5, 6, 9]);
+ assert_eq!(
+ (0..after.len())
+ .map(|i| after.get_physical_index(i))
+ .collect::<Vec<_>>(),
+ vec![0, 1, 1, 1, 2, 3]
+ );
+ assert_map_values_unchanged(before.values(), after.values());
+ }
+
fn chunks(batch: &RecordBatch, chunk_rows: usize, chunk_bytes: usize) ->
Vec<Vec<RecordBatch>> {
let options = Options {
size_limit: usize::MAX,
@@ -282,7 +1122,8 @@ mod tests {
upload_batch_bytes: usize::MAX,
};
let mut chunks = Vec::new();
- serialize_chunks(std::slice::from_ref(batch), &options, &mut |chunk| {
+ let normalized =
normalize_map_names(std::slice::from_ref(batch)).unwrap();
+ serialize_chunks(&normalized, &options, &mut |chunk| {
chunks.push(chunk);
Ok(())
})
diff --git a/crates/spark-connect/src/session.rs
b/crates/spark-connect/src/session.rs
index cf475324..28832f64 100644
--- a/crates/spark-connect/src/session.rs
+++ b/crates/spark-connect/src/session.rs
@@ -643,6 +643,8 @@ impl SparkSession {
/// Internal binding bridge for Arrow IPC already validated and encoded
from
/// local rows or columns. Retains the row path's transport policy:
inline, unless the
/// IPC is large enough to consult the server's local-relation settings.
+ /// Large IPC is decoded and its map field names normalized before
re-encoding;
+ /// smaller IPC is passed through unchanged.
#[doc(hidden)]
pub fn create_dataframe_from_inline_arrow_ipc(
&self,
@@ -673,6 +675,10 @@ impl SparkSession {
/// `schema` optionally supplies the Spark schema applied by the server.
With
/// `None`, Spark infers it from Arrow. All batches must share the same
Arrow
/// schema; IPC serialization is still required for transport.
+ /// Map child fields are normalized to Spark's `entries`, `key`, and
`value`
+ /// names, including nested maps, without copying their data buffers.
+ /// Columns rebuilt for map names are validated against the schema before
any
+ /// RPC, rejecting incompatible child types or union ordering.
/// Non-empty inputs read the server's local-relation configuration. Large
inputs
/// are uploaded as cache artifacts and referenced by hash in the
resulting plan.
pub fn create_dataframe_from_arrow(
diff --git a/crates/spark-connect/tests/arrow_map_input.rs
b/crates/spark-connect/tests/arrow_map_input.rs
new file mode 100644
index 00000000..a077c9a5
--- /dev/null
+++ b/crates/spark-connect/tests/arrow_map_input.rs
@@ -0,0 +1,197 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Arrow Rust's default map names are accepted by Spark through both local
+//! relation input paths. Live checks run only when SPARK_REMOTE is set.
+
+use arrow::array::builder::{Int64Builder, StringBuilder};
+use arrow::array::{ArrayRef, Int64Array, ListArray, MapBuilder, StructArray};
+use arrow::buffer::OffsetBuffer;
+use arrow::datatypes::{Field, Schema};
+use arrow::ipc::reader::StreamReader;
+use arrow::record_batch::RecordBatch;
+use spark_connect::row::Value;
+use spark_connect::SparkSession;
+use spark_connect_proto::relation::RelType;
+use std::io::Cursor;
+use std::sync::Arc;
+
+struct Session(SparkSession);
+
+impl Session {
+ fn open() -> Option<Self> {
+ let remote = std::env::var("SPARK_REMOTE").ok()?;
+ Some(Self(
+ SparkSession::builder()
+ .remote(&remote)
+ .get_or_create()
+ .unwrap(),
+ ))
+ }
+}
+
+impl Drop for Session {
+ fn drop(&mut self) {
+ let _ = self.0.stop();
+ }
+}
+
+fn input() -> RecordBatch {
+ let mut builder = MapBuilder::new(None, StringBuilder::new(),
Int64Builder::new());
+ builder.keys().append_value("alpha");
+ builder.values().append_value(1);
+ builder.keys().append_value("beta");
+ builder.values().append_null();
+ builder.append(true).unwrap();
+ builder.append(false).unwrap();
+ builder.append(true).unwrap();
+ builder.keys().append_value("π😀");
+ builder.values().append_value(3);
+ builder.append(true).unwrap();
+ let map: ArrayRef = Arc::new(builder.finish());
+ let list: ArrayRef = Arc::new(ListArray::new(
+ Arc::new(Field::new("user_items", map.data_type().clone(), true)),
+ OffsetBuffer::new(vec![0, 1, 2, 3, 4].into()),
+ Arc::clone(&map),
+ None,
+ ));
+ let nested: ArrayRef = Arc::new(StructArray::new(
+ vec![Field::new("maps", list.data_type().clone(), true)].into(),
+ vec![list],
+ None,
+ ));
+ let schema = Schema::new(vec![
+ Field::new("id", arrow::datatypes::DataType::Int64, false),
+ Field::new("mapping", map.data_type().clone(), true),
+ Field::new("nested", nested.data_type().clone(), true),
+ ]);
+ RecordBatch::try_new(
+ Arc::new(schema),
+ vec![Arc::new(Int64Array::from(vec![0, 1, 2, 3])), map, nested],
+ )
+ .unwrap()
+}
+
+fn expected_maps() -> Vec<Value> {
+ vec![
+ Value::Map(vec![
+ (Value::String("alpha".into()), Value::Long(1)),
+ (Value::String("beta".into()), Value::Null),
+ ]),
+ Value::Null,
+ Value::Map(vec![]),
+ Value::Map(vec![(Value::String("π😀".into()), Value::Long(3))]),
+ ]
+}
+
+#[test]
+fn default_arrow_maps_round_trip_inline_and_chunked() {
+ let Some(session) = Session::open() else {
+ return;
+ };
+ // Older servers can expose a unit-suffixed default (such as "3GB"). Use
+ // explicit bytes to keep this test focused on Arrow map interoperability.
+ session
+ .0
+ .conf()
+ .set("spark.sql.session.localRelationSizeLimit", "16777216")
+ .unwrap();
+ for (threshold, chunked) in [("1073741824", false), ("0", true)] {
+ session
+ .0
+ .conf()
+ .set("spark.sql.session.localRelationCacheThreshold", threshold)
+ .unwrap();
+ session
+ .0
+ .conf()
+ .set("spark.sql.session.localRelationChunkSizeRows", "2")
+ .unwrap();
+ session
+ .0
+ .conf()
+ .set("spark.sql.session.localRelationChunkSizeBytes", "1")
+ .unwrap();
+ let input = input();
+ let dataframe = session
+ .0
+ .create_dataframe_from_arrow(vec![input.slice(0, 2),
input.slice(2, 2)], None)
+ .unwrap();
+ let relation = dataframe.to_relation();
+ match (chunked, relation.rel_type) {
+ (true, Some(RelType::ChunkedCachedLocalRelation(local))) => {
+ assert!(
+ local.data_hashes.len() >= 2,
+ "expected multiple IPC artifacts"
+ );
+ }
+ (false, Some(RelType::LocalRelation(_))) => {}
+ _ => panic!("cache threshold did not select the intended input
path"),
+ }
+ assert_eq!(dataframe.schema().unwrap().simple_string(),
+
"struct<id:bigint,mapping:map<string,bigint>,nested:struct<maps:array<map<string,bigint>>>>");
+ let mut rows = dataframe.collect().unwrap();
+ rows.sort_by_key(|row| match row.get(0) {
+ Some(Value::Long(id)) => *id,
+ value => panic!("unexpected id: {value:?}"),
+ });
+ let expected = expected_maps();
+ assert_eq!(rows.len(), expected.len());
+ for (index, (row, expected)) in rows.iter().zip(expected).enumerate() {
+ assert_eq!(row.get(0), Some(&Value::Long(index as i64)));
+ assert_eq!(row.get(1), Some(&expected));
+ let Some(Value::Struct(nested)) = row.get(2) else {
+ panic!("expected nested struct");
+ };
+ assert_eq!(nested.fields(), &["maps".to_owned()]);
+ assert_eq!(nested.get(0), Some(&Value::List(vec![expected])));
+ }
+ }
+ session.0.stop().unwrap();
+}
+
+#[test]
+fn zero_row_arrow_maps_keep_a_canonical_header_and_analyze_on_spark() {
+ let Some(session) = Session::open() else {
+ return;
+ };
+ let dataframe = session
+ .0
+ .create_dataframe_from_arrow(vec![input().slice(0, 0)], None)
+ .unwrap();
+ let Some(RelType::LocalRelation(local)) = dataframe.to_relation().rel_type
else {
+ panic!("zero rows must use an inline relation");
+ };
+ let reader = StreamReader::try_new(Cursor::new(local.data.unwrap()),
None).unwrap();
+ let arrow::datatypes::DataType::Map(entries, _) =
reader.schema().field(1).data_type().clone()
+ else {
+ panic!("expected map schema");
+ };
+ let arrow::datatypes::DataType::Struct(fields) = entries.data_type() else {
+ panic!("expected map entry struct");
+ };
+ // Successful server analysis with no rows is only smoke evidence: Spark
+ // need not access a map vector. These canonical IPC-header assertions are
+ // the regression proof for the zero-row path.
+ assert_eq!(entries.name(), "entries");
+ assert_eq!(fields[0].name(), "key");
+ assert_eq!(fields[1].name(), "value");
+ assert_eq!(dataframe.schema().unwrap().simple_string(),
+
"struct<id:bigint,mapping:map<string,bigint>,nested:struct<maps:array<map<string,bigint>>>>");
+ assert!(dataframe.collect().unwrap().is_empty());
+ session.0.stop().unwrap();
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]