This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 091e14b [SPARK-60666][CONNECT] Dispatch Arrow decoding once per typed
column
091e14b is described below
commit 091e14b504295fda2baee62553940488836469f9
Author: DB Tsai <[email protected]>
AuthorDate: Fri Nov 6 17:12:26 2026 +0900
[SPARK-60666][CONNECT] Dispatch Arrow decoding once per typed column
### What changes were proposed in this pull request?
The decoding of Arrow results into rows now resolves the type of each
column, and of each nested child, once per batch: `ArrowValueLayout` holds the
downcast array of a column (an enum of the supported Arrow types), and
`value_at` reads a value of a row from it. Before, every cell went through the
chain of downcasts again, and sliced list and map children again.
Rows are still built in row-major order, errors stay lazy (a column of an
unsupported type fails at its first non-null value, in the same order as
before), list and map children use the absolute offsets of their unsliced child
arrays, and sliced, nested and null values decode as before.
### Why are the changes needed?
[SPARK-60666](https://issues.apache.org/jira/browse/SPARK-60666): the
per-cell downcasts were the main cost of decoding results. Measured with the
Python client on macOS, against a local Spark 4.3.0 server, as the client CPU
time of `collect()` (median of 5, after a warm-up, the same in two rounds):
| Result | Master | This change |
| --- | ---: | ---: |
| 400,000 rows of `bigint`, `bigint`, `string`, `double` | 0.284 s | 0.147
s (-48%) |
| 200,000 rows of `array<bigint>`,
`struct<x:bigint,y:double,z:array<bigint>>`, `map<string,bigint>` | 0.685 s |
0.174 s (-75%) |
The two builds collect the same rows, compared by a hash of their values,
for flat, nested and null values and for decimal, date, timestamp, binary,
boolean, tinyint, smallint, float, day-time interval and variant columns.
### Does this PR introduce _any_ user-facing change?
No, other than speed.
### How was this patch tested?
- New unit tests decode sliced and nested arrays, nulls, the exact values
of each scalar type and the order of decoding errors; the mixed nullable test
also sends empty IPC batches, and a duplicated variant field checks the
first-match and lazy-error behavior.
- Against the current master, merged into this branch, and a local Spark
4.3.0 server: the lib and every integration test of `apache-spark-connect` and
`apache-spark-connect-core`, e2e included (779 passed), the build of all tests
with `wasm-udf`, the offline Python suites (733 passed, 9 skipped), the live
drop-in and Arrow suites (77 passed) and the live cases (10 passed). `cargo fmt
--all --check` passes.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-6); the merge with master and the
measurement by Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #112 from dbtsai/codex/rsi-typed-arrow-dispatch.
Lead-authored-by: DB Tsai <[email protected]>
Co-authored-by: DB Tsai <[email protected]>
Co-authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/spark-connect/src/dataframe.rs | 937 ++++++++++++++++++++++++----------
1 file changed, 657 insertions(+), 280 deletions(-)
diff --git a/crates/spark-connect/src/dataframe.rs
b/crates/spark-connect/src/dataframe.rs
index 1004999..0013770 100644
--- a/crates/spark-connect/src/dataframe.rs
+++ b/crates/spark-connect/src/dataframe.rs
@@ -2019,15 +2019,14 @@ pub(crate) fn decode_arrow_batch(
let layouts = record_batch
.columns()
.iter()
- .map(|column| ArrowValueLayout::new(column.data_type()))
+ .map(|column| ArrowValueLayout::new(column.as_ref()))
.collect::<Vec<_>>();
for row_idx in 0..num_rows {
let mut values = Vec::with_capacity(record_batch.num_columns());
- for (column, layout) in
record_batch.columns().iter().zip(&layouts) {
- let value = arrow_value_at_with_layout(column.as_ref(),
row_idx, layout)?;
- values.push(value);
+ for layout in &layouts {
+ values.push(layout.value_at(row_idx)?);
}
rows.push(Row::with_shared_fields(Arc::clone(&field_names),
values));
@@ -2150,293 +2149,349 @@ fn record_batches_to_polars(
.map_err(|e| SparkError::connect_msg(format!("Failed to create Polars
DataFrame: {e}")))
}
-/// Immutable decoding metadata prepared once for each column in a record
batch.
-/// Container layouts follow their children so struct names are also shared
inside
-/// lists and maps, without retaining the Arrow arrays after decoding.
-enum ArrowValueLayout {
- Scalar,
- List(Box<Self>),
+/// A borrowed column decoder, resolved once per record batch. Returned Values
+/// own their contents; neither rows nor nested values retain the Arrow arrays.
+struct ArrowValueLayout<'a> {
+ array: &'a dyn arrow::array::Array,
+ decoder: ArrowValueDecoder<'a>,
+}
+
+enum ArrowValueDecoder<'a> {
+ Null,
+ Boolean(&'a arrow::array::BooleanArray),
+ Int8(&'a arrow::array::Int8Array),
+ Int16(&'a arrow::array::Int16Array),
+ Int32(&'a arrow::array::Int32Array),
+ Int64(&'a arrow::array::Int64Array),
+ Float32(&'a arrow::array::Float32Array),
+ Float64(&'a arrow::array::Float64Array),
+ String(&'a arrow::array::StringArray),
+ Binary(&'a arrow::array::BinaryArray),
+ Date32(&'a arrow::array::Date32Array),
+ TimestampMicrosecond(&'a arrow::array::TimestampMicrosecondArray),
+ UInt8(&'a arrow::array::UInt8Array),
+ UInt16(&'a arrow::array::UInt16Array),
+ UInt32(&'a arrow::array::UInt32Array),
+ UInt64(&'a arrow::array::UInt64Array),
+ Decimal128(&'a arrow::array::Decimal128Array),
+ LargeString(&'a arrow::array::LargeStringArray),
+ LargeBinary(&'a arrow::array::LargeBinaryArray),
+ StringView(&'a arrow::array::StringViewArray),
+ BinaryView(&'a arrow::array::BinaryViewArray),
+ TimestampSecond(&'a arrow::array::TimestampSecondArray),
+ TimestampMillisecond(&'a arrow::array::TimestampMillisecondArray),
+ TimestampNanosecond(&'a arrow::array::TimestampNanosecondArray),
+ Date64(&'a arrow::array::Date64Array),
+ DurationMicrosecond(&'a arrow::array::DurationMicrosecondArray),
+ Decimal256(&'a arrow::array::Decimal256Array),
+ FixedSizeBinary(&'a arrow::array::FixedSizeBinaryArray),
+ Time64Microsecond(&'a arrow::array::Time64MicrosecondArray),
+ Time64Nanosecond(&'a arrow::array::Time64NanosecondArray),
+ Time32Millisecond(&'a arrow::array::Time32MillisecondArray),
+ Time32Second(&'a arrow::array::Time32SecondArray),
+ IntervalYearMonth(&'a arrow::array::IntervalYearMonthArray),
+ IntervalDayTime(&'a arrow::array::IntervalDayTimeArray),
+ IntervalMonthDayNano(&'a arrow::array::IntervalMonthDayNanoArray),
+ List {
+ array: &'a arrow::array::ListArray,
+ child: Box<ArrowValueLayout<'a>>,
+ },
Struct {
names: Arc<[String]>,
- children: Vec<Self>,
- is_variant: bool,
+ children: Vec<ArrowValueLayout<'a>>,
+ variant_fields: Option<(Option<usize>, Option<usize>)>,
+ },
+ Map {
+ array: &'a arrow::array::MapArray,
+ children: Box<(ArrowValueLayout<'a>, ArrowValueLayout<'a>)>,
},
- Map(Box<(Self, Self)>),
+ Unsupported,
}
-impl ArrowValueLayout {
- fn new(data_type: &arrow::datatypes::DataType) -> Self {
- use arrow::datatypes::DataType;
- match data_type {
- DataType::List(field) =>
Self::List(Box::new(Self::new(field.data_type()))),
- DataType::Struct(fields) => Self::Struct {
+impl<'a> ArrowValueLayout<'a> {
+ fn new(array: &'a dyn arrow::array::Array) -> Self {
+ use arrow::array::*;
+
+ // Resolution never fails: null parents and empty containers must not
+ // reject unsupported children that are never read.
+ let any = array.as_any();
+ let decoder = if any.is::<NullArray>() {
+ ArrowValueDecoder::Null
+ } else if let Some(array) = any.downcast_ref::<BooleanArray>() {
+ ArrowValueDecoder::Boolean(array)
+ } else if let Some(array) = any.downcast_ref::<Int8Array>() {
+ ArrowValueDecoder::Int8(array)
+ } else if let Some(array) = any.downcast_ref::<Int16Array>() {
+ ArrowValueDecoder::Int16(array)
+ } else if let Some(array) = any.downcast_ref::<Int32Array>() {
+ ArrowValueDecoder::Int32(array)
+ } else if let Some(array) = any.downcast_ref::<Int64Array>() {
+ ArrowValueDecoder::Int64(array)
+ } else if let Some(array) = any.downcast_ref::<Float32Array>() {
+ ArrowValueDecoder::Float32(array)
+ } else if let Some(array) = any.downcast_ref::<Float64Array>() {
+ ArrowValueDecoder::Float64(array)
+ } else if let Some(array) = any.downcast_ref::<StringArray>() {
+ ArrowValueDecoder::String(array)
+ } else if let Some(array) = any.downcast_ref::<BinaryArray>() {
+ ArrowValueDecoder::Binary(array)
+ } else if let Some(array) = any.downcast_ref::<Date32Array>() {
+ ArrowValueDecoder::Date32(array)
+ } else if let Some(array) =
any.downcast_ref::<TimestampMicrosecondArray>() {
+ ArrowValueDecoder::TimestampMicrosecond(array)
+ } else if let Some(array) = any.downcast_ref::<UInt8Array>() {
+ ArrowValueDecoder::UInt8(array)
+ } else if let Some(array) = any.downcast_ref::<UInt16Array>() {
+ ArrowValueDecoder::UInt16(array)
+ } else if let Some(array) = any.downcast_ref::<UInt32Array>() {
+ ArrowValueDecoder::UInt32(array)
+ } else if let Some(array) = any.downcast_ref::<UInt64Array>() {
+ ArrowValueDecoder::UInt64(array)
+ } else if let Some(array) = any.downcast_ref::<Decimal128Array>() {
+ ArrowValueDecoder::Decimal128(array)
+ } else if let Some(array) = any.downcast_ref::<LargeStringArray>() {
+ ArrowValueDecoder::LargeString(array)
+ } else if let Some(array) = any.downcast_ref::<LargeBinaryArray>() {
+ ArrowValueDecoder::LargeBinary(array)
+ } else if let Some(array) = any.downcast_ref::<StringViewArray>() {
+ ArrowValueDecoder::StringView(array)
+ } else if let Some(array) = any.downcast_ref::<BinaryViewArray>() {
+ ArrowValueDecoder::BinaryView(array)
+ } else if let Some(array) = any.downcast_ref::<TimestampSecondArray>()
{
+ ArrowValueDecoder::TimestampSecond(array)
+ } else if let Some(array) =
any.downcast_ref::<TimestampMillisecondArray>() {
+ ArrowValueDecoder::TimestampMillisecond(array)
+ } else if let Some(array) =
any.downcast_ref::<TimestampNanosecondArray>() {
+ ArrowValueDecoder::TimestampNanosecond(array)
+ } else if let Some(array) = any.downcast_ref::<Date64Array>() {
+ ArrowValueDecoder::Date64(array)
+ } else if let Some(array) =
any.downcast_ref::<DurationMicrosecondArray>() {
+ ArrowValueDecoder::DurationMicrosecond(array)
+ } else if let Some(array) = any.downcast_ref::<Decimal256Array>() {
+ ArrowValueDecoder::Decimal256(array)
+ } else if let Some(array) = any.downcast_ref::<FixedSizeBinaryArray>()
{
+ ArrowValueDecoder::FixedSizeBinary(array)
+ } else if let Some(array) =
any.downcast_ref::<Time64MicrosecondArray>() {
+ ArrowValueDecoder::Time64Microsecond(array)
+ } else if let Some(array) =
any.downcast_ref::<Time64NanosecondArray>() {
+ ArrowValueDecoder::Time64Nanosecond(array)
+ } else if let Some(array) =
any.downcast_ref::<Time32MillisecondArray>() {
+ ArrowValueDecoder::Time32Millisecond(array)
+ } else if let Some(array) = any.downcast_ref::<Time32SecondArray>() {
+ ArrowValueDecoder::Time32Second(array)
+ } else if let Some(array) =
any.downcast_ref::<IntervalYearMonthArray>() {
+ ArrowValueDecoder::IntervalYearMonth(array)
+ } else if let Some(array) = any.downcast_ref::<IntervalDayTimeArray>()
{
+ ArrowValueDecoder::IntervalDayTime(array)
+ } else if let Some(array) =
any.downcast_ref::<IntervalMonthDayNanoArray>() {
+ ArrowValueDecoder::IntervalMonthDayNano(array)
+ } else if let Some(array) = any.downcast_ref::<ListArray>() {
+ ArrowValueDecoder::List {
+ array,
+ child: Box::new(Self::new(array.values().as_ref())),
+ }
+ } else if let Some(array) = any.downcast_ref::<StructArray>() {
+ let fields = array.fields();
+ let is_variant = fields.iter().any(|field| {
+ field
+ .metadata()
+ .get("variant")
+ .is_some_and(|value| value == "true")
+ });
+ ArrowValueDecoder::Struct {
names: fields.iter().map(|field|
field.name().clone()).collect(),
- children: fields
+ children: array
+ .columns()
.iter()
- .map(|field| Self::new(field.data_type()))
+ .map(|child| Self::new(child.as_ref()))
.collect(),
- is_variant: fields.iter().any(|field| {
- field
- .metadata()
- .get("variant")
- .is_some_and(|value| value == "true")
+ variant_fields: is_variant.then(|| {
+ (
+ fields.iter().position(|field| field.name() ==
"value"),
+ fields.iter().position(|field| field.name() ==
"metadata"),
+ )
}),
- },
- DataType::Map(entries, _) => match entries.data_type() {
- DataType::Struct(fields) => Self::Map(Box::new((
- Self::new(fields[0].data_type()),
- Self::new(fields[1].data_type()),
- ))),
- _ => unreachable!("Arrow map entries must be a struct"),
- },
- _ => Self::Scalar,
- }
- }
-}
-
-#[cfg(test)]
-fn arrow_value_at(array: &dyn arrow::array::Array, index: usize) ->
Result<Value> {
- arrow_value_at_with_layout(array, index,
&ArrowValueLayout::new(array.data_type()))
-}
-
-fn arrow_value_at_with_layout(
- array: &dyn arrow::array::Array,
- index: usize,
- value_layout: &ArrowValueLayout,
-) -> Result<Value> {
- use arrow::array::*;
-
- if array.is_null(index) {
- return Ok(Value::Null);
- }
-
- // A NullType column arrives as an all-null NullArray; every element is
Null.
- if array.as_any().downcast_ref::<NullArray>().is_some() {
- return Ok(Value::Null);
+ }
+ } else if let Some(array) = any.downcast_ref::<MapArray>() {
+ ArrowValueDecoder::Map {
+ array,
+ children: Box::new((
+ Self::new(array.keys().as_ref()),
+ Self::new(array.values().as_ref()),
+ )),
+ }
+ } else {
+ ArrowValueDecoder::Unsupported
+ };
+ Self { array, decoder }
}
- // Try each array type
- if let Some(arr) = array.as_any().downcast_ref::<BooleanArray>() {
- return Ok(Value::Bool(arr.value(index)));
- }
- if let Some(arr) = array.as_any().downcast_ref::<Int8Array>() {
- return Ok(Value::Byte(arr.value(index)));
- }
- if let Some(arr) = array.as_any().downcast_ref::<Int16Array>() {
- return Ok(Value::Short(arr.value(index)));
- }
- if let Some(arr) = array.as_any().downcast_ref::<Int32Array>() {
- return Ok(Value::Integer(arr.value(index)));
- }
- if let Some(arr) = array.as_any().downcast_ref::<Int64Array>() {
- return Ok(Value::Long(arr.value(index)));
- }
- if let Some(arr) = array.as_any().downcast_ref::<Float32Array>() {
- return Ok(Value::Float(arr.value(index)));
- }
- if let Some(arr) = array.as_any().downcast_ref::<Float64Array>() {
- return Ok(Value::Double(arr.value(index)));
- }
- if let Some(arr) = array.as_any().downcast_ref::<StringArray>() {
- return Ok(Value::String(arr.value(index).to_string()));
- }
- if let Some(arr) = array.as_any().downcast_ref::<BinaryArray>() {
- return Ok(Value::Binary(arr.value(index).to_vec()));
- }
- if let Some(arr) = array.as_any().downcast_ref::<Date32Array>() {
- return Ok(Value::Date(arr.value(index)));
- }
- if let Some(arr) =
array.as_any().downcast_ref::<TimestampMicrosecondArray>() {
- return Ok(Value::Timestamp(arr.value(index)));
- }
- // Unsigned integers.
- if let Some(arr) = array.as_any().downcast_ref::<UInt8Array>() {
- return Ok(Value::Short(arr.value(index) as i16));
- }
- if let Some(arr) = array.as_any().downcast_ref::<UInt16Array>() {
- return Ok(Value::Integer(arr.value(index) as i32));
- }
- if let Some(arr) = array.as_any().downcast_ref::<UInt32Array>() {
- return Ok(Value::Long(arr.value(index) as i64));
- }
- if let Some(arr) = array.as_any().downcast_ref::<UInt64Array>() {
- let val = arr.value(index);
- let i64_val = i64::try_from(val).map_err(|_| {
- SparkError::connect_msg(format!("UInt64 value {} exceeds i64
range", val))
- })?;
- return Ok(Value::Long(i64_val));
- }
- // Decimal128 -> Value::Decimal (exact, not lossy f64), preserving
precision/scale.
- if let Some(arr) = array.as_any().downcast_ref::<Decimal128Array>() {
- let scale = arr.scale() as i32;
- return Ok(Value::Decimal {
- value: i128_to_decimal_string(arr.value(index), scale),
- precision: Some(arr.precision() as i32),
- scale: Some(scale),
- });
- }
- // Large / view string & binary variants.
- if let Some(arr) = array.as_any().downcast_ref::<LargeStringArray>() {
- return Ok(Value::String(arr.value(index).to_string()));
- }
- if let Some(arr) = array.as_any().downcast_ref::<LargeBinaryArray>() {
- return Ok(Value::Binary(arr.value(index).to_vec()));
- }
- if let Some(arr) = array.as_any().downcast_ref::<StringViewArray>() {
- return Ok(Value::String(arr.value(index).to_string()));
- }
- if let Some(arr) = array.as_any().downcast_ref::<BinaryViewArray>() {
- return Ok(Value::Binary(arr.value(index).to_vec()));
- }
- // Other timestamp units, normalized to microseconds.
- if let Some(arr) = array.as_any().downcast_ref::<TimestampSecondArray>() {
- return Ok(Value::Timestamp(arr.value(index) * 1_000_000));
- }
- if let Some(arr) =
array.as_any().downcast_ref::<TimestampMillisecondArray>() {
- return Ok(Value::Timestamp(arr.value(index) * 1_000));
- }
- if let Some(arr) =
array.as_any().downcast_ref::<TimestampNanosecondArray>() {
- return Ok(Value::Timestamp(arr.value(index) / 1_000));
- }
- if let Some(arr) = array.as_any().downcast_ref::<Date64Array>() {
- return Ok(Value::Date((arr.value(index) / 86_400_000) as i32));
- }
- // Spark DayTimeIntervalType uses Duration(Microsecond), not an Arrow
- // Interval physical type. Keep signed micros exactly, including i64::MIN.
- // Other duration units remain unsupported rather than being rounded.
- if let Some(arr) =
array.as_any().downcast_ref::<DurationMicrosecondArray>() {
- return Ok(Value::DayTimeInterval(arr.value(index)));
- }
- // Nested: list, struct, map (recurse).
- if let Some(arr) = array.as_any().downcast_ref::<ListArray>() {
- let ArrowValueLayout::List(child_layout) = value_layout else {
- unreachable!("layout is prepared from the array's data type");
- };
- let child = arr.value(index);
- let mut items = Vec::with_capacity(child.len());
- for i in 0..child.len() {
- items.push(arrow_value_at_with_layout(child.as_ref(), i,
child_layout)?);
+ fn value_at(&self, index: usize) -> Result<Value> {
+ if self.array.is_null(index) {
+ return Ok(Value::Null);
}
- return Ok(Value::List(items));
- }
- if let Some(arr) = array.as_any().downcast_ref::<StructArray>() {
- let ArrowValueLayout::Struct {
- names,
- children,
- is_variant,
- } = value_layout
- else {
- unreachable!("layout is prepared from the array's data type");
- };
- // A VARIANT column arrives as struct<value: binary, metadata: binary>
where the
- // `metadata` field carries arrow metadata {"variant": "true"}.
Recognize it and
- // return a Value::Variant (raw bytes) so it materializes as a
VariantVal (matching
- // pyspark) rather than a plain {value, metadata} struct/dict.
- if *is_variant {
- let bin_field = |name: &str| -> Result<Vec<u8>> {
- match arr.fields().iter().position(|field| field.name() ==
name) {
- Some(col_idx) => match arrow_value_at_with_layout(
- arr.column(col_idx).as_ref(),
- index,
- &children[col_idx],
- )? {
- Value::Binary(b) => Ok(b),
- Value::Null => Ok(vec![]),
- _ => Err(SparkError::connect_msg("variant field is not
binary")),
- },
- None => Ok(vec![]),
+ match &self.decoder {
+ // NullType may have no null buffer; every element is still Null.
+ ArrowValueDecoder::Null => Ok(Value::Null),
+ ArrowValueDecoder::Boolean(arr) =>
Ok(Value::Bool(arr.value(index))),
+ ArrowValueDecoder::Int8(arr) => Ok(Value::Byte(arr.value(index))),
+ ArrowValueDecoder::Int16(arr) =>
Ok(Value::Short(arr.value(index))),
+ ArrowValueDecoder::Int32(arr) =>
Ok(Value::Integer(arr.value(index))),
+ ArrowValueDecoder::Int64(arr) => Ok(Value::Long(arr.value(index))),
+ ArrowValueDecoder::Float32(arr) =>
Ok(Value::Float(arr.value(index))),
+ ArrowValueDecoder::Float64(arr) =>
Ok(Value::Double(arr.value(index))),
+ ArrowValueDecoder::String(arr) =>
Ok(Value::String(arr.value(index).to_string())),
+ ArrowValueDecoder::Binary(arr) =>
Ok(Value::Binary(arr.value(index).to_vec())),
+ ArrowValueDecoder::Date32(arr) =>
Ok(Value::Date(arr.value(index))),
+ ArrowValueDecoder::TimestampMicrosecond(arr) =>
Ok(Value::Timestamp(arr.value(index))),
+ ArrowValueDecoder::UInt8(arr) => Ok(Value::Short(arr.value(index)
as i16)),
+ ArrowValueDecoder::UInt16(arr) =>
Ok(Value::Integer(arr.value(index) as i32)),
+ ArrowValueDecoder::UInt32(arr) => Ok(Value::Long(arr.value(index)
as i64)),
+ ArrowValueDecoder::UInt64(arr) => {
+ let val = arr.value(index);
+ let i64_val = i64::try_from(val).map_err(|_| {
+ SparkError::connect_msg(format!("UInt64 value {} exceeds
i64 range", val))
+ })?;
+ Ok(Value::Long(i64_val))
+ }
+ // Decimal128 stays exact, preserving precision/scale rather than
using lossy f64.
+ ArrowValueDecoder::Decimal128(arr) => {
+ let scale = arr.scale() as i32;
+ Ok(Value::Decimal {
+ value: i128_to_decimal_string(arr.value(index), scale),
+ precision: Some(arr.precision() as i32),
+ scale: Some(scale),
+ })
+ }
+ ArrowValueDecoder::LargeString(arr) =>
Ok(Value::String(arr.value(index).to_string())),
+ ArrowValueDecoder::LargeBinary(arr) =>
Ok(Value::Binary(arr.value(index).to_vec())),
+ ArrowValueDecoder::StringView(arr) =>
Ok(Value::String(arr.value(index).to_string())),
+ ArrowValueDecoder::BinaryView(arr) =>
Ok(Value::Binary(arr.value(index).to_vec())),
+ ArrowValueDecoder::TimestampSecond(arr) => {
+ Ok(Value::Timestamp(arr.value(index) * 1_000_000))
+ }
+ ArrowValueDecoder::TimestampMillisecond(arr) => {
+ Ok(Value::Timestamp(arr.value(index) * 1_000))
+ }
+ ArrowValueDecoder::TimestampNanosecond(arr) => {
+ Ok(Value::Timestamp(arr.value(index) / 1_000))
+ }
+ ArrowValueDecoder::Date64(arr) => {
+ Ok(Value::Date((arr.value(index) / 86_400_000) as i32))
+ }
+ ArrowValueDecoder::DurationMicrosecond(arr) => {
+ Ok(Value::DayTimeInterval(arr.value(index)))
+ }
+ // Decimal256 also preserves exact digits; Arrow formats with the
scale.
+ ArrowValueDecoder::Decimal256(arr) => Ok(Value::Decimal {
+ value: arr.value_as_string(index),
+ precision: Some(arr.precision() as i32),
+ scale: Some(arr.scale() as i32),
+ }),
+ ArrowValueDecoder::FixedSizeBinary(arr) =>
Ok(Value::Binary(arr.value(index).to_vec())),
+ // TimeType has no dedicated Value variant, so render as an ISO
time string.
+ ArrowValueDecoder::Time64Microsecond(arr) => {
+ Ok(Value::String(micros_to_time_string(arr.value(index))))
+ }
+ ArrowValueDecoder::Time64Nanosecond(arr) =>
Ok(Value::String(micros_to_time_string(
+ arr.value(index) / 1_000,
+ ))),
+ ArrowValueDecoder::Time32Millisecond(arr) =>
Ok(Value::String(micros_to_time_string(
+ arr.value(index) as i64 * 1_000,
+ ))),
+ ArrowValueDecoder::Time32Second(arr) =>
Ok(Value::String(micros_to_time_string(
+ arr.value(index) as i64 * 1_000_000,
+ ))),
+ // Intervals likewise have no dedicated Value variant, so render
as strings.
+ ArrowValueDecoder::IntervalYearMonth(arr) => {
+ let months = arr.value(index);
+ Ok(Value::String(format!(
+ "{}-{}",
+ months / 12,
+ (months % 12).abs()
+ )))
+ }
+ ArrowValueDecoder::IntervalDayTime(arr) => {
+ let v = arr.value(index);
+ Ok(Value::String(format!(
+ "{} days {} ms",
+ v.days, v.milliseconds
+ )))
+ }
+ ArrowValueDecoder::IntervalMonthDayNano(arr) => {
+ let v = arr.value(index);
+ Ok(Value::String(format!(
+ "{} months {} days {} ns",
+ v.months, v.days, v.nanoseconds
+ )))
+ }
+ ArrowValueDecoder::List { array, child } => {
+ // Sliced list/map arrays retain their original children. Their
+ // offsets already identify the correct absolute child range.
+ let offsets = array.value_offsets();
+ let start = offsets[index] as usize;
+ let end = offsets[index + 1] as usize;
+ let mut items = Vec::with_capacity(end - start);
+ for child_index in start..end {
+ items.push(child.value_at(child_index)?);
}
- };
- return Ok(Value::Variant {
- value: bin_field("value")?,
- metadata: bin_field("metadata")?,
- });
- }
- let mut values = Vec::with_capacity(children.len());
- for (col, child_layout) in arr.columns().iter().zip(children) {
- values.push(arrow_value_at_with_layout(
- col.as_ref(),
- index,
- child_layout,
- )?);
- }
- return Ok(Value::Struct(StructValue::new(Arc::clone(names), values)));
- }
- if let Some(arr) = array.as_any().downcast_ref::<MapArray>() {
- let ArrowValueLayout::Map(children) = value_layout else {
- unreachable!("layout is prepared from the array's data type");
- };
- let entries = arr.value(index);
- let keys = entries.column(0);
- let vals = entries.column(1);
- // The entries in their order, with keys of their own type, as PySpark
reads them.
- let mut map = Vec::with_capacity(entries.len());
- for i in 0..entries.len() {
- map.push((
- arrow_value_at_with_layout(keys.as_ref(), i, &children.0)?,
- arrow_value_at_with_layout(vals.as_ref(), i, &children.1)?,
- ));
+ Ok(Value::List(items))
+ }
+ ArrowValueDecoder::Struct {
+ names,
+ children,
+ variant_fields,
+ } => {
+ // VARIANT arrives as struct<value: binary, metadata: binary>,
with
+ // {"variant": "true"} on the metadata field. Return raw-byte
+ // Value::Variant so it materializes as pyspark's VariantVal
rather
+ // than a plain {value, metadata} struct/dict.
+ if let Some((value, metadata)) = variant_fields {
+ let binary_field = |column: Option<usize>| ->
Result<Vec<u8>> {
+ match column {
+ Some(column) => match
children[column].value_at(index)? {
+ Value::Binary(bytes) => Ok(bytes),
+ Value::Null => Ok(vec![]),
+ _ => Err(SparkError::connect_msg("variant
field is not binary")),
+ },
+ None => Ok(vec![]),
+ }
+ };
+ return Ok(Value::Variant {
+ value: binary_field(*value)?,
+ metadata: binary_field(*metadata)?,
+ });
+ }
+ let mut values = Vec::with_capacity(children.len());
+ for child in children {
+ values.push(child.value_at(index)?);
+ }
+ Ok(Value::Struct(StructValue::new(Arc::clone(names), values)))
+ }
+ ArrowValueDecoder::Map { array, children } => {
+ let offsets = array.value_offsets();
+ let start = offsets[index] as usize;
+ let end = offsets[index + 1] as usize;
+ let mut map = Vec::with_capacity(end - start);
+ for child_index in start..end {
+ map.push((
+ children.0.value_at(child_index)?,
+ children.1.value_at(child_index)?,
+ ));
+ }
+ Ok(Value::Map(map))
+ }
+ ArrowValueDecoder::Unsupported =>
Err(SparkError::connect_msg(format!(
+ "Unsupported Arrow type {:?} - cannot convert to Value",
+ self.array.data_type()
+ ))),
}
- return Ok(Value::Map(map));
- }
- // Decimal256 -> exact string-preserving Decimal (Arrow formats with the
scale).
- if let Some(arr) = array.as_any().downcast_ref::<Decimal256Array>() {
- return Ok(Value::Decimal {
- value: arr.value_as_string(index),
- precision: Some(arr.precision() as i32),
- scale: Some(arr.scale() as i32),
- });
}
- if let Some(arr) = array.as_any().downcast_ref::<FixedSizeBinaryArray>() {
- return Ok(Value::Binary(arr.value(index).to_vec()));
- }
- // TimeType (no dedicated Value): render as an ISO time string, normalized
to micros.
- if let Some(arr) = array.as_any().downcast_ref::<Time64MicrosecondArray>()
{
- return Ok(Value::String(micros_to_time_string(arr.value(index))));
- }
- if let Some(arr) = array.as_any().downcast_ref::<Time64NanosecondArray>() {
- return Ok(Value::String(micros_to_time_string(
- arr.value(index) / 1_000,
- )));
- }
- if let Some(arr) = array.as_any().downcast_ref::<Time32MillisecondArray>()
{
- return Ok(Value::String(micros_to_time_string(
- arr.value(index) as i64 * 1_000,
- )));
- }
- if let Some(arr) = array.as_any().downcast_ref::<Time32SecondArray>() {
- return Ok(Value::String(micros_to_time_string(
- arr.value(index) as i64 * 1_000_000,
- )));
- }
- // Interval types (no dedicated Value): render a compact string.
- if let Some(arr) = array.as_any().downcast_ref::<IntervalYearMonthArray>()
{
- let months = arr.value(index);
- return Ok(Value::String(format!(
- "{}-{}",
- months / 12,
- (months % 12).abs()
- )));
- }
- if let Some(arr) = array.as_any().downcast_ref::<IntervalDayTimeArray>() {
- let v = arr.value(index);
- return Ok(Value::String(format!(
- "{} days {} ms",
- v.days, v.milliseconds
- )));
- }
- if let Some(arr) =
array.as_any().downcast_ref::<IntervalMonthDayNanoArray>() {
- let v = arr.value(index);
- return Ok(Value::String(format!(
- "{} months {} days {} ns",
- v.months, v.days, v.nanoseconds
- )));
- }
-
- Err(SparkError::connect_msg(format!(
- "Unsupported Arrow type {:?} - cannot convert to Value",
- array.data_type()
- )))
+}
+
+#[cfg(test)]
+fn arrow_value_at(array: &dyn arrow::array::Array, index: usize) ->
Result<Value> {
+ ArrowValueLayout::new(array).value_at(index)
}
#[cfg(test)]
@@ -2448,7 +2503,11 @@ mod nested_decode_tests {
use arrow::ipc::writer::StreamWriter;
use arrow::record_batch::RecordBatch;
- fn decode_batches(columns: Vec<(&str, ArrayRef)>, repeats: usize) ->
Vec<Row> {
+ fn encoded_batches(
+ columns: Vec<(&str, ArrayRef)>,
+ repeats: usize,
+ empty_boundaries: bool,
+ ) -> proto::execute_plan_response::ArrowBatch {
let schema = Arc::new(Schema::new(
columns
.iter()
@@ -2463,16 +2522,29 @@ mod nested_decode_tests {
let mut data = vec![];
{
let mut writer = StreamWriter::try_new(&mut data,
&schema).unwrap();
+ if empty_boundaries {
+ writer
+ .write(&RecordBatch::new_empty(schema.clone()))
+ .unwrap();
+ }
for _ in 0..repeats {
writer.write(&batch).unwrap();
}
+ if empty_boundaries {
+ writer
+ .write(&RecordBatch::new_empty(schema.clone()))
+ .unwrap();
+ }
writer.finish().unwrap();
}
- decode_arrow_batch(&proto::execute_plan_response::ArrowBatch {
+ proto::execute_plan_response::ArrowBatch {
data: data.into(),
..Default::default()
- })
- .unwrap()
+ }
+ }
+
+ fn decode_batches(columns: Vec<(&str, ArrayRef)>, repeats: usize) ->
Vec<Row> {
+ decode_arrow_batch(&encoded_batches(columns, repeats, false)).unwrap()
}
fn as_struct(value: &Value) -> &StructValue {
@@ -2482,6 +2554,230 @@ mod nested_decode_tests {
row
}
+ #[test]
+ fn mixed_nullable_slices_decode_across_empty_and_repeated_batches() {
+ let columns: Vec<(&str, ArrayRef)> = vec![
+ (
+ "dup",
+ Arc::new(Int64Array::from(vec![Some(-1), Some(7), None])),
+ ),
+ (
+ "dup",
+ Arc::new(StringArray::from(vec![Some("skip"), Some("é\0雪"),
None])),
+ ),
+ (
+ "bool",
+ Arc::new(BooleanArray::from(vec![Some(true), Some(false),
None])),
+ ),
+ (
+ "bytes",
+ Arc::new(BinaryArray::from(vec![
+ Some(b"skip".as_ref()),
+ Some(b"\0\xff".as_ref()),
+ None,
+ ])),
+ ),
+ ("null", Arc::new(NullArray::new(3))),
+ ];
+ let batch = encoded_batches(
+ columns
+ .into_iter()
+ .map(|(name, array)| (name, array.slice(1, 2)))
+ .collect(),
+ 2,
+ true,
+ );
+ let rows = decode_arrow_batch(&batch).unwrap();
+ assert_eq!(rows.len(), 4);
+ for batch in rows.chunks(2) {
+ assert_eq!(batch[0].fields(), &["dup", "dup", "bool", "bytes",
"null"]);
+ assert_eq!(
+ batch[0].values(),
+ &[
+ Value::Long(7),
+ Value::String("é\0雪".into()),
+ Value::Bool(false),
+ Value::Binary(vec![0, 255]),
+ Value::Null,
+ ]
+ );
+ assert_eq!(batch[1].values(), &vec![Value::Null; 5]);
+ assert_eq!(batch[0].get_by_name("dup"), Some(&Value::Long(7)));
+ assert!(std::ptr::eq(batch[0].fields(), batch[1].fields()));
+ }
+ // The IPC buffers and borrowed column layouts have already been
dropped.
+ assert_eq!(rows[2].get(1), Some(&Value::String("é\0雪".into())));
+ }
+
+ #[test]
+ fn sliced_parents_decode_children_that_were_already_sliced() {
+ let leaf = StructArray::from(vec![(
+ Arc::new(Field::new("n", DataType::Int32, false)),
+ Arc::new(Int32Array::from(vec![-9, 10, 20, 30, 40, 50, 60])) as
ArrayRef,
+ )]);
+ let entries = StructArray::from(vec![
+ (
+ Arc::new(Field::new("key", DataType::Utf8, false)),
+ Arc::new(StringArray::from(vec![
+ "skip", "a", "b", "c", "d", "e", "f",
+ ])) as ArrayRef,
+ ),
+ (
+ Arc::new(Field::new("value", leaf.data_type().clone(), true)),
+ Arc::new(leaf.clone()) as ArrayRef,
+ ),
+ ])
+ .slice(1, 6);
+ let child = leaf.slice(1, 6);
+ let offsets = OffsetBuffer::new(vec![0, 1, 3, 4, 6].into());
+ let list = ListArray::new(
+ Arc::new(Field::new("item", child.data_type().clone(), true)),
+ offsets.clone(),
+ Arc::new(child),
+ None,
+ )
+ .slice(1, 2);
+ let map = MapArray::new(
+ Arc::new(Field::new("entries", entries.data_type().clone(),
false)),
+ offsets,
+ entries,
+ None,
+ false,
+ )
+ .slice(1, 2);
+ // Exercise the unserialized sliced arrays as well as the IPC path:
+ // serialization is allowed to normalize their physical child offsets.
+ let value = |n| Value::Struct(Row::new(vec!["n".into()],
vec![Value::Integer(n)]).into());
+ let expected_lists = [
+ Value::List(vec![value(20), value(30)]),
+ Value::List(vec![value(40)]),
+ ];
+ let expected_maps = [
+ Value::Map(vec![
+ (Value::String("b".into()), value(20)),
+ (Value::String("c".into()), value(30)),
+ ]),
+ Value::Map(vec![(Value::String("d".into()), value(40))]),
+ ];
+ let list_layout = ArrowValueLayout::new(&list);
+ let map_layout = ArrowValueLayout::new(&map);
+ for i in 0..2 {
+ assert_eq!(list_layout.value_at(i).unwrap(), expected_lists[i]);
+ assert_eq!(map_layout.value_at(i).unwrap(), expected_maps[i]);
+ }
+ drop((list_layout, map_layout));
+ let rows = decode_batches(vec![("list", Arc::new(list)), ("map",
Arc::new(map))], 1);
+ for i in 0..2 {
+ assert_eq!(
+ rows[i].values(),
+ &[expected_lists[i].clone(), expected_maps[i].clone()]
+ );
+ }
+ }
+
+ #[test]
+ fn
unsupported_children_remain_lazy_under_nulls_empty_containers_and_variants() {
+ let null_duration = DurationSecondArray::from(vec![None]);
+ assert_eq!(arrow_value_at(&null_duration, 0).unwrap(), Value::Null);
+ let populated: ArrayRef =
Arc::new(DurationSecondArray::from(vec![Some(7)]));
+ let list = ListArray::new(
+ Arc::new(Field::new("item", populated.data_type().clone(), true)),
+ OffsetBuffer::new(vec![0, 0].into()),
+ populated.clone(),
+ None,
+ );
+ assert_eq!(arrow_value_at(&list, 0).unwrap(), Value::List(vec![]));
+ let entries = StructArray::from(vec![
+ (
+ Arc::new(Field::new("key", DataType::Utf8, false)),
+ Arc::new(StringArray::from(vec!["a"])) as ArrayRef,
+ ),
+ (
+ Arc::new(Field::new("value", populated.data_type().clone(),
true)),
+ populated.clone(),
+ ),
+ ]);
+ let map = MapArray::new(
+ Arc::new(Field::new("entries", entries.data_type().clone(),
false)),
+ OffsetBuffer::new(vec![0, 0].into()),
+ entries,
+ None,
+ false,
+ );
+ assert_eq!(
+ arrow_value_at(&map, 0).unwrap(),
+ Value::Map(Default::default())
+ );
+ let parent = StructArray::new(
+ vec![Arc::new(Field::new(
+ "unused",
+ populated.data_type().clone(),
+ true,
+ ))]
+ .into(),
+ vec![populated.clone()],
+ Some(NullBuffer::from(vec![false])),
+ );
+ assert_eq!(arrow_value_at(&parent, 0).unwrap(), Value::Null);
+ // Deliberately noncanonical: detection accepts a marker on any field,
+ // and the first matching "value" wins while an unsupported duplicate
+ // remains unread. The absent "metadata" field must still decode as
empty.
+ let variant = StructArray::from(vec![
+ (
+ Arc::new(
+ Field::new("value", DataType::Binary, false)
+ .with_metadata([("variant".into(),
"true".into())].into()),
+ ),
+ Arc::new(BinaryArray::from_iter_values([b"x".as_ref()])) as
ArrayRef,
+ ),
+ (
+ Arc::new(Field::new("value", populated.data_type().clone(),
true)),
+ populated,
+ ),
+ ]);
+ assert_eq!(
+ arrow_value_at(&variant, 0).unwrap(),
+ Value::Variant {
+ value: vec![b'x'],
+ metadata: vec![]
+ }
+ );
+ }
+
+ #[test]
+ fn decoding_errors_keep_row_major_column_order() {
+ let first_column = encoded_batches(
+ vec![
+ ("u", Arc::new(UInt64Array::from(vec![u64::MAX]))),
+ (
+ "duration",
+ Arc::new(DurationSecondArray::from(vec![Some(1)])),
+ ),
+ ],
+ 1,
+ false,
+ );
+ assert!(decode_arrow_batch(&first_column)
+ .unwrap_err()
+ .to_string()
+ .contains("UInt64 value 18446744073709551615 exceeds i64 range"));
+ let first_row = encoded_batches(
+ vec![
+ ("u", Arc::new(UInt64Array::from(vec![1, u64::MAX]))),
+ (
+ "duration",
+ Arc::new(DurationSecondArray::from(vec![Some(1), None])),
+ ),
+ ],
+ 1,
+ false,
+ );
+ assert!(decode_arrow_batch(&first_row)
+ .unwrap_err()
+ .to_string()
+ .contains("Unsupported Arrow type Duration(Second) - cannot
convert to Value"));
+ }
+
#[test]
fn nested_names_preserve_duplicates_unicode_nulls_and_sibling_layouts() {
let names = ["dup", "dup", "", "名前"];
@@ -3748,6 +4044,87 @@ mod arrow_value_tests {
);
}
+ #[test]
+ fn typed_scalar_layouts_preserve_exact_values_and_normalization() {
+ let decimal128 = Decimal128Array::from(vec![-12345i128])
+ .with_precision_and_scale(10, 2)
+ .unwrap();
+ let decimal256 = Decimal256Array::from(vec![i256::from_i128(-12345)])
+ .with_precision_and_scale(10, 2)
+ .unwrap();
+ let decimal = Value::Decimal {
+ value: "-123.45".into(),
+ precision: Some(10),
+ scale: Some(2),
+ };
+ let cases: Vec<(ArrayRef, Value)> = vec![
+ (
+ Arc::new(TimestampSecondArray::from(vec![-1])),
+ Value::Timestamp(-1_000_000),
+ ),
+ (
+ Arc::new(TimestampMillisecondArray::from(vec![-1])),
+ Value::Timestamp(-1_000),
+ ),
+ (
+ Arc::new(TimestampNanosecondArray::from(vec![-1999])),
+ Value::Timestamp(-1),
+ ),
+ (
+ Arc::new(Date64Array::from(vec![-86_400_001])),
+ Value::Date(-1),
+ ),
+ (
+ Arc::new(UInt32Array::from(vec![u32::MAX])),
+ Value::Long(u32::MAX as i64),
+ ),
+ (Arc::new(decimal128), decimal.clone()),
+ (Arc::new(decimal256), decimal),
+ (
+ Arc::new(LargeStringArray::from_iter_values(["é\0雪"])),
+ Value::String("é\0雪".into()),
+ ),
+ (
+ Arc::new(StringViewArray::from_iter_values(["é\0雪"])),
+ Value::String("é\0雪".into()),
+ ),
+ (
+
Arc::new(LargeBinaryArray::from_iter_values([b"\0\xff".as_ref()])),
+ Value::Binary(vec![0, 255]),
+ ),
+ (
+
Arc::new(BinaryViewArray::from_iter_values([b"\0\xff".as_ref()])),
+ Value::Binary(vec![0, 255]),
+ ),
+ (
+ Arc::new(Time64NanosecondArray::from(vec![1_234_567])),
+ Value::String("00:00:00.001234".into()),
+ ),
+ (
+ Arc::new(Time32MillisecondArray::from(vec![1])),
+ Value::String("00:00:00.001000".into()),
+ ),
+ ];
+ for (array, expected) in cases {
+ assert_eq!(
+ ArrowValueLayout::new(array.as_ref()).value_at(0).unwrap(),
+ expected
+ );
+ }
+ for value in [
+ -0.0,
+ f64::INFINITY,
+ f64::NEG_INFINITY,
+ f64::from_bits(0x7ff8_0000_0000_0001),
+ ] {
+ let array = Float64Array::from(vec![value]);
+ let Value::Double(decoded) = arrow_value_at(&array, 0).unwrap()
else {
+ panic!("expected double")
+ };
+ assert_eq!(decoded.to_bits(), value.to_bits());
+ }
+ }
+
#[test]
fn primitives_and_signed_ints() {
assert!(matches!(
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]