DB Tsai created SPARK-60607:
-------------------------------
Summary: Decode Arrow day-time intervals as timedelta in the Rust
Spark Connect client
Key: SPARK-60607
URL: https://issues.apache.org/jira/browse/SPARK-60607
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: DB Tsai
Fix For: connect-rust-4.3.0
The Rust Spark Connect client cannot collect non-null Spark DayTimeIntervalType
values returned as Arrow Duration(Microsecond). The generic decoder raises:
Unsupported Arrow type Duration(Microsecond) - cannot convert to Value
This is an existing correctness gap, reproduced on master-equivalent source
74b8e4125ee8c80f4aad51beabfd37277e39215a (tree identical to
c36c846fcb4f22c9995e5de5c69c45ae76a5ad9a), before the input-allocation
optimization in SPARK-60606 /
https://github.com/apache/spark-connect-rust/pull/109. It is not a regression
caused by that PR.
The benchmark's types-mixed-record-w256-collect-scan fixture contains 16 rows
of nested struct columns including a DayTimeIntervalType leaf encoded as
pa.duration('us'). Both Rust builds fail all 15 fresh-process samples during
warmup in each screening/confirmation phase; the official Python client
succeeds and materializes datetime.timedelta values. The frozen fixture
includes nulls and populated positive multi-day/microsecond values. No timed
Rust CPU/wall/RSS metrics exist for the failed operation.
Harness and fixture source:
https://github.com/dbtsai/pyspark-connect-benchmark/tree/23bdf9046815ae615c921b2e6bab827eed00aaae
The Rust Arrow decoder supports Arrow IntervalYearMonth, IntervalDayTime and
IntervalMonthDayNano physical types, but these are different from
Duration(Microsecond). There is no DurationMicrosecondArray conversion arm or
exact Python timedelta Value materialization. All-null arrays can pass because
nulls are handled before the unsupported-type branch.
Add exact signed-microsecond decoding and Python datetime.timedelta
materialization, preserving the logical type and schema. Do not replace
durations with integers, strings, zero or null. Cover positive, negative, zero
and multi-day values; null and all-null arrays; nested structs/lists/maps;
slices/offsets; empty and multiple batches; collect and fully consumed
iteration; actual Python timedelta type/value equality. Test signed 64-bit
microsecond boundaries and specify range/overflow behavior and supported units
without lossy intermediate nanosecond conversion. Preserve the existing Arrow
interval conversions.
SPARK-60573 concerns JVM saturation for large intervals and is related but
distinct: this issue is missing Rust support even for small values.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
