[
https://issues.apache.org/jira/browse/SPARK-60607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60607:
-----------------------------------
Labels: pull-request-available (was: )
> Decode Arrow day-time intervals as timedelta in the Rust Spark Connect client
> -----------------------------------------------------------------------------
>
> Key: SPARK-60607
> URL: https://issues.apache.org/jira/browse/SPARK-60607
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: DB Tsai
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> The Rust Spark Connect client cannot collect non-null Spark
> DayTimeIntervalType values returned as Arrow Duration(Microsecond). The
> generic decoder raises:
> Unsupported Arrow type Duration(Microsecond) - cannot convert to Value
> This is an existing correctness gap, reproduced on master-equivalent source
> 74b8e4125ee8c80f4aad51beabfd37277e39215a (tree identical to
> c36c846fcb4f22c9995e5de5c69c45ae76a5ad9a), before the input-allocation
> optimization in SPARK-60606 /
> https://github.com/apache/spark-connect-rust/pull/109. It is not a regression
> caused by that PR.
> The benchmark's types-mixed-record-w256-collect-scan fixture contains 16 rows
> of nested struct columns including a DayTimeIntervalType leaf encoded as
> pa.duration('us'). Both Rust builds fail all 15 fresh-process samples during
> warmup in each screening/confirmation phase; the official Python client
> succeeds and materializes datetime.timedelta values. The frozen fixture
> includes nulls and populated positive multi-day/microsecond values. No timed
> Rust CPU/wall/RSS metrics exist for the failed operation.
> Harness and fixture source:
> https://github.com/dbtsai/pyspark-connect-benchmark/tree/23bdf9046815ae615c921b2e6bab827eed00aaae
> The Rust Arrow decoder supports Arrow IntervalYearMonth, IntervalDayTime and
> IntervalMonthDayNano physical types, but these are different from
> Duration(Microsecond). There is no DurationMicrosecondArray conversion arm or
> exact Python timedelta Value materialization. All-null arrays can pass
> because nulls are handled before the unsupported-type branch.
> Add exact signed-microsecond decoding and Python datetime.timedelta
> materialization, preserving the logical type and schema. Do not replace
> durations with integers, strings, zero or null. Cover positive, negative,
> zero and multi-day values; null and all-null arrays; nested
> structs/lists/maps; slices/offsets; empty and multiple batches; collect and
> fully consumed iteration; actual Python timedelta type/value equality. Test
> signed 64-bit microsecond boundaries and specify range/overflow behavior and
> supported units without lossy intermediate nanosecond conversion. Preserve
> the existing Arrow interval conversions.
> SPARK-60573 concerns JVM saturation for large intervals and is related but
> distinct: this issue is missing Rust support even for small values.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
