Linhong Liu created SPARK-60573:
-----------------------------------
Summary: Preserve the full day-time interval range in Spark
Connect Arrow decoding
Key: SPARK-60573
URL: https://issues.apache.org/jira/browse/SPARK-60573
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: 5.0.0
Reporter: Linhong Liu
Spark Connect's JVM Arrow reader silently changes large day-time intervals when
decoding results. Spark stores day-time intervals as signed 64-bit microsecond
counts. Arrow DurationVector.getObject converts microseconds through a signed
64-bit nanosecond count, which saturates for valid large Spark intervals.
For example, a duration of 10000000000000000 microseconds is decoded as
approximately 9223372036854775 microseconds. Values at both ends of Spark's
interval range are affected.
Read the raw microseconds and use SparkIntervalUtils.microsToDuration so the
conversion retains the full supported interval range. The Arrow schema and wire
representation remain unchanged.
Regression coverage should include Long.MinValue, Long.MaxValue, small positive
and negative values, zero, NULL, and interval-to-string decoding.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
