viirya opened a new pull request, #57088: URL: https://github.com/apache/spark/pull/57088
### What changes were proposed in this pull request? This extends the opt-in lossless Arrow encoding introduced by SPARK-57975 (#57053) to `CalendarIntervalType`, and hardens the default interval writer's overflow error: - **Lossless struct encoding**: with the opt-in flag, a `CalendarInterval` column maps to an Arrow struct of `(months: int32, days: int32, microseconds: int64)` -- the type's own field layout, mirroring the default in-memory cache's `CALENDAR_INTERVAL` `ColumnType`. The components are stored as-is with no unit conversion, so the full `Long` microsecond domain round-trips. The struct is tagged through child-field metadata (the geometry/variant pattern) and is self-describing on read: `fromArrowField` recovers `CalendarIntervalType`, `ArrowWriter` selects a dedicated struct writer, and `ArrowColumnVector` serves `getInterval` from the child vectors, including nested inside arrays, structs, and maps. - **Flag rename**: the parameter is renamed from `losslessTimestampNanos` to `losslessInternalTypes`, since it now selects the lossless encoding for both kinds of types whose standard Arrow encoding cannot cover their full Spark value domain. `ArrowUtils` is `private[sql]`, so the rename has no compatibility impact; the only intended caller (the Arrow-based Dataset cache, #56334) wants both types, and the flag expresses one intent: internal storage wants fidelity. - **Structured error at the conversion site**: `IntervalMonthDayNanoWriter` now catches the `Math.multiplyExact(microseconds, 1000L)` overflow exactly at the conversion and raises the structured `DATETIME_OVERFLOW` (new `QueryExecutionErrors.calendarIntervalArrowNanosOverflowError`, the same pattern as `TimestampNTZNanosWriter`'s `timestampNanosEpochNanosOverflowError`) instead of letting a raw `ArithmeticException: long overflow` escape. Because the catch is scoped to the single conversion expression, it cannot re-label unrelated arithmetic failures (e.g. an ANSI `DIVIDE_BY_ZERO` raised by lazily-evaluated upstream input), which was a live mis-attribution risk with any wider catch (see https://github.com/apache/spark/pull/56334#discussion_r3531469848). The default `Interval(MONTH_DAY_NANO)` mapping and every existing caller are unchanged. ### Why are the changes needed? Spark permits the full `Long` microsecond range in `CalendarInterval`, but Arrow's `IntervalMonthDayNano` stores the sub-day component as int64 nanoseconds, so any `|microseconds| > Long.MaxValue / 1000` (roughly +/-292 years) is structurally unrepresentable in the standard encoding -- the default in-memory cache serializer stores the three components raw and has no such limit. As with the nanosecond timestamps in SPARK-57975, the interchange mapping must keep the standard encoding for external consumers, so internal storage (the Arrow-based Dataset cache proposed in #56334) needs a per-call-site lossless alternative; with it, the cache can delete its schema-wide overflow-translation wrapper entirely. Raised in https://github.com/apache/spark/pull/56334#discussion_r3486641463 and https://github.com/apache/spark/pull/56334#discussion_r3531469848. ### Does this PR introduce _any_ user-facing change? The lossless encoding itself is opt-in via an internal API parameter and changes nothing by default. One user-visible improvement on the existing paths: writing an out-of-range `CalendarInterval` through Arrow (e.g. `toPandas`, Arrow UDFs) now fails with the structured `DATETIME_OVERFLOW` condition naming the value and the limit, instead of an opaque `java.lang.ArithmeticException: long overflow`. ### How was this patch tested? New tests: - `ArrowUtilsSuite` "calendar interval lossless struct": schema shape (struct of int32/int32/int64, non-null children), round-trip, nested array/struct/map coverage, user-metadata preservation, no misfire on an untagged struct with the same child names, and the default `Interval(MONTH_DAY_NANO)` mapping staying unchanged when the flag is off. - `ArrowWriterSuite` "calendar interval overflow raises DATETIME_OVERFLOW at the conversion site": the default writer raises the structured condition for `microseconds = Long.MaxValue / 1000 + 1`. - `ArrowWriterSuite` "calendar interval lossless struct round-trip covers the full value domain": write-and-read-back through `ArrowWriter` + `ArrowColumnVector` for values including `Long.MaxValue` / `Long.MinValue` microseconds and full-range months/days (all far outside the default mapping's limit) plus nulls. - `ArrowWriterSuite` "calendar interval lossless struct round-trip inside nested types": the same extreme values inside `array<...>`, `struct<...>`, and `map<int, ...>`. Existing regression suites pass: `ArrowUtilsSuite`, `ArrowWriterSuite`, `ArrowConvertersSuite`, `ColumnVectorSuite`, `ColumnarBatchSuite`. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code This pull request and its description were written by Claude Code. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
