viirya opened a new pull request, #57053: URL: https://github.com/apache/spark/pull/57053
### What changes were proposed in this pull request? This adds an opt-in Arrow mapping for the nanosecond timestamp types (`TimestampNTZNanosType` / `TimestampLTZNanosType`), selected by a new `losslessTimestampNanos` parameter on `ArrowUtils.toArrowSchema` / `toArrowField` (default `false`). When enabled, a nanosecond timestamp column maps to an Arrow struct of `(epochMicros: int64, nanosWithinMicro: int16)` -- `TimestampNanosVal`'s own layout -- instead of the default single int64 of epoch-nanoseconds: - **Schema (`ArrowUtils`)**: the struct's `epochMicros` child is tagged through field metadata with the NTZ/LTZ kind and the column precision (following the geometry/variant struct tag pattern), so `fromArrowField` recovers the exact Spark type on read with no out-of-band information. Nested occurrences (array/struct/map/UDT sqlType) are covered by threading the flag through the recursive schema construction. - **Write (`ArrowWriter`)**: new `TimestampNTZNanosStructWriter` / `TimestampLTZNanosStructWriter` store the two components as-is -- no unit conversion, hence no overflow. `TimestampNanosTypeOps.createArrowFieldWriter` now dispatches on the vector shape instead of unconditionally casting to the native nanos vectors. - **Read (`ArrowColumnVector`)**: a dedicated `TimestampNanosStructAccessor` recognizes the tagged struct and serves `getTimestampNTZNanos` / `getTimestampLTZNanos` from the child vectors, including nested inside arrays, structs, and maps. The default `Timestamp(NANOSECOND)` mapping and every existing caller are unchanged. ### Why are the changes needed? Spark defines the nanosecond timestamp types over years 0001-9999, and stores values losslessly as `(epochMicros, nanosWithinMicro)`. The standard Arrow mapping packs the value into a single int64 of epoch-nanoseconds, which only covers roughly years 1677-2262: a common sentinel value like `9999-12-31 23:59:59.999999999` fails with `DATETIME_OVERFLOW`. That mapping must stay as-is on interchange paths (pandas conversion, Arrow UDFs) because external consumers expect the standard encoding, but internal Arrow-based storage -- specifically the Arrow-backed Dataset cache proposed in #56334, where the default in-memory cache handles the full domain (SPARK-57735) -- needs a representation that covers the full domain of the types. This was raised in https://github.com/apache/spark/pull/56334#discussion_r3531469977. ### Does this PR introduce _any_ user-facing change? No. The new mapping is opt-in via an internal API parameter that defaults to off; no existing behavior changes. ### How was this patch tested? New tests: - `ArrowUtilsSuite` "timestamp nanos lossless struct": schema shape (struct of int64 + int16, non-null children), type/precision round-trip for NTZ/LTZ at p=7/8/9, LTZ requiring no time zone, nested array/struct/map coverage, user-metadata preservation, precision fallback for a missing/invalid tag, no misfire on an untagged struct with the same child names, and the default mapping staying unchanged. - `ArrowWriterSuite` "timestamp nanos lossless struct round-trip covers the full value domain": write-and-read-back through `ArrowWriter` + `ArrowColumnVector` for values including `9999-12-31T23:59:59.999999999` and `0001-01-01T00:00:00.000000001` (both far outside the int64 epoch-nanos range) plus nulls, for NTZ/LTZ at p=9 and p=7. - `ArrowWriterSuite` "timestamp nanos lossless struct round-trip inside nested types": the same extreme values inside `array<...>`, `struct<...>`, and `map<int, ...>`. Existing regression suites pass: `ArrowUtilsSuite`, `ArrowWriterSuite`, `ArrowConvertersSuite`, `ColumnVectorSuite`, `ColumnarBatchSuite`. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
