MaxGekk opened a new pull request, #56810: URL: https://github.com/apache/spark/pull/56810
### What changes were proposed in this pull request? This PR defines a precision-safe store-assignment / up-cast contract for the whole LTZ/NTZ timestamp family - the microsecond types (`TIMESTAMP` / `TIMESTAMP_NTZ`) and their nanosecond-precision counterparts (`TIMESTAMP_LTZ(p)` / `TIMESTAMP_NTZ(p)`, `p` in `[7, 9]`) - using a single notion of effective fractional-second precision (micros = 6, nanos = `p`). For any ordered pair of timestamp-family types (including across the LTZ/NTZ boundary, which Spark already treats as a mutual up-cast for the micro types): - target precision `>=` source precision: lossless widening -> up-cast (STRICT) and ANSI-store-assignable; - target precision `<` source precision: lossy narrowing -> not an up-cast, blocked under ANSI so it can never silently truncate. `DATE <-> nanos` is aligned to the micro `DATE <-> TIMESTAMP` behavior: `DATE -> nanos` is a lossless widening (up-cast + ANSI-store-assignable), while `nanos -> DATE` drops the time-of-day (not an up-cast, but still ANSI-store-assignable). LEGACY policy and explicit `CAST` are unchanged (they still truncate on narrowing). `TIME <-> timestamp` is unchanged and stays consistent with `TIME <-> micros` (never an up-cast, ANSI-store-assignable both ways). Concretely: - New shared `private[sql] object TimestampFamily` (`sql/api`) with `fractionalPrecision(dt): Option[Int]` plus `isLtz` / `isNtz`, reused across the rule sites (no type-hierarchy change, so no MiMa impact). - `UpCastRule.canUpCast`: a single lossless-widening arm for the family (subsuming the existing `TimestampType <-> TimestampNTZType` cases), plus a generalized `DATE -> family` widening arm. - `Cast.canANSIStoreAssign`: replaced the piecemeal per-subtask arms with one family narrowing block built on the shared helper, before the generic `DatetimeType` arm. - `TypeCoercionHelper.findWiderDateTimeType`: refactored onto the shared helper (behavior-preserving) and updated the now-stale comment, since common-type resolution and the cast rules now agree on admissibility. ### Why are the changes needed? Before this change, the nanosecond timestamp types fell through the generic `(_: DatetimeType, _: DatetimeType)` arm in `Cast.canANSIStoreAssign` (risking silent sub-microsecond truncation handled only narrowly), and they were absent from `UpCastRule.canUpCast`, so STRICT store assignment and up-cast resolution rejected even lossless widening. This PR gives the family a complete, precision-safe contract consistent with the microsecond precedent. ### Does this PR introduce _any_ user-facing change? No. The nanosecond-precision timestamp types are unreleased (`@Unstable`), so this only affects behavior within the unreleased branch. ### How was this patch tested? - Updated the `SPARK-57293` / `SPARK-57490` / cross-family / micro-boundary contract tests in `CastSuiteBase` to the precision-safe widening model. - Added a full-matrix predicate test over all 8 timestamp-family types asserting `canUpCast` and `canANSIStoreAssign` are true iff target precision `>=` source precision, plus `DATE` and `TIME` consistency anchors. - Ran `CastSuite`, `CastWithAnsiOn/Off`, `TypeCoercionSuite`, `AnsiTypeCoercionSuite`, `DataTypeWriteCompatibilitySuite`, `V2WriteAnalysisSuite`, and `SQLQueryTestSuite` (cast / try_cast / nanos / typeCoercion) - all pass with no golden-file changes. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Cursor (Claude Opus 4.8) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
