[
https://issues.apache.org/jira/browse/SPARK-57490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk resolved SPARK-57490.
------------------------------
Fix Version/s: 4.3.0
Resolution: Fixed
Issue resolved by pull request 56544
[https://github.com/apache/spark/pull/56544]
> Support CAST between nanosecond timestamp types of different precision
> ----------------------------------------------------------------------
>
> Key: SPARK-57490
> URL: https://issues.apache.org/jira/browse/SPARK-57490
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 4.3.0
> Reporter: Max Gekk
> Assignee: Max Gekk
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.3.0
>
>
> h2. Summary
> Add support for {{CAST}} between same-family nanosecond-precision timestamp
> types of
> different precision:
> * {{TIMESTAMP_NTZ(p1)}} <-> {{TIMESTAMP_NTZ(p2)}}
> * {{TIMESTAMP_LTZ(q1)}} <-> {{TIMESTAMP_LTZ(q2)}}
> where {{p1, p2, q1, q2}} are in {{[7, 9]}}.
> h2. Background
> The nanosecond-capable timestamp types {{TIMESTAMP_NTZ(p)}} and
> {{TIMESTAMP_LTZ(p)}}
> (p in [7, 9]) are gated behind {{spark.sql.timestampNanosTypes.enabled}}.
> Casts between
> these types and their microsecond counterparts ({{TIMESTAMP_NTZ}} /
> {{TIMESTAMP}}), as
> well as to/from {{DATE}} and {{STRING}}, are already supported. However,
> casting between
> two nanosecond timestamps of the *same family but different precision* was
> not allowed:
> such a cast is currently absent from {{Cast.canCast}} / {{Cast.canAnsiCast}}
> and fails
> type checking.
> h2. Semantics
> Both {{TimestampNTZNanosType}} and {{TimestampLTZNanosType}} share the same
> physical
> value {{TimestampNanosVal(epochMicros, nanosWithinMicro)}} (nanosWithinMicro
> in [0, 999]).
> A same-family cross-precision cast keeps {{epochMicros}} unchanged and only
> adjusts the
> sub-microsecond component:
> * *Widening* (target precision >= source precision): lossless; the value is
> unchanged.
> * *Narrowing* (target precision < source precision): floors
> {{nanosWithinMicro}} toward
> the past to the target precision step (drop the lowest {{9 - precision}}
> sub-microsecond
> digits), consistent with the existing nanos -> micros narrowing rule.
> No time-zone conversion is involved (the conversion stays within one family).
> h2. Store-assignment / up-cast contract
> Following the established micros <-> nanos precedent:
> * Widening (lossless) is allowed as an ANSI store assignment but is not an
> up-cast.
> * Narrowing (lossy) is explicit-CAST only (rejected by both up-cast and store
> assignment).
> * Equal precision is the identity cast.
> h2. Scope
> * In scope: same-family cross-precision casts (NTZ<->NTZ, LTZ<->LTZ).
> * Out of scope: cross-family precision casts (e.g. {{TIMESTAMP_NTZ(9)}} ->
> {{TIMESTAMP_LTZ(7)}}) and implicit type coercion /
> {{findWiderDateTimeType}} widening
> for mixed-precision nanos operands (UNION, COALESCE, IF). These can be
> handled as
> follow-ups.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
