Max Gekk created SPARK-57490:
--------------------------------
Summary: Support CAST between nanosecond timestamp types of
different precision
Key: SPARK-57490
URL: https://issues.apache.org/jira/browse/SPARK-57490
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 4.3.0
Reporter: Max Gekk
Assignee: Max Gekk
h2. Summary
Add support for {{CAST}} between same-family nanosecond-precision timestamp
types of
different precision:
* {{TIMESTAMP_NTZ(p1)}} <-> {{TIMESTAMP_NTZ(p2)}}
* {{TIMESTAMP_LTZ(q1)}} <-> {{TIMESTAMP_LTZ(q2)}}
where {{p1, p2, q1, q2}} are in {{[7, 9]}}.
h2. Background
The nanosecond-capable timestamp types {{TIMESTAMP_NTZ(p)}} and
{{TIMESTAMP_LTZ(p)}}
(p in [7, 9]) are gated behind {{spark.sql.timestampNanosTypes.enabled}}. Casts
between
these types and their microsecond counterparts ({{TIMESTAMP_NTZ}} /
{{TIMESTAMP}}), as
well as to/from {{DATE}} and {{STRING}}, are already supported. However,
casting between
two nanosecond timestamps of the *same family but different precision* was not
allowed:
such a cast is currently absent from {{Cast.canCast}} / {{Cast.canAnsiCast}}
and fails
type checking.
h2. Semantics
Both {{TimestampNTZNanosType}} and {{TimestampLTZNanosType}} share the same
physical
value {{TimestampNanosVal(epochMicros, nanosWithinMicro)}} (nanosWithinMicro in
[0, 999]).
A same-family cross-precision cast keeps {{epochMicros}} unchanged and only
adjusts the
sub-microsecond component:
* *Widening* (target precision >= source precision): lossless; the value is
unchanged.
* *Narrowing* (target precision < source precision): floors
{{nanosWithinMicro}} toward
the past to the target precision step (drop the lowest {{9 - precision}}
sub-microsecond
digits), consistent with the existing nanos -> micros narrowing rule.
No time-zone conversion is involved (the conversion stays within one family).
h2. Store-assignment / up-cast contract
Following the established micros <-> nanos precedent:
* Widening (lossless) is allowed as an ANSI store assignment but is not an
up-cast.
* Narrowing (lossy) is explicit-CAST only (rejected by both up-cast and store
assignment).
* Equal precision is the identity cast.
h2. Scope
* In scope: same-family cross-precision casts (NTZ<->NTZ, LTZ<->LTZ).
* Out of scope: cross-family precision casts (e.g. {{TIMESTAMP_NTZ(9)}} ->
{{TIMESTAMP_LTZ(7)}}) and implicit type coercion / {{findWiderDateTimeType}}
widening
for mixed-precision nanos operands (UNION, COALESCE, IF). These can be
handled as
follow-ups.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
