Max Gekk created SPARK-57486:
--------------------------------
Summary: Reuse AnyTimestampNanoType for nanosecond-precision
timestamp type checks
Key: SPARK-57486
URL: https://issues.apache.org/jira/browse/SPARK-57486
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 4.3.0
Reporter: Max Gekk
SPARK-57469 introduced the AnyTimestampNanoType abstraction (an AbstractDataType
plus the AnyTimestampNanoTypeExpression extractor) but it was used in only one
place. Meanwhile, many sites across catalyst/core/hive still discriminate on the
nanosecond-precision timestamp types by spelling out the pair explicitly, e.g.:
case _: TimestampNTZNanosType | _: TimestampLTZNanosType => ...
t.isInstanceOf[TimestampNTZNanosType] ||
t.isInstanceOf[TimestampLTZNanosType]
This is duplicated logic that must be kept in sync whenever the set of
nanosecond timestamp types changes.
This is a code-cleanup/refactoring task to centralize that check:
* Follow the existing AnyTimeType design: add a companion
`abstract class AnyTimestampNanoType extends DatetimeType { def precision:
Int }`
and make TimestampNTZNanosType / TimestampLTZNanosType extend it. This lets
the
abstraction be used as a plain type pattern `case _: AnyTimestampNanoType`.
* Simplify AnyTimestampNanoType.acceptsType and AnyTimestampNanoTypeExpression
to
`isInstanceOf[AnyTimestampNanoType]`.
* Replace the explicit TimestampNTZNanosType + TimestampLTZNanosType pair-checks
across the codebase (type-coercion, codegen, projections, hashing, data-source
supportDataType checks, etc.) with `case _: AnyTimestampNanoType`.
The change is behavior-preserving: no user-facing change and no new
functionality.
Because the nanosecond timestamp types are an unreleased preview feature,
changing
their superclass has no binary-compatibility impact (verified with MiMa).
This is a sub-task of SPARK-56822 (SPIP: Timestamps with nanosecond precision).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
