[
https://issues.apache.org/jira/browse/SPARK-57836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uroš Bojanić resolved SPARK-57836.
----------------------------------
Fix Version/s: 4.4.0
Resolution: Fixed
Issue resolved by pull request 59577
[https://github.com/apache/spark/pull/59577]
> Produce and accept nanosecond precision in to_timestamp, to_timestamp_ntz,
> to_timestamp_ltz and try_to_timestamp
> ----------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-57836
> URL: https://issues.apache.org/jira/browse/SPARK-57836
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 4.3.0
> Reporter: Max Gekk
> Assignee: Stevo Mitric
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.4.0
>
>
> This sub-task is part of the umbrella SPARK-56822 (timestamps with nanosecond
> precision).
> h2. Problem
> {{ParseToTimestamp}} (datetimeExpressions.scala ~L2641-2674) and the builders
> (~L1506-1591) pin the target {{dataType}} to {{TimestampNTZType}} /
> {{TimestampType}} / {{SQLConf.get.timestampType}} (all microsecond), and
> {{inputTypes}} omit nanosecond. So there is no way to PARSE a string into a
> nanosecond timestamp, and a nanosecond input is narrowed to microseconds.
> h2. Goal
> Allow {{to_timestamp}}, {{to_timestamp_ntz}}, {{to_timestamp_ltz}} and
> {{try_to_timestamp}} to produce nanosecond-precision output (at least via
> {{TIMESTAMP_NTZ(p)}} / {{TIMESTAMP_LTZ(p)}} target semantics), parse up to 9
> fractional digits, and accept nanosecond inputs without narrowing.
> h2. Scope
> Thread a target precision/type through the builders; use the nanosecond-aware
> {{TimestampFormatter}} for parsing; accept {{AnyTimestampNanoType}} inputs in
> {{ToTimestamp}} / {{GetTimestamp}} {{inputTypes}}.
> h2. Acceptance criteria
> * Parsing a 7-9 digit fractional string yields a nanosecond timestamp; the
> {{try_}} variant returns null instead of throwing on overflow/invalid input.
> h2. Testing
> {{DateExpressionsSuite}}; nanos golden files.
> h2. Dependencies
> None hard - coordinate with the overflow/range-hardening sub-task (parse
> overflow error taxonomy).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
