[
https://issues.apache.org/jira/browse/SPARK-56876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk updated SPARK-56876:
-----------------------------
Description:
Add TimestampNTZNanosType and TimestampLTZNanosType to the Spark SQL type
system as part of SPARK-56822 (SPIP: Timestamps with nanosecond precision).
The new types shall represent parameterized SQL timestamps whose fractional
seconds in the SECOND field follow SQL optional feature F555, “Enhanced seconds
precision”, in the nanosecond-capable range only:
* TimestampNTZNanosType shall correspond to timestamp without time zone
({{{}timestamp_ntz(p){}}}) in terms of SQL semantics (calendar/local timestamp,
no time-zone attribute stored on the value).
* TimestampLTZNanosType shall correspond to timestamp with local time zone
({{{}timestamp_ltz(p){}}}) in Spark’s existing LTZ sense (session time zone
affects interpretation and display; internal form remains comparable across
sessions in line with current {{timestamp_ltz}} behavior).
The precision parameter p shall be supported only for p = 7, 8, and 9 decimal
digits of fractional seconds (up to full nanosecond resolution in that field).
Values outside that range shall be rejected with clear, documented errors.
The types shall expose an appropriate default size for Catalyst row-size
estimation consistent with the chosen logical representation of microsecond
epoch plus sub-microsecond fractional information.
Unparameterized {{timestamp_ntz}} / {{timestamp_ltz}} shall continue to behave
as today (existing microsecond-oriented types).
Tests: extend {{DataTypeSuite}} to cover valid precisions (7–9), invalid
precisions, naming/catalog strings as applicable, and {{DataType}}
materialization from type metadata; add or update tests for any new error
conditions introduced for invalid precision (including {{SparkThrowableSuite}}
or equivalent checks if error-conditions entries are added or changed).
> Add TimestampNTZNanosType and TimestampLTZNanosType
> ---------------------------------------------------
>
> Key: SPARK-56876
> URL: https://issues.apache.org/jira/browse/SPARK-56876
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 4.2.0
> Reporter: Max Gekk
> Priority: Major
>
> Add TimestampNTZNanosType and TimestampLTZNanosType to the Spark SQL type
> system as part of SPARK-56822 (SPIP: Timestamps with nanosecond precision).
> The new types shall represent parameterized SQL timestamps whose fractional
> seconds in the SECOND field follow SQL optional feature F555, “Enhanced
> seconds precision”, in the nanosecond-capable range only:
> * TimestampNTZNanosType shall correspond to timestamp without time zone
> ({{{}timestamp_ntz(p){}}}) in terms of SQL semantics (calendar/local
> timestamp, no time-zone attribute stored on the value).
> * TimestampLTZNanosType shall correspond to timestamp with local time zone
> ({{{}timestamp_ltz(p){}}}) in Spark’s existing LTZ sense (session time zone
> affects interpretation and display; internal form remains comparable across
> sessions in line with current {{timestamp_ltz}} behavior).
> The precision parameter p shall be supported only for p = 7, 8, and 9 decimal
> digits of fractional seconds (up to full nanosecond resolution in that
> field). Values outside that range shall be rejected with clear, documented
> errors.
> The types shall expose an appropriate default size for Catalyst row-size
> estimation consistent with the chosen logical representation of microsecond
> epoch plus sub-microsecond fractional information.
> Unparameterized {{timestamp_ntz}} / {{timestamp_ltz}} shall continue to
> behave as today (existing microsecond-oriented types).
> Tests: extend {{DataTypeSuite}} to cover valid precisions (7–9), invalid
> precisions, naming/catalog strings as applicable, and {{DataType}}
> materialization from type metadata; add or update tests for any new error
> conditions introduced for invalid precision (including
> {{SparkThrowableSuite}} or equivalent checks if error-conditions entries are
> added or changed).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
