Max Gekk created SPARK-56969:
--------------------------------
Summary: Add SQLConf preview flag for nanosecond-capable timestamp
types
Key: SPARK-56969
URL: https://issues.apache.org/jira/browse/SPARK-56969
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 4.2.0
Reporter: Max Gekk
h3. Summary
Introduce a preview SQL configuration that gates use of
{{TimestampNTZNanosType(p)}} and {{TimestampLTZNanosType(p)}} (p ∈ [7, 9])
added in SPARK-56876. The flag is *off by default* (except in tests, following
{{{}spark.sql.timeType.enabled{}}}).
h3. What to do
* Add {{SQLConf}} entry (name TBD; suggest
{{{}spark.sql.timestampNanos.preview.enabled{}}}, {{{}.internal(){}}}, default
{{{}false{}}}, default {{true}} under {{Utils.isTesting}} — mirror
{{{}TIME_TYPE_ENABLED{}}}).
* Add accessor on {{SQLConf}} (e.g. {{{}isTimestampNanosPreviewEnabled{}}}).
* Extend {{TypeUtils.failUnsupportedDataType}} to reject schemas/plans that
contain {{TimestampNTZNanosType}} or {{TimestampLTZNanosType}} recursively when
the flag is disabled.
* Add error class {{UNSUPPORTED_TIMESTAMP_NANOS_TYPE}} (or reuse a single
message for both NTZ/LTZ) in {{error-conditions.json}} and
{{{}QueryCompilationErrors{}}}, analogous to {{{}UNSUPPORTED_TIME_TYPE{}}}.
* Document the conf in {{SQLConf.scala}} (when true, nanosecond-capable
timestamp types may appear in table schemas, casts, etc.).
* Register conf in binding-policy exceptions list if required (see
{{configs-without-binding-policy-exceptions}} for {{timeType.enabled}}
precedent).
h3. Tests
* Unit: {{SQLConf}} entry exists, default false in non-test mode, readable via
{{{}SQLConf.get{}}}.
* Unit: {{TypeUtils.failUnsupportedDataType}} throws when flag off and schema
contains {{TimestampNTZNanosType(9)}} / {{{}TimestampLTZNanosType(9){}}};
passes when flag on.
* Regression: existing {{DataTypeSuite}} / types without nanos types
unaffected.
* Optional: small {{AnalysisSuite}} or {{DDLParsingSuite}} — {{CREATE TABLE t
(c TIMESTAMP_NTZ(9))}} fails with new error when preview off once parser JIRA
exists (can be follow-up test in parser JIRA if parser not merged yet).
h3. Acceptance criteria
* With preview *disabled* (default), any analyzed schema or cast target that
recursively contains {{TimestampNTZNanosType}} or {{TimestampLTZNanosType}}
fails analysis with a clear, documented error (sqlState consistent with other
unsupported-type errors).
* With preview {*}enabled{*}, {{failUnsupportedDataType}} does not block those
types (feature JIRAs may still fail for other reasons, e.g. missing UnsafeRow).
* No change to behavior of {{{}TimestampType{}}}, {{{}TimestampNTZType{}}}, or
zero-arg {{TIMESTAMP}} / {{TIMESTAMP_NTZ}} DDL.
* Conf documented and discoverable via {{SET
spark.sql.timestampNanos.preview.enabled}} (exact key as implemented).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
