[
https://issues.apache.org/jira/browse/SPARK-57831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk updated SPARK-57831:
-----------------------------
Shepherd: Max Gekk
> Align Hive-metastore compatibility classification for nanosecond timestamp
> types
> --------------------------------------------------------------------------------
>
> Key: SPARK-57831
> URL: https://issues.apache.org/jira/browse/SPARK-57831
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 4.3.0
> Reporter: Max Gekk
> Priority: Major
>
> This sub-task is part of the umbrella SPARK-56822 (timestamps with nanosecond
> precision).
> h2. Problem
> {{HiveExternalCatalog.isHiveCompatibleDataType}} (~L1543-1558) returns false
> for microsecond {{TimestampNTZType}}, but the nanosecond timestamp types fall
> through to {{case _ => true}}, so a nanosecond-column table may take the
> Hive-compatible metastore path and store {{timestamp_ntz(9)}} /
> {{timestamp_ltz(9)}} in the HMS {{FieldSchema}} (via {{toHiveColumn}} /
> {{catalogString}}) - unlike microsecond NTZ, which stores its schema in table
> properties. The USING-datasource schema itself round-trips via the
> {{spark.sql.sources.schema}} JSON, but the HMS {{FieldSchema}} type strings
> may be non-standard for Hive.
> h2. Goal
> Treat nanosecond types like {{TimestampNTZType}} ({{isHiveCompatibleDataType
> = false}}) so the schema is stored in table properties, or explicitly justify
> and test the {{FieldSchema}} round-trip.
> h2. Scope
> Add nanosecond arms to {{isHiveCompatibleDataType}}; add HMS {{toHiveColumn}}
> <-> {{fromHiveColumn}} round-trip tests; add a {{CREATE TABLE ... USING
> parquet}} metastore reload e2e test.
> h2. Acceptance criteria
> * Nanosecond-column tables persist/reload from HMS correctly; classification
> matches microsecond NTZ.
> h2. Testing
> {{HiveExternalCatalogSuite}}, {{MetastoreDataSourcesSuite}}.
> h2. Dependencies
> Relates to the preview-flag metastore-read-policy sub-task (same
> catalog-persistence area); can land independently.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
