This is an automated email from the ASF dual-hosted git repository.
uros-b pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 5ab0537be5d3 [SPARK-57835][SQL] Read persisted nanosecond-typed tables
when the preview flag is off
5ab0537be5d3 is described below
commit 5ab0537be5d3c3934bda4699268f94e858244dfb
Author: Stevo Mitric <[email protected]>
AuthorDate: Sun Aug 2 16:49:32 2026 +0000
[SPARK-57835][SQL] Read persisted nanosecond-typed tables when the preview
flag is off
### What changes were proposed in this pull request?
Defines read-through behavior for persisted tables with nanosecond
timestamp columns (TIMESTAMP_NTZ(p)/TIMESTAMP_LTZ(p), p in [7,9]) when
spark.sql.timestampNanosTypes.enabled is off. DataType.fromJson now
reconstructs nanos types regardless of the flag (removes the two
checkTimestampNanosTypesEnabled() calls in DataType.nameToType), mirroring the
TIME type. The SQL parser path and analysis-time gate
(TypeUtils.failUnsupportedDataType) stay gated.
### Why are the changes needed?
With the flag off, restoring a persisted nanos table's schema went through
DataType.fromJson and threw FEATURE_NOT_ENABLED, so DESCRIBE, SHOW CREATE
TABLE, and even DROP TABLE failed — the table was completely unmanageable. TIME
types don't have this problem. Read-through fixes it while still blocking data
access until the flag is on.
### Does this PR introduce _any_ user-facing change?
Yes, within unreleased master (preview feature under SPARK-56822). With the
flag off: metadata/DDL commands now succeed and render the nanos columns
(DESCRIBE shows timestamp_ntz(9)), and the table can be dropped; reading the
data (SELECT *) still fails with an actionable FEATURE_NOT_ENABLED.
Previously all of these failed.
### How was this patch tested?
New/updated unit tests: DataTypeSuite (fromJson read-through),
HiveExternalCatalogSuite (catalog restore with flag off), HiveDDLSuite
(DESCRIBE / SHOW CREATE / DROP / view persistence succeed, SELECT * fails). All
green.
### Was this patch authored or co-authored using generative AI tooling?
Co-authored-by: Claude Code (Claude Opus 4.8)
Closes #57700 from stevomitric/stevomitric/spark-57835-nanos-read.
Authored-by: Stevo Mitric <[email protected]>
Signed-off-by: Uros Bojanic <[email protected]>
(cherry picked from commit dd3a24207f4c8ead61dfb2e3701a5859b575fd32)
Signed-off-by: Uros Bojanic <[email protected]>
---
.../org/apache/spark/sql/types/DataType.scala | 20 ++++---
.../org/apache/spark/sql/types/DataTypeSuite.scala | 31 ++++++-----
.../spark/sql/hive/HiveExternalCatalogSuite.scala | 50 ++++++++++++++++-
.../spark/sql/hive/execution/HiveDDLSuite.scala | 62 ++++++++++++++++++++++
4 files changed, 142 insertions(+), 21 deletions(-)
diff --git a/sql/api/src/main/scala/org/apache/spark/sql/types/DataType.scala
b/sql/api/src/main/scala/org/apache/spark/sql/types/DataType.scala
index dc79267e120c..91e6eade4eae 100644
--- a/sql/api/src/main/scala/org/apache/spark/sql/types/DataType.scala
+++ b/sql/api/src/main/scala/org/apache/spark/sql/types/DataType.scala
@@ -285,11 +285,18 @@ object DataType {
TimestampType
} else if (p < TimestampLTZNanosType.MIN_PRECISION ||
p > TimestampLTZNanosType.MAX_PRECISION) {
- // Reject out-of-range precisions before the feature-flag check so
the error is always
- // INVALID_TIMESTAMP_PRECISION, not FEATURE_NOT_ENABLED.
+ // Reject out-of-range precisions so the error is always
INVALID_TIMESTAMP_PRECISION.
throw DataTypeErrors.invalidTimestampPrecisionError(precision,
"TIMESTAMP_LTZ")
} else {
- DataTypeErrors.checkTimestampNanosTypesEnabled()
+ // The nanos preview flag is intentionally NOT enforced here
(SPARK-57835). This JSON
+ // path is how a persisted schema is reconstructed from the catalog
(e.g.
+ // HiveExternalCatalog.getTable -> DataType.fromJson), so gating it
would make a table
+ // written with the flag on completely inaccessible once it is off
-- DESCRIBE, SHOW
+ // CREATE TABLE, and even DROP would fail. Instead we mirror TIME
(also flag-gated but
+ // reconstructed unconditionally): metadata reads succeed, and the
flag is enforced at
+ // analysis/execution time via TypeUtils.failUnsupportedDataType
when the data is
+ // actually read, written, or queried. The user-facing SQL parser
path
+ // (DataTypeAstBuilder) stays gated, so DDL like TIMESTAMP_LTZ(9)
still fails fast.
TimestampLTZNanosType(p)
}
case TIMESTAMP_NTZ_NANOS_TYPE(precision) =>
@@ -305,11 +312,12 @@ object DataType {
TimestampNTZType
} else if (p < TimestampNTZNanosType.MIN_PRECISION ||
p > TimestampNTZNanosType.MAX_PRECISION) {
- // Reject out-of-range precisions before the feature-flag check so
the error is always
- // INVALID_TIMESTAMP_PRECISION, not FEATURE_NOT_ENABLED.
+ // Reject out-of-range precisions so the error is always
INVALID_TIMESTAMP_PRECISION.
throw DataTypeErrors.invalidTimestampPrecisionError(precision,
"TIMESTAMP_NTZ")
} else {
- DataTypeErrors.checkTimestampNanosTypesEnabled()
+ // Not flag-gated on purpose (SPARK-57835); see the TIMESTAMP_LTZ
branch above for the
+ // rationale (catalog restoration must be able to reconstruct
persisted nanos schemas
+ // regardless of the preview flag).
TimestampNTZNanosType(p)
}
case "timestamp_ltz" => TimestampType
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/types/DataTypeSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/types/DataTypeSuite.scala
index 285d840eed6c..6403e1509858 100644
--- a/sql/catalyst/src/test/scala/org/apache/spark/sql/types/DataTypeSuite.scala
+++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/types/DataTypeSuite.scala
@@ -1656,22 +1656,27 @@ class DataTypeSuite extends SparkFunSuite with
SQLHelper {
}
}
- test("SPARK-56965: JSON parser rejects nanos timestamp types when preview
flag is off") {
+ test("SPARK-57835: JSON parser reconstructs nanos timestamp types when
preview flag is off") {
+ // Read-through policy: unlike the SQL parser (DataTypeAstBuilder), the
JSON path is how a
+ // persisted schema is restored from the catalog, so it must reconstruct
nanos types even
+ // when the preview flag is off. Otherwise a table written with the flag
on would become
+ // completely inaccessible (DESCRIBE / SHOW CREATE TABLE / DROP) once it
is off. The flag is
+ // instead enforced at analysis/execution time
(TypeUtils.failUnsupportedDataType). This
+ // mirrors how TIME types round-trip through fromJson regardless of their
own flag.
withSQLConf(SQLConf.TIMESTAMP_NANOS_TYPES_ENABLED.key -> "false") {
Seq(
- "\"timestamp_ltz(7)\"" -> "Nanosecond-precision timestamp types",
- "\"timestamp_ntz(9)\"" -> "Nanosecond-precision timestamp
types").foreach {
- case (json, featureName) =>
- checkError(
- exception = intercept[SparkException] {
- DataType.fromJson(json)
- },
- condition = "FEATURE_NOT_ENABLED",
- parameters = Map(
- "featureName" -> featureName,
- "configKey" -> "spark.sql.timestampNanosTypes.enabled",
- "configValue" -> "true"))
+ "\"timestamp_ltz(7)\"" -> TimestampLTZNanosType(7),
+ "\"timestamp_ntz(9)\"" -> TimestampNTZNanosType(9)).foreach {
+ case (json, expected) =>
+ assert(DataType.fromJson(json) === expected)
}
+ // Nested nanos types (inside struct/array/map) also reconstruct with
the flag off, since
+ // that is exactly how catalog schemas are shaped.
+ val nested = StructType(Seq(
+ StructField("s", StructType(Seq(StructField("ntz",
TimestampNTZNanosType(7))))),
+ StructField("a", ArrayType(TimestampLTZNanosType(8), containsNull =
false)),
+ StructField("m", MapType(StringType, TimestampNTZNanosType(9)))))
+ assert(DataType.fromJson(nested.json) === nested)
// Precision 6 maps to the GA types and stays accepted with the gate off.
assert(DataType.fromJson("\"timestamp_ltz(6)\"") === TimestampType)
assert(DataType.fromJson("\"timestamp_ntz(6)\"") === TimestampNTZType)
diff --git
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogSuite.scala
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogSuite.scala
index db522b72e4cc..fb74d83c04d9 100644
---
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogSuite.scala
+++
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogSuite.scala
@@ -23,15 +23,17 @@ import org.apache.logging.log4j.Level
import org.apache.spark.SparkConf
import org.apache.spark.sql.catalyst.TableIdentifier
import org.apache.spark.sql.catalyst.catalog._
+import org.apache.spark.sql.catalyst.plans.SQLHelper
import org.apache.spark.sql.catalyst.types.DataTypeUtils
import org.apache.spark.sql.execution.QueryExecutionException
import org.apache.spark.sql.execution.command.DDLUtils
-import org.apache.spark.sql.types.{StringType, StructField, StructType}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{IntegerType, StringType, StructField,
StructType, TimestampLTZNanosType, TimestampNTZNanosType}
/**
* Test suite for the [[HiveExternalCatalog]].
*/
-class HiveExternalCatalogSuite extends ExternalCatalogSuite {
+class HiveExternalCatalogSuite extends ExternalCatalogSuite with SQLHelper {
private val externalCatalog: HiveExternalCatalog = {
val catalog = new HiveExternalCatalog(new SparkConf, new Configuration)
@@ -243,6 +245,50 @@ class HiveExternalCatalogSuite extends
ExternalCatalogSuite {
assert(DataTypeUtils.sameType(alteredTable.schema, newSchema))
}
+ test("SPARK-57835: restore a persisted nanos-typed table when the preview
flag is off") {
+ val catalog = newBasicCatalog()
+ val tableName = "nanos_tbl"
+
+ // Nanos types are not Hive-compatible (SPARK-57831), so a datasource
table persists an empty
+ // schema in the metastore and the real schema as JSON in the table
properties. The table is
+ // created with the preview flag on (the test default via Utils.isTesting).
+ val nanosSchema = StructType(Seq(
+ StructField("id", IntegerType),
+ StructField("ntz", TimestampNTZNanosType(9)),
+ StructField("ltz", TimestampLTZNanosType(7))))
+
+ val tableDDL = CatalogTable(
+ identifier = TableIdentifier(tableName, Some("db1")),
+ tableType = CatalogTableType.MANAGED,
+ storage = storageFormat,
+ schema = nanosSchema,
+ provider = Some("parquet"))
+
+ catalog.createTable(tableDDL, ignoreIfExists = false)
+
+ // Because nanos types are not Hive-compatible, the metastore-visible
(raw) schema is the
+ // placeholder EMPTY_DATA_SCHEMA; the true schema lives in the
Spark-specific table properties.
+ val rawTable = externalCatalog.getRawTable("db1", tableName)
+ assert(rawTable.schema == HiveExternalCatalog.EMPTY_DATA_SCHEMA)
+
+ // Read-through policy (SPARK-57835): with the preview flag off, catalog
restoration must still
+ // reconstruct the persisted nanos schema so the table remains describable
and droppable.
+ // Before this change getTable threw FEATURE_NOT_ENABLED here.
+ withSQLConf(SQLConf.TIMESTAMP_NANOS_TYPES_ENABLED.key -> "false") {
+ val restored = externalCatalog.getTable("db1", tableName)
+ assert(DataTypeUtils.sameType(restored.schema, nanosSchema))
+ // The restored nanos columns render with their precision (used by
DESCRIBE / SHOW CREATE).
+ assert(restored.schema("ntz").dataType === TimestampNTZNanosType(9))
+ assert(restored.schema("ltz").dataType === TimestampLTZNanosType(7))
+ }
+
+ // And it still round-trips with the flag on.
+ withSQLConf(SQLConf.TIMESTAMP_NANOS_TYPES_ENABLED.key -> "true") {
+ assert(DataTypeUtils.sameType(
+ externalCatalog.getTable("db1", tableName).schema, nanosSchema))
+ }
+ }
+
test("SPARK-50137: Avoid fallback to Hive-incompatible ways on thrift
exception") {
val hadoopConf = new Configuration()
// Use an unavailable uri to mock client connection timeout.
diff --git
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
index 6ce4e8702c32..89f96d5627ea 100644
---
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
+++
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
@@ -3417,4 +3417,66 @@ class HiveDDLSuite
any[String], any[String], any[StructType])
}
}
+
+ test("SPARK-57835: read persisted nanos-typed tables when the preview flag
is off") {
+ withTable("nanos_ddl_tbl") {
+ // Create the table with the preview flag on (the test default via
Utils.isTesting).
+ sql(
+ """CREATE TABLE nanos_ddl_tbl (id INT, ntz TIMESTAMP_NTZ(9), ltz
TIMESTAMP_LTZ(7))
+ |USING parquet""".stripMargin)
+
+ withSQLConf(SQLConf.TIMESTAMP_NANOS_TYPES_ENABLED.key -> "false") {
+ // Read-through policy (SPARK-57835): metadata reads succeed and
render the nanos columns
+ // with their precision, even though the preview flag is off. Before
this change these
+ // commands failed at getTable time with FEATURE_NOT_ENABLED.
+ val describeRows = sql("DESCRIBE TABLE nanos_ddl_tbl").collect()
+ .map(r => r.getString(0) -> r.getString(1)).toMap
+ assert(describeRows("ntz") === "timestamp_ntz(9)")
+ assert(describeRows("ltz") === "timestamp_ltz(7)")
+
+ val showCreate = sql("SHOW CREATE TABLE
nanos_ddl_tbl").head().getString(0)
+ assert(showCreate.contains("TIMESTAMP_NTZ(9)"))
+ assert(showCreate.contains("TIMESTAMP_LTZ(7)"))
+
+ // But actually reading the data still fails with an actionable,
feature-flag error.
+ checkError(
+ exception = intercept[SparkException] {
+ sql("SELECT * FROM nanos_ddl_tbl").collect()
+ },
+ condition = "FEATURE_NOT_ENABLED",
+ parameters = Map(
+ "featureName" -> "Nanosecond-precision timestamp types",
+ "configKey" -> "spark.sql.timestampNanosTypes.enabled",
+ "configValue" -> "true"))
+
+ // The table remains droppable with the flag off (a key reason for
read-through: a table
+ // written with the flag on must never become un-manageable once it is
off).
+ sql("DROP TABLE nanos_ddl_tbl")
+
assert(!spark.sessionState.catalog.tableExists(TableIdentifier("nanos_ddl_tbl")))
+ }
+ }
+ }
+
+ test("SPARK-57835: read a persisted view over a nanos column when the
preview flag is off") {
+ withTable("nanos_view_base") {
+ withView("nanos_view") {
+ sql("CREATE TABLE nanos_view_base (id INT, ntz TIMESTAMP_NTZ(9)) USING
parquet")
+ // The view persists its analyzed output schema (which includes the
nanos column) into
+ // table properties; this is created with the flag on.
+ sql("CREATE VIEW nanos_view AS SELECT id, ntz FROM nanos_view_base")
+
+ withSQLConf(SQLConf.TIMESTAMP_NANOS_TYPES_ENABLED.key -> "false") {
+ // Restoring the view schema from properties (DataType.fromJson)
succeeds, so DESCRIBE
+ // renders the nanos column.
+ val describeRows = sql("DESCRIBE TABLE nanos_view").collect()
+ .map(r => r.getString(0) -> r.getString(1)).toMap
+ assert(describeRows("ntz") === "timestamp_ntz(9)")
+
+ // The view can still be dropped with the flag off.
+ sql("DROP VIEW nanos_view")
+
assert(!spark.sessionState.catalog.tableExists(TableIdentifier("nanos_view")))
+ }
+ }
+ }
+ }
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]