This is an automated email from the ASF dual-hosted git repository.

uros-b pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new 5ab0537be5d3 [SPARK-57835][SQL] Read persisted nanosecond-typed tables 
when the preview flag is off
5ab0537be5d3 is described below

commit 5ab0537be5d3c3934bda4699268f94e858244dfb
Author: Stevo Mitric <[email protected]>
AuthorDate: Sun Aug 2 16:49:32 2026 +0000

    [SPARK-57835][SQL] Read persisted nanosecond-typed tables when the preview 
flag is off
    
    ### What changes were proposed in this pull request?
    Defines read-through behavior for persisted tables with nanosecond 
timestamp columns (TIMESTAMP_NTZ(p)/TIMESTAMP_LTZ(p), p in [7,9]) when 
spark.sql.timestampNanosTypes.enabled is off. DataType.fromJson now 
reconstructs nanos types regardless of the flag (removes the two 
checkTimestampNanosTypesEnabled() calls in DataType.nameToType), mirroring the 
TIME type. The SQL parser path and analysis-time gate 
(TypeUtils.failUnsupportedDataType) stay gated.
    
    ### Why are the changes needed?
    With the flag off, restoring a persisted nanos table's schema went through 
DataType.fromJson and threw FEATURE_NOT_ENABLED, so DESCRIBE, SHOW CREATE 
TABLE, and even DROP TABLE failed — the table was completely unmanageable. TIME 
types don't have this problem. Read-through fixes it while still blocking data 
access until the flag is on.
    
    ### Does this PR introduce _any_ user-facing change?
    Yes, within unreleased master (preview feature under SPARK-56822). With the 
flag off: metadata/DDL commands now succeed and render the nanos columns 
(DESCRIBE shows timestamp_ntz(9)), and the table can be dropped; reading the 
data (SELECT *) still fails with an actionable FEATURE_NOT_ENABLED.
    
    Previously all of these failed.
    
    ### How was this patch tested?
    
    New/updated unit tests: DataTypeSuite (fromJson read-through), 
HiveExternalCatalogSuite (catalog restore with flag off), HiveDDLSuite 
(DESCRIBE / SHOW CREATE / DROP / view persistence succeed, SELECT * fails). All 
green.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Co-authored-by: Claude Code (Claude Opus 4.8)
    
    Closes #57700 from stevomitric/stevomitric/spark-57835-nanos-read.
    
    Authored-by: Stevo Mitric <[email protected]>
    Signed-off-by: Uros Bojanic <[email protected]>
    (cherry picked from commit dd3a24207f4c8ead61dfb2e3701a5859b575fd32)
    Signed-off-by: Uros Bojanic <[email protected]>
---
 .../org/apache/spark/sql/types/DataType.scala      | 20 ++++---
 .../org/apache/spark/sql/types/DataTypeSuite.scala | 31 ++++++-----
 .../spark/sql/hive/HiveExternalCatalogSuite.scala  | 50 ++++++++++++++++-
 .../spark/sql/hive/execution/HiveDDLSuite.scala    | 62 ++++++++++++++++++++++
 4 files changed, 142 insertions(+), 21 deletions(-)

diff --git a/sql/api/src/main/scala/org/apache/spark/sql/types/DataType.scala 
b/sql/api/src/main/scala/org/apache/spark/sql/types/DataType.scala
index dc79267e120c..91e6eade4eae 100644
--- a/sql/api/src/main/scala/org/apache/spark/sql/types/DataType.scala
+++ b/sql/api/src/main/scala/org/apache/spark/sql/types/DataType.scala
@@ -285,11 +285,18 @@ object DataType {
           TimestampType
         } else if (p < TimestampLTZNanosType.MIN_PRECISION ||
           p > TimestampLTZNanosType.MAX_PRECISION) {
-          // Reject out-of-range precisions before the feature-flag check so 
the error is always
-          // INVALID_TIMESTAMP_PRECISION, not FEATURE_NOT_ENABLED.
+          // Reject out-of-range precisions so the error is always 
INVALID_TIMESTAMP_PRECISION.
           throw DataTypeErrors.invalidTimestampPrecisionError(precision, 
"TIMESTAMP_LTZ")
         } else {
-          DataTypeErrors.checkTimestampNanosTypesEnabled()
+          // The nanos preview flag is intentionally NOT enforced here 
(SPARK-57835). This JSON
+          // path is how a persisted schema is reconstructed from the catalog 
(e.g.
+          // HiveExternalCatalog.getTable -> DataType.fromJson), so gating it 
would make a table
+          // written with the flag on completely inaccessible once it is off 
-- DESCRIBE, SHOW
+          // CREATE TABLE, and even DROP would fail. Instead we mirror TIME 
(also flag-gated but
+          // reconstructed unconditionally): metadata reads succeed, and the 
flag is enforced at
+          // analysis/execution time via TypeUtils.failUnsupportedDataType 
when the data is
+          // actually read, written, or queried. The user-facing SQL parser 
path
+          // (DataTypeAstBuilder) stays gated, so DDL like TIMESTAMP_LTZ(9) 
still fails fast.
           TimestampLTZNanosType(p)
         }
       case TIMESTAMP_NTZ_NANOS_TYPE(precision) =>
@@ -305,11 +312,12 @@ object DataType {
           TimestampNTZType
         } else if (p < TimestampNTZNanosType.MIN_PRECISION ||
           p > TimestampNTZNanosType.MAX_PRECISION) {
-          // Reject out-of-range precisions before the feature-flag check so 
the error is always
-          // INVALID_TIMESTAMP_PRECISION, not FEATURE_NOT_ENABLED.
+          // Reject out-of-range precisions so the error is always 
INVALID_TIMESTAMP_PRECISION.
           throw DataTypeErrors.invalidTimestampPrecisionError(precision, 
"TIMESTAMP_NTZ")
         } else {
-          DataTypeErrors.checkTimestampNanosTypesEnabled()
+          // Not flag-gated on purpose (SPARK-57835); see the TIMESTAMP_LTZ 
branch above for the
+          // rationale (catalog restoration must be able to reconstruct 
persisted nanos schemas
+          // regardless of the preview flag).
           TimestampNTZNanosType(p)
         }
       case "timestamp_ltz" => TimestampType
diff --git 
a/sql/catalyst/src/test/scala/org/apache/spark/sql/types/DataTypeSuite.scala 
b/sql/catalyst/src/test/scala/org/apache/spark/sql/types/DataTypeSuite.scala
index 285d840eed6c..6403e1509858 100644
--- a/sql/catalyst/src/test/scala/org/apache/spark/sql/types/DataTypeSuite.scala
+++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/types/DataTypeSuite.scala
@@ -1656,22 +1656,27 @@ class DataTypeSuite extends SparkFunSuite with 
SQLHelper {
     }
   }
 
-  test("SPARK-56965: JSON parser rejects nanos timestamp types when preview 
flag is off") {
+  test("SPARK-57835: JSON parser reconstructs nanos timestamp types when 
preview flag is off") {
+    // Read-through policy: unlike the SQL parser (DataTypeAstBuilder), the 
JSON path is how a
+    // persisted schema is restored from the catalog, so it must reconstruct 
nanos types even
+    // when the preview flag is off. Otherwise a table written with the flag 
on would become
+    // completely inaccessible (DESCRIBE / SHOW CREATE TABLE / DROP) once it 
is off. The flag is
+    // instead enforced at analysis/execution time 
(TypeUtils.failUnsupportedDataType). This
+    // mirrors how TIME types round-trip through fromJson regardless of their 
own flag.
     withSQLConf(SQLConf.TIMESTAMP_NANOS_TYPES_ENABLED.key -> "false") {
       Seq(
-        "\"timestamp_ltz(7)\"" -> "Nanosecond-precision timestamp types",
-        "\"timestamp_ntz(9)\"" -> "Nanosecond-precision timestamp 
types").foreach {
-        case (json, featureName) =>
-          checkError(
-            exception = intercept[SparkException] {
-              DataType.fromJson(json)
-            },
-            condition = "FEATURE_NOT_ENABLED",
-            parameters = Map(
-              "featureName" -> featureName,
-              "configKey" -> "spark.sql.timestampNanosTypes.enabled",
-              "configValue" -> "true"))
+        "\"timestamp_ltz(7)\"" -> TimestampLTZNanosType(7),
+        "\"timestamp_ntz(9)\"" -> TimestampNTZNanosType(9)).foreach {
+        case (json, expected) =>
+          assert(DataType.fromJson(json) === expected)
       }
+      // Nested nanos types (inside struct/array/map) also reconstruct with 
the flag off, since
+      // that is exactly how catalog schemas are shaped.
+      val nested = StructType(Seq(
+        StructField("s", StructType(Seq(StructField("ntz", 
TimestampNTZNanosType(7))))),
+        StructField("a", ArrayType(TimestampLTZNanosType(8), containsNull = 
false)),
+        StructField("m", MapType(StringType, TimestampNTZNanosType(9)))))
+      assert(DataType.fromJson(nested.json) === nested)
       // Precision 6 maps to the GA types and stays accepted with the gate off.
       assert(DataType.fromJson("\"timestamp_ltz(6)\"") === TimestampType)
       assert(DataType.fromJson("\"timestamp_ntz(6)\"") === TimestampNTZType)
diff --git 
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogSuite.scala
 
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogSuite.scala
index db522b72e4cc..fb74d83c04d9 100644
--- 
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogSuite.scala
+++ 
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogSuite.scala
@@ -23,15 +23,17 @@ import org.apache.logging.log4j.Level
 import org.apache.spark.SparkConf
 import org.apache.spark.sql.catalyst.TableIdentifier
 import org.apache.spark.sql.catalyst.catalog._
+import org.apache.spark.sql.catalyst.plans.SQLHelper
 import org.apache.spark.sql.catalyst.types.DataTypeUtils
 import org.apache.spark.sql.execution.QueryExecutionException
 import org.apache.spark.sql.execution.command.DDLUtils
-import org.apache.spark.sql.types.{StringType, StructField, StructType}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{IntegerType, StringType, StructField, 
StructType, TimestampLTZNanosType, TimestampNTZNanosType}
 
 /**
  * Test suite for the [[HiveExternalCatalog]].
  */
-class HiveExternalCatalogSuite extends ExternalCatalogSuite {
+class HiveExternalCatalogSuite extends ExternalCatalogSuite with SQLHelper {
 
   private val externalCatalog: HiveExternalCatalog = {
     val catalog = new HiveExternalCatalog(new SparkConf, new Configuration)
@@ -243,6 +245,50 @@ class HiveExternalCatalogSuite extends 
ExternalCatalogSuite {
     assert(DataTypeUtils.sameType(alteredTable.schema, newSchema))
   }
 
+  test("SPARK-57835: restore a persisted nanos-typed table when the preview 
flag is off") {
+    val catalog = newBasicCatalog()
+    val tableName = "nanos_tbl"
+
+    // Nanos types are not Hive-compatible (SPARK-57831), so a datasource 
table persists an empty
+    // schema in the metastore and the real schema as JSON in the table 
properties. The table is
+    // created with the preview flag on (the test default via Utils.isTesting).
+    val nanosSchema = StructType(Seq(
+      StructField("id", IntegerType),
+      StructField("ntz", TimestampNTZNanosType(9)),
+      StructField("ltz", TimestampLTZNanosType(7))))
+
+    val tableDDL = CatalogTable(
+      identifier = TableIdentifier(tableName, Some("db1")),
+      tableType = CatalogTableType.MANAGED,
+      storage = storageFormat,
+      schema = nanosSchema,
+      provider = Some("parquet"))
+
+    catalog.createTable(tableDDL, ignoreIfExists = false)
+
+    // Because nanos types are not Hive-compatible, the metastore-visible 
(raw) schema is the
+    // placeholder EMPTY_DATA_SCHEMA; the true schema lives in the 
Spark-specific table properties.
+    val rawTable = externalCatalog.getRawTable("db1", tableName)
+    assert(rawTable.schema == HiveExternalCatalog.EMPTY_DATA_SCHEMA)
+
+    // Read-through policy (SPARK-57835): with the preview flag off, catalog 
restoration must still
+    // reconstruct the persisted nanos schema so the table remains describable 
and droppable.
+    // Before this change getTable threw FEATURE_NOT_ENABLED here.
+    withSQLConf(SQLConf.TIMESTAMP_NANOS_TYPES_ENABLED.key -> "false") {
+      val restored = externalCatalog.getTable("db1", tableName)
+      assert(DataTypeUtils.sameType(restored.schema, nanosSchema))
+      // The restored nanos columns render with their precision (used by 
DESCRIBE / SHOW CREATE).
+      assert(restored.schema("ntz").dataType === TimestampNTZNanosType(9))
+      assert(restored.schema("ltz").dataType === TimestampLTZNanosType(7))
+    }
+
+    // And it still round-trips with the flag on.
+    withSQLConf(SQLConf.TIMESTAMP_NANOS_TYPES_ENABLED.key -> "true") {
+      assert(DataTypeUtils.sameType(
+        externalCatalog.getTable("db1", tableName).schema, nanosSchema))
+    }
+  }
+
   test("SPARK-50137: Avoid fallback to Hive-incompatible ways on thrift 
exception") {
     val hadoopConf = new Configuration()
     // Use an unavailable uri to mock client connection timeout.
diff --git 
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
 
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
index 6ce4e8702c32..89f96d5627ea 100644
--- 
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
+++ 
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
@@ -3417,4 +3417,66 @@ class HiveDDLSuite
         any[String], any[String], any[StructType])
     }
   }
+
+  test("SPARK-57835: read persisted nanos-typed tables when the preview flag 
is off") {
+    withTable("nanos_ddl_tbl") {
+      // Create the table with the preview flag on (the test default via 
Utils.isTesting).
+      sql(
+        """CREATE TABLE nanos_ddl_tbl (id INT, ntz TIMESTAMP_NTZ(9), ltz 
TIMESTAMP_LTZ(7))
+          |USING parquet""".stripMargin)
+
+      withSQLConf(SQLConf.TIMESTAMP_NANOS_TYPES_ENABLED.key -> "false") {
+        // Read-through policy (SPARK-57835): metadata reads succeed and 
render the nanos columns
+        // with their precision, even though the preview flag is off. Before 
this change these
+        // commands failed at getTable time with FEATURE_NOT_ENABLED.
+        val describeRows = sql("DESCRIBE TABLE nanos_ddl_tbl").collect()
+          .map(r => r.getString(0) -> r.getString(1)).toMap
+        assert(describeRows("ntz") === "timestamp_ntz(9)")
+        assert(describeRows("ltz") === "timestamp_ltz(7)")
+
+        val showCreate = sql("SHOW CREATE TABLE 
nanos_ddl_tbl").head().getString(0)
+        assert(showCreate.contains("TIMESTAMP_NTZ(9)"))
+        assert(showCreate.contains("TIMESTAMP_LTZ(7)"))
+
+        // But actually reading the data still fails with an actionable, 
feature-flag error.
+        checkError(
+          exception = intercept[SparkException] {
+            sql("SELECT * FROM nanos_ddl_tbl").collect()
+          },
+          condition = "FEATURE_NOT_ENABLED",
+          parameters = Map(
+            "featureName" -> "Nanosecond-precision timestamp types",
+            "configKey" -> "spark.sql.timestampNanosTypes.enabled",
+            "configValue" -> "true"))
+
+        // The table remains droppable with the flag off (a key reason for 
read-through: a table
+        // written with the flag on must never become un-manageable once it is 
off).
+        sql("DROP TABLE nanos_ddl_tbl")
+        
assert(!spark.sessionState.catalog.tableExists(TableIdentifier("nanos_ddl_tbl")))
+      }
+    }
+  }
+
+  test("SPARK-57835: read a persisted view over a nanos column when the 
preview flag is off") {
+    withTable("nanos_view_base") {
+      withView("nanos_view") {
+        sql("CREATE TABLE nanos_view_base (id INT, ntz TIMESTAMP_NTZ(9)) USING 
parquet")
+        // The view persists its analyzed output schema (which includes the 
nanos column) into
+        // table properties; this is created with the flag on.
+        sql("CREATE VIEW nanos_view AS SELECT id, ntz FROM nanos_view_base")
+
+        withSQLConf(SQLConf.TIMESTAMP_NANOS_TYPES_ENABLED.key -> "false") {
+          // Restoring the view schema from properties (DataType.fromJson) 
succeeds, so DESCRIBE
+          // renders the nanos column.
+          val describeRows = sql("DESCRIBE TABLE nanos_view").collect()
+            .map(r => r.getString(0) -> r.getString(1)).toMap
+          assert(describeRows("ntz") === "timestamp_ntz(9)")
+
+          // The view can still be dropped with the flag off.
+          sql("DROP VIEW nanos_view")
+          
assert(!spark.sessionState.catalog.tableExists(TableIdentifier("nanos_view")))
+        }
+      }
+    }
+  }
 }


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to