This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.2 by this push:
     new ed8fc889e1a7 [SPARK-60482][SQL] Mark `TIME` as a Hive incompatible 
data type
ed8fc889e1a7 is described below

commit ed8fc889e1a7e0c91fa4afa15f13c1b675c82b20
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Thu Oct 22 17:00:00 2026 -0700

    [SPARK-60482][SQL] Mark `TIME` as a Hive incompatible data type
    
    ### What changes were proposed in this pull request?
    
    This PR aims to mark `TIME` as a Hive incompatible data type in 
`HiveExternalCatalog.isHiveCompatibleDataType`, like `TIMESTAMP_NTZ`, ANSI 
intervals, the nanosecond timestamp types, and `VARIANT`.
    
    Hive has no `TIME` type, and `HiveFileFormat.supportDataType` already 
rejects it (SPARK-57556, #56850). However, `isHiveCompatibleDataType` returns 
`true` for `TimeType`, so `HiveExternalCatalog` tries to store `TIME` columns 
in Hive metastore as they are. With this PR,
    - Data source tables with `TIME` columns are stored in the Spark SQL 
specific format without trying the Hive compatible format first.
    - `TIME` partition columns of data source tables are stored with the 
placeholder type, `<derived from deserializer>`, like the other Hive 
incompatible partition columns (SPARK-51418, #50182).
    - Views with `TIME` columns are stored with the empty Hive metastore schema 
without trying the actual schema first.
    
    ### Why are the changes needed?
    
    A data source table with a `TIME` partition column cannot be created in 
Hive metastore.
    
    ```sql
    CREATE TABLE t (i INT, p TIME) USING PARQUET PARTITIONED BY (p);
    ```
    
    ```
    org.apache.spark.sql.AnalysisException: 
org.apache.hadoop.hive.ql.metadata.HiveException: 
InvalidObjectException(message:Invalid partition column type: time(6))
    ```
    
    Hive metastore rejects `time(6)` in the Hive compatible format. Spark falls 
back to the Spark SQL specific format, but it still stores `time(6)` for the 
partition column because the placeholder type is used only for Hive 
incompatible types. `CREATE TABLE AS SELECT` and `DataFrameWriter.saveAsTable` 
with `partitionBy` fail in the same way. Since `CREATE TABLE AS SELECT` writes 
the data before creating the table, it also leaves the data files at the table 
location, so the next attempt f [...]
    
    In addition, a data source table with `TIME` data columns, including nested 
ones like `ARRAY<TIME>`, is created only after a failed attempt in the Hive 
compatible format because the Hive SerDe cannot parse `time(6)`.
    
    ```sql
    CREATE TABLE t (i INT, c TIME) USING PARQUET;
    ```
    
    ```
    WARN HiveExternalCatalog: Could not persist `spark_catalog`.`default`.`t` 
in a Hive compatible way. Persisting it into Hive metastore in Spark SQL 
specific format.
    org.apache.hadoop.hive.ql.metadata.HiveException: 
java.lang.IllegalArgumentException: Error: type expected at the position 4 of 
'int:time(6)' but 'time' is found.
    ```
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes for the `TIME` type, which is disabled by default 
(`spark.sql.timeType.enabled`).
    - Data source tables can have `TIME` partition columns in Hive metastore.
    - For data source tables with `TIME` data columns, Spark logs the following 
instead of the above warning with the stack trace. The table metadata in Hive 
metastore is the same as before.
      ```
      WARN HiveExternalCatalog: Hive incompatible types found: time(6). 
Persisting data source table `spark_catalog`.`default`.`t` into Hive metastore 
in Spark SQL specific format, which is NOT compatible with Hive.
      ```
    - Views with nested `TIME` types, e.g., `ARRAY<TIME>`, have the empty Hive 
metastore schema instead of `array<time(6)>`, which Hive cannot parse. Spark 
restores the view schema from the table properties as before.
    
    ### How was this patch tested?
    
    Pass the CIs with the newly added test cases in 
`MetastoreDataSourcesSuite`. Both test cases fail without this fix.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    Closes #59720 from dongjoon-hyun/SPARK-60482.
    
    Authored-by: Dongjoon Hyun <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
    (cherry picked from commit 4d27ea419a0293072b44e8a199fca096ba151426)
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 .../spark/sql/hive/HiveExternalCatalog.scala       |  1 +
 .../spark/sql/hive/MetastoreDataSourcesSuite.scala | 61 ++++++++++++++++++++++
 2 files changed, 62 insertions(+)

diff --git 
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala 
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala
index 5d3a872d047c..2bf5ef483578 100644
--- 
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala
+++ 
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala
@@ -1541,6 +1541,7 @@ object HiveExternalCatalog {
   private[spark] def isHiveCompatibleDataType(dt: DataType): Boolean = dt 
match {
     case _: AnsiIntervalType => false
     case _: TimestampNTZType => false
+    case _: TimeType => false
     case _: VariantType => false
     case s: StructType =>
       s.forall { f =>
diff --git 
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/MetastoreDataSourcesSuite.scala
 
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/MetastoreDataSourcesSuite.scala
index 3588e6d8567d..dcbc225aa567 100644
--- 
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/MetastoreDataSourcesSuite.scala
+++ 
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/MetastoreDataSourcesSuite.scala
@@ -18,6 +18,7 @@
 package org.apache.spark.sql.hive
 
 import java.io.File
+import java.time.LocalTime
 
 import scala.collection.mutable.ArrayBuffer
 
@@ -1595,6 +1596,66 @@ class MetastoreDataSourcesSuite extends QueryTest
     }
   }
 
+  test("SPARK-60482: TIME columns are stored in Spark SQL specific format") {
+    withTable("t") {
+      val logAppender = new LogAppender("Check that TIME types are reported as 
Hive incompatible")
+      logAppender.setThreshold(Level.WARN)
+      withLogAppender(logAppender) {
+        sql(
+          """
+            |CREATE TABLE t(
+            |  c1 TIME,
+            |  c2 TIME(0),
+            |  c3 ARRAY<TIME(3)>,
+            |  c4 STRUCT<a: INT, b: TIME>,
+            |  c5 MAP<TIME, STRING>
+            |) USING Parquet""".stripMargin)
+      }
+      val expectedMsg = "Hive incompatible types found: time(6), time(0), 
array<time(3)>, " +
+        "struct<a:int,b:time(6)>, map<time(6),string>. " +
+        s"Persisting data source table `$SESSION_CATALOG_NAME`.`default`.`t` 
into Hive " +
+        "metastore in Spark SQL specific format, which is NOT compatible with 
Hive."
+      val actualMessages = logAppender.loggingEvents
+        .map(_.getMessage.getFormattedMessage)
+        .filter(_.contains("incompatible"))
+      assert(actualMessages.contains(expectedMsg))
+      assert(hiveClient.getTable("default", "t").schema
+        .forall(_.dataType == ArrayType(StringType)))
+      assert(sql("SELECT * FROM t").schema ===
+        StructType(Seq(
+          StructField("c1", TimeType()),
+          StructField("c2", TimeType(0)),
+          StructField("c3", ArrayType(TimeType(3))),
+          StructField("c4",
+            StructType(Seq(
+              StructField("a", IntegerType),
+              StructField("b", TimeType())))),
+          StructField("c5", MapType(TimeType(), StringType)))))
+    }
+  }
+
+  test("SPARK-60482: TIME partition columns are stored in Spark SQL specific 
format") {
+    withTable("t") {
+      sql("CREATE TABLE t(id INT, p TIME) USING Parquet PARTITIONED BY (p)")
+      // The partition column is stored with 
INCOMPATIBLE_PARTITION_TYPE_PLACEHOLDER, which the
+      // raw catalog view filters out. The real schema is restored from table 
properties.
+      assert(hiveClient.getTable("default", "t").schema
+        .forall(_.dataType == ArrayType(StringType)))
+      assert(sql("SELECT * FROM t").schema ===
+        StructType(Seq(
+          StructField("id", IntegerType),
+          StructField("p", TimeType()))))
+
+      sql("INSERT INTO t PARTITION (p = TIME'12:34:56') VALUES (1)")
+      sql("INSERT INTO t VALUES (2, TIME'01:02:03.456789')")
+      checkAnswer(sql("SELECT * FROM t"),
+        Seq(Row(1, LocalTime.of(12, 34, 56)), Row(2, LocalTime.of(1, 2, 3, 
456789000))))
+      checkAnswer(sql("SELECT id FROM t WHERE p = TIME'12:34:56'"), Row(1))
+      checkAnswer(sql("SHOW PARTITIONS t"),
+        Seq(Row("p=01%3A02%3A03.456789"), Row("p=12%3A34%3A56")))
+    }
+  }
+
   private def withDebugMode(f: => Unit): Unit = {
     val previousValue = sparkSession.sparkContext.conf.get(DEBUG_MODE)
     try {


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to