This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.2 by this push:
new ed8fc889e1a7 [SPARK-60482][SQL] Mark `TIME` as a Hive incompatible
data type
ed8fc889e1a7 is described below
commit ed8fc889e1a7e0c91fa4afa15f13c1b675c82b20
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Thu Oct 22 17:00:00 2026 -0700
[SPARK-60482][SQL] Mark `TIME` as a Hive incompatible data type
### What changes were proposed in this pull request?
This PR aims to mark `TIME` as a Hive incompatible data type in
`HiveExternalCatalog.isHiveCompatibleDataType`, like `TIMESTAMP_NTZ`, ANSI
intervals, the nanosecond timestamp types, and `VARIANT`.
Hive has no `TIME` type, and `HiveFileFormat.supportDataType` already
rejects it (SPARK-57556, #56850). However, `isHiveCompatibleDataType` returns
`true` for `TimeType`, so `HiveExternalCatalog` tries to store `TIME` columns
in Hive metastore as they are. With this PR,
- Data source tables with `TIME` columns are stored in the Spark SQL
specific format without trying the Hive compatible format first.
- `TIME` partition columns of data source tables are stored with the
placeholder type, `<derived from deserializer>`, like the other Hive
incompatible partition columns (SPARK-51418, #50182).
- Views with `TIME` columns are stored with the empty Hive metastore schema
without trying the actual schema first.
### Why are the changes needed?
A data source table with a `TIME` partition column cannot be created in
Hive metastore.
```sql
CREATE TABLE t (i INT, p TIME) USING PARQUET PARTITIONED BY (p);
```
```
org.apache.spark.sql.AnalysisException:
org.apache.hadoop.hive.ql.metadata.HiveException:
InvalidObjectException(message:Invalid partition column type: time(6))
```
Hive metastore rejects `time(6)` in the Hive compatible format. Spark falls
back to the Spark SQL specific format, but it still stores `time(6)` for the
partition column because the placeholder type is used only for Hive
incompatible types. `CREATE TABLE AS SELECT` and `DataFrameWriter.saveAsTable`
with `partitionBy` fail in the same way. Since `CREATE TABLE AS SELECT` writes
the data before creating the table, it also leaves the data files at the table
location, so the next attempt f [...]
In addition, a data source table with `TIME` data columns, including nested
ones like `ARRAY<TIME>`, is created only after a failed attempt in the Hive
compatible format because the Hive SerDe cannot parse `time(6)`.
```sql
CREATE TABLE t (i INT, c TIME) USING PARQUET;
```
```
WARN HiveExternalCatalog: Could not persist `spark_catalog`.`default`.`t`
in a Hive compatible way. Persisting it into Hive metastore in Spark SQL
specific format.
org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.IllegalArgumentException: Error: type expected at the position 4 of
'int:time(6)' but 'time' is found.
```
### Does this PR introduce _any_ user-facing change?
Yes for the `TIME` type, which is disabled by default
(`spark.sql.timeType.enabled`).
- Data source tables can have `TIME` partition columns in Hive metastore.
- For data source tables with `TIME` data columns, Spark logs the following
instead of the above warning with the stack trace. The table metadata in Hive
metastore is the same as before.
```
WARN HiveExternalCatalog: Hive incompatible types found: time(6).
Persisting data source table `spark_catalog`.`default`.`t` into Hive metastore
in Spark SQL specific format, which is NOT compatible with Hive.
```
- Views with nested `TIME` types, e.g., `ARRAY<TIME>`, have the empty Hive
metastore schema instead of `array<time(6)>`, which Hive cannot parse. Spark
restores the view schema from the table properties as before.
### How was this patch tested?
Pass the CIs with the newly added test cases in
`MetastoreDataSourcesSuite`. Both test cases fail without this fix.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #59720 from dongjoon-hyun/SPARK-60482.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit 4d27ea419a0293072b44e8a199fca096ba151426)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../spark/sql/hive/HiveExternalCatalog.scala | 1 +
.../spark/sql/hive/MetastoreDataSourcesSuite.scala | 61 ++++++++++++++++++++++
2 files changed, 62 insertions(+)
diff --git
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala
index 5d3a872d047c..2bf5ef483578 100644
---
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala
+++
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala
@@ -1541,6 +1541,7 @@ object HiveExternalCatalog {
private[spark] def isHiveCompatibleDataType(dt: DataType): Boolean = dt
match {
case _: AnsiIntervalType => false
case _: TimestampNTZType => false
+ case _: TimeType => false
case _: VariantType => false
case s: StructType =>
s.forall { f =>
diff --git
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/MetastoreDataSourcesSuite.scala
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/MetastoreDataSourcesSuite.scala
index 3588e6d8567d..dcbc225aa567 100644
---
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/MetastoreDataSourcesSuite.scala
+++
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/MetastoreDataSourcesSuite.scala
@@ -18,6 +18,7 @@
package org.apache.spark.sql.hive
import java.io.File
+import java.time.LocalTime
import scala.collection.mutable.ArrayBuffer
@@ -1595,6 +1596,66 @@ class MetastoreDataSourcesSuite extends QueryTest
}
}
+ test("SPARK-60482: TIME columns are stored in Spark SQL specific format") {
+ withTable("t") {
+ val logAppender = new LogAppender("Check that TIME types are reported as
Hive incompatible")
+ logAppender.setThreshold(Level.WARN)
+ withLogAppender(logAppender) {
+ sql(
+ """
+ |CREATE TABLE t(
+ | c1 TIME,
+ | c2 TIME(0),
+ | c3 ARRAY<TIME(3)>,
+ | c4 STRUCT<a: INT, b: TIME>,
+ | c5 MAP<TIME, STRING>
+ |) USING Parquet""".stripMargin)
+ }
+ val expectedMsg = "Hive incompatible types found: time(6), time(0),
array<time(3)>, " +
+ "struct<a:int,b:time(6)>, map<time(6),string>. " +
+ s"Persisting data source table `$SESSION_CATALOG_NAME`.`default`.`t`
into Hive " +
+ "metastore in Spark SQL specific format, which is NOT compatible with
Hive."
+ val actualMessages = logAppender.loggingEvents
+ .map(_.getMessage.getFormattedMessage)
+ .filter(_.contains("incompatible"))
+ assert(actualMessages.contains(expectedMsg))
+ assert(hiveClient.getTable("default", "t").schema
+ .forall(_.dataType == ArrayType(StringType)))
+ assert(sql("SELECT * FROM t").schema ===
+ StructType(Seq(
+ StructField("c1", TimeType()),
+ StructField("c2", TimeType(0)),
+ StructField("c3", ArrayType(TimeType(3))),
+ StructField("c4",
+ StructType(Seq(
+ StructField("a", IntegerType),
+ StructField("b", TimeType())))),
+ StructField("c5", MapType(TimeType(), StringType)))))
+ }
+ }
+
+ test("SPARK-60482: TIME partition columns are stored in Spark SQL specific
format") {
+ withTable("t") {
+ sql("CREATE TABLE t(id INT, p TIME) USING Parquet PARTITIONED BY (p)")
+ // The partition column is stored with
INCOMPATIBLE_PARTITION_TYPE_PLACEHOLDER, which the
+ // raw catalog view filters out. The real schema is restored from table
properties.
+ assert(hiveClient.getTable("default", "t").schema
+ .forall(_.dataType == ArrayType(StringType)))
+ assert(sql("SELECT * FROM t").schema ===
+ StructType(Seq(
+ StructField("id", IntegerType),
+ StructField("p", TimeType()))))
+
+ sql("INSERT INTO t PARTITION (p = TIME'12:34:56') VALUES (1)")
+ sql("INSERT INTO t VALUES (2, TIME'01:02:03.456789')")
+ checkAnswer(sql("SELECT * FROM t"),
+ Seq(Row(1, LocalTime.of(12, 34, 56)), Row(2, LocalTime.of(1, 2, 3,
456789000))))
+ checkAnswer(sql("SELECT id FROM t WHERE p = TIME'12:34:56'"), Row(1))
+ checkAnswer(sql("SHOW PARTITIONS t"),
+ Seq(Row("p=01%3A02%3A03.456789"), Row("p=12%3A34%3A56")))
+ }
+ }
+
private def withDebugMode(f: => Unit): Unit = {
val previousValue = sparkSession.sparkContext.conf.get(DEBUG_MODE)
try {
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]