This is an automated email from the ASF dual-hosted git repository.
MaxGekk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new e80f4207726e [SPARK-57556][SQL] Raise a clear error for the TIME data
type in Hive SerDe interop
e80f4207726e is described below
commit e80f4207726efc9ab465a6ba15c949db7498bd27
Author: Maxim Gekk <[email protected]>
AuthorDate: Tue Jun 30 18:44:55 2026 +0200
[SPARK-57556][SQL] Raise a clear error for the TIME data type in Hive SerDe
interop
### What changes were proposed in this pull request?
Apache Hive has no TIME type, so `TimeType` has no faithful representation
in Hive SerDe interop. This PR (the Option B / "clear, documented error" path
from [SPARK-57556](https://issues.apache.org/jira/browse/SPARK-57556)) makes
`TimeType` produce a clear `AnalysisException` instead of a
`scala.MatchError`/internal error when it reaches the `HiveInspectors` mapping
functions, and rejects it in the Hive SerDe write path:
- `HiveInspectors.toInspector(dataType)`, `toInspector(expr)` (TIME
literal) and `toTypeInfo` now throw `UNSUPPORTED_DATATYPE` via a shared
`unsupportedHiveType` helper. Previously `toInspector(dataType)` had no
`TimeType` case and no default branch, so a TIME column hit a raw
`scala.MatchError`.
- `HiveFileFormat.supportDataType` rejects `TimeType` (recursing into
nested struct/array/map/UDT types, preserving the prior default for all other
types) so Hive SerDe writes raise `UNSUPPORTED_DATA_TYPE_FOR_DATASOURCE`
(format `Hive`) via `FileFormatWriter.verifySchema`.
- Documented the limitation on the TIME entry in
`docs/sql-ref-datatypes.md`.
### Why are the changes needed?
`HiveInspectors` had no `TimeType` case, so object-inspector creation and
TypeInfo mapping fell through to a `MatchError`/internal error when a TIME
column or literal reached Hive SerDe paths (for example, a TIME argument to a
Hive UDF/UDAF/UDTF). This makes the behavior explicit and documented,
consistent with the existing TIME rejection for Hive ORC (SPARK-51590).
### Does this PR introduce _any_ user-facing change?
Yes. Using TIME with Hive UDFs or in a Hive SerDe write now fails with a
clear error that names the unsupported TIME type, instead of a
`MatchError`/internal error. For example, `SELECT myHiveUDF(TIME'12:01:02')`
now reports `[UNSUPPORTED_DATATYPE] Unsupported data type "TIME(6)"` (wrapped
by the Hive UDF resolver), and writing a TIME column through the Hive SerDe
write path reports `[UNSUPPORTED_DATA_TYPE_FOR_DATASOURCE] The Hive datasource
doesn't support the column ... of the type [...]
### How was this patch tested?
Added tests and ran them locally (`build/sbt 'hive/testOnly
*HiveInspectorSuite *HiveUDFSuite *InsertSuite'`):
- `HiveInspectorSuite`: `toInspector(TimeType())`, a TIME literal, and
`TimeType().toTypeInfo` raise `UNSUPPORTED_DATATYPE`.
- `HiveUDFSuite`: passing `TIME'12:01:02'` to a Hive `GenericUDFHash` fails
with a message naming the unsupported TIME type.
- `InsertSuite`: `INSERT OVERWRITE LOCAL DIRECTORY ... STORED AS PARQUET
SELECT TIME'...'` (with `spark.sql.hive.convertMetastoreInsertDir=false`)
raises `UNSUPPORTED_DATA_TYPE_FOR_DATASOURCE`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor (Claude Opus 4.8)
Closes #56850 from MaxGekk/time-hive-serde.
Authored-by: Maxim Gekk <[email protected]>
Signed-off-by: Max Gekk <[email protected]>
---
docs/sql-ref-datatypes.md | 1 +
.../org/apache/spark/sql/hive/HiveInspectors.scala | 13 +++++++++++
.../spark/sql/hive/execution/HiveFileFormat.scala | 19 +++++++++++++++-
.../apache/spark/sql/hive/HiveInspectorSuite.scala | 19 +++++++++++++++-
.../org/apache/spark/sql/hive/InsertSuite.scala | 25 ++++++++++++++++++++++
.../spark/sql/hive/execution/HiveUDFSuite.scala | 17 +++++++++++++++
6 files changed, 92 insertions(+), 2 deletions(-)
diff --git a/docs/sql-ref-datatypes.md b/docs/sql-ref-datatypes.md
index 27663763d6bd..a5da7949d059 100644
--- a/docs/sql-ref-datatypes.md
+++ b/docs/sql-ref-datatypes.md
@@ -48,6 +48,7 @@ Spark SQL and DataFrames support the following data types:
time-zone.
- `TimeType(precision)`: Represents values comprising values of fields hour,
minute and second with the number of decimal digits `precision` following the
decimal point in the seconds field, without a time-zone.
The range of values is from `00:00:00` to `23:59:59` for min precision `0`,
and to `23:59:59.999999999` for max precision `9`. The default precision is `6`.
+ - Note: Apache Hive has no TIME type, so `TimeType` is not supported in
Hive SerDe interop. Storing it in a Hive SerDe table (including `INSERT
OVERWRITE DIRECTORY ... STORED AS`) or passing it to a Hive UDF/UDAF/UDTF
raises an error rather than silently converting the value.
- `TimestampType`: Timestamp with local time zone(TIMESTAMP_LTZ). It
represents values comprising values of fields year, month, day,
hour, minute, and second, with the session local time-zone. The timestamp
value represents an
absolute point in time.
diff --git
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala
index 49b9cc798a1b..285548086a09 100644
--- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala
+++ b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala
@@ -961,6 +961,14 @@ private[hive] trait HiveInspectors {
case _: UserDefinedType[_] =>
val sqlType = dataType.asInstanceOf[UserDefinedType[_]].sqlType
toInspector(sqlType)
+ // Hive has no TIME type, so it cannot be represented by any Hive object
inspector.
+ case _: TimeType => throw unsupportedHiveType(dataType)
+ }
+
+ private def unsupportedHiveType(dataType: DataType): AnalysisException = {
+ new AnalysisException(
+ errorClass = "UNSUPPORTED_DATATYPE",
+ messageParameters = Map("typeName" -> toSQLType(dataType)))
}
/**
@@ -1029,6 +1037,9 @@ private[hive] trait HiveInspectors {
toInspector(dt)
case Literal(_, dt: UserDefinedType[_]) =>
toInspector(dt.sqlType)
+ // Hive has no TIME type, so a TIME constant cannot be mapped to a Hive
object inspector.
+ case Literal(_, dt: TimeType) =>
+ throw unsupportedHiveType(dt)
// We will enumerate all of the possible constant expressions, throw
exception if we missed
case Literal(_, dt) =>
throw SparkException.internalError(s"Hive doesn't support the constant
type [$dt].")
@@ -1281,6 +1292,8 @@ private[hive] trait HiveInspectors {
case NullType => voidTypeInfo
case _: DayTimeIntervalType => intervalDayTimeTypeInfo
case _: YearMonthIntervalType => intervalYearMonthTypeInfo
+ // Hive has no TIME type, so there is no Hive TypeInfo to map it to.
+ case _: TimeType => throw unsupportedHiveType(dt)
case dt =>
throw new AnalysisException(
errorClass = "_LEGACY_ERROR_TEMP_3095", messageParameters = Map("dt"
-> toSQLType(dt)))
diff --git
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/HiveFileFormat.scala
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/HiveFileFormat.scala
index 5506cf8dae07..acbc72fbf7e0 100644
---
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/HiveFileFormat.scala
+++
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/HiveFileFormat.scala
@@ -41,7 +41,7 @@ import
org.apache.spark.sql.execution.datasources.{FileFormat, OutputWriter, Out
import org.apache.spark.sql.hive.{HiveInspectors, HiveTableUtil}
import org.apache.spark.sql.internal.SessionStateHelper
import org.apache.spark.sql.sources.DataSourceRegister
-import org.apache.spark.sql.types.StructType
+import org.apache.spark.sql.types.{ArrayType, DataType, MapType, StructType,
TimeType, UserDefinedType}
import org.apache.spark.util.SerializableJobConf
/**
@@ -115,6 +115,23 @@ case class HiveFileFormat(fileSinkConf: FileSinkDesc)
}
}
+ override def supportDataType(dataType: DataType): Boolean = dataType match {
+ // Hive has no TIME type, so it cannot be stored in a Hive serde table.
Reject it explicitly
+ // (recursing into nested types) while preserving the default behavior for
all other types.
+ case _: TimeType => false
+
+ case st: StructType => st.forall { f => supportDataType(f.dataType) }
+
+ case ArrayType(elementType, _) => supportDataType(elementType)
+
+ case MapType(keyType, valueType, _) =>
+ supportDataType(keyType) && supportDataType(valueType)
+
+ case udt: UserDefinedType[_] => supportDataType(udt.sqlType)
+
+ case _ => true
+ }
+
override def supportFieldName(name: String): Boolean = {
fileSinkConf.getTableInfo.getOutputFileFormatClassName match {
case "org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat" =>
diff --git
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveInspectorSuite.scala
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveInspectorSuite.scala
index 8acabd579d44..b7fb506f07b5 100644
--- a/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveInspectorSuite.scala
+++ b/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveInspectorSuite.scala
@@ -28,7 +28,7 @@ import org.apache.hadoop.hive.serde2.typeinfo.DecimalTypeInfo
import org.apache.hadoop.io.LongWritable
import org.apache.spark.SparkFunSuite
-import org.apache.spark.sql.{Row, TestUserClassUDT}
+import org.apache.spark.sql.{AnalysisException, Row, TestUserClassUDT}
import org.apache.spark.sql.catalyst.InternalRow
import org.apache.spark.sql.catalyst.expressions.Literal
import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData,
GenericArrayData, MapData}
@@ -291,4 +291,21 @@ class HiveInspectorSuite extends SparkFunSuite with
HiveInspectors {
assert(typeInfo2.precision() === 18)
assert(typeInfo2.scale() === 10)
}
+
+ test("SPARK-57556: TIME type is unsupported in Hive object inspectors") {
+ val timeType = TimeType()
+ val expectedParams = Map("typeName" -> s"\"${timeType.sql}\"")
+ checkError(
+ exception = intercept[AnalysisException](toInspector(timeType)),
+ condition = "UNSUPPORTED_DATATYPE",
+ parameters = expectedParams)
+ checkError(
+ exception =
intercept[AnalysisException](toInspector(Literal.create(null, timeType))),
+ condition = "UNSUPPORTED_DATATYPE",
+ parameters = expectedParams)
+ checkError(
+ exception = intercept[AnalysisException](timeType.toTypeInfo),
+ condition = "UNSUPPORTED_DATATYPE",
+ parameters = expectedParams)
+ }
}
diff --git
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/InsertSuite.scala
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/InsertSuite.scala
index b6f20c42f543..7385410e1a71 100644
--- a/sql/hive/src/test/scala/org/apache/spark/sql/hive/InsertSuite.scala
+++ b/sql/hive/src/test/scala/org/apache/spark/sql/hive/InsertSuite.scala
@@ -683,6 +683,31 @@ class InsertSuite extends QueryTest with TestHiveSingleton
with BeforeAndAfter {
}
}
+ test("SPARK-57556: TIME type is unsupported when writing to a Hive serde
directory") {
+ // Disable native data source conversion so that the write goes through
the Hive serde
+ // path (HiveFileFormat) instead of a native data source that may support
TIME.
+ withSQLConf(HiveUtils.CONVERT_METASTORE_INSERT_DIR.key -> "false") {
+ withTempDir { dir =>
+ // InsertIntoHiveDirCommand wraps the failure in a SparkException, so
assert on the cause.
+ val e = intercept[SparkException] {
+ sql(
+ s"""
+ |INSERT OVERWRITE LOCAL DIRECTORY '${dir.toURI.getPath}'
+ |STORED AS PARQUET
+ |SELECT TIME'12:01:02' AS c
+ """.stripMargin)
+ }
+ checkError(
+ exception = e.getCause.asInstanceOf[AnalysisException],
+ condition = "UNSUPPORTED_DATA_TYPE_FOR_DATASOURCE",
+ parameters = Map(
+ "columnName" -> "`c`",
+ "columnType" -> s"\"${TimeType().sql}\"",
+ "format" -> "Hive"))
+ }
+ }
+ }
+
test("insert overwrite to dir from temp table") {
withTempView("test_insert_table") {
spark.range(10).selectExpr("id", "id AS
str").createOrReplaceTempView("test_insert_table")
diff --git
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveUDFSuite.scala
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveUDFSuite.scala
index 35ae3ea29d6d..53c245394daf 100644
---
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveUDFSuite.scala
+++
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveUDFSuite.scala
@@ -41,6 +41,7 @@ import org.apache.spark.sql.execution.WholeStageCodegenExec
import org.apache.spark.sql.functions.{call_function, max}
import org.apache.spark.sql.hive.test.{TestHiveSingleton, TestUDTFJar}
import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.TimeType
import org.apache.spark.tags.SlowHiveTest
import org.apache.spark.util.Utils
@@ -412,6 +413,22 @@ class HiveUDFSuite extends QueryTest with
TestHiveSingleton {
}
}
+ test("SPARK-57556: TIME type is unsupported as a Hive UDF argument") {
+ withUserDefinedFunction("testGenericUDFHash" -> true) {
+ sql(s"CREATE TEMPORARY FUNCTION testGenericUDFHash AS
'${classOf[GenericUDFHash].getName}'")
+ // The Hive UDF resolver wraps the failure in
CANNOT_INSTANTIATE_HIVE_FUNCTION and attaches
+ // the underlying failure as the cause, which clearly identifies the
unsupported TIME type
+ // rather than surfacing a MatchError/internal error.
+ val e = intercept[AnalysisException] {
+ sql("SELECT testGenericUDFHash(TIME'12:01:02')").collect()
+ }
+ checkError(
+ exception = e.getCause.asInstanceOf[AnalysisException],
+ condition = "UNSUPPORTED_DATATYPE",
+ parameters = Map("typeName" -> s"\"${TimeType().sql}\""))
+ }
+ }
+
test("Hive UDFs with insufficient number of input arguments should trigger
an analysis error") {
withTempView("testUDF") {
Seq((1, 2)).toDF("a", "b").createOrReplaceTempView("testUDF")
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
