This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 53d6711db13c [SPARK-60519][SQL] Raise a clear error for `GEOMETRY` and
`GEOGRAPHY` in Hive SerDe interop
53d6711db13c is described below
commit 53d6711db13cf281dff22f27a12a96264cf15b5e
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Sat Oct 24 09:54:37 2026 -0700
[SPARK-60519][SQL] Raise a clear error for `GEOMETRY` and `GEOGRAPHY` in
Hive SerDe interop
### What changes were proposed in this pull request?
This PR aims to raise a clear error for the geospatial types, `GEOMETRY`
and `GEOGRAPHY`, in Hive SerDe interop like `TIME` (SPARK-57556) because Hive
has no geospatial types.
- `HiveFileFormat.supportDataType` rejects `GeometryType` and
`GeographyType`, including nested ones, so `INSERT OVERWRITE [LOCAL] DIRECTORY
... STORED AS` through the Hive SerDe fails with
`UNSUPPORTED_DATA_TYPE_FOR_DATASOURCE` before running a job.
- The Hive ORC `OrcFileFormat.supportDataType` (`spark.sql.orc.impl=hive`)
rejects them like the native ORC data source (SPARK-55262).
- `HiveInspectors.toInspector` throws `UNSUPPORTED_DATATYPE` for them
instead of `scala.MatchError` and `INTERNAL_ERROR`, like
`HiveInspectors.toTypeInfo` already does.
`INSERT OVERWRITE [LOCAL] DIRECTORY ... STORED AS PARQUET` with the default
configurations, `spark.sql.hive.convertMetastoreInsertDir=true` and
`spark.sql.hive.convertMetastoreParquet=true`, is not affected because it uses
the native Parquet data source, which supports the geospatial types.
### Why are the changes needed?
Hive SerDe cannot parse `geometry(<srid>)` and `geography(<srid>)`. So, the
following fails inside a task after the job starts.
```sql
INSERT OVERWRITE LOCAL DIRECTORY '/tmp/geo' STORED AS TEXTFILE
SELECT 1 AS i,
ST_GeomFromWKB(X'0101000000000000000000F03F0000000000000040', 4326) AS c;
```
```
org.apache.spark.SparkException: Failed inserting overwrite directory
/tmp/geo
...
Caused by: java.lang.IllegalArgumentException: Error: type expected at the
position 4 of 'int:geometry(4326)' but 'geometry' is found.
```
With `spark.sql.orc.impl=hive`, a data source table with geospatial columns
is created while the native ORC data source rejects it. However, writing to it
fails inside a task in the same way. Reading ORC files with a user-specified
geospatial schema also fails inside a task with `ClassCastException`.
```sql
SET spark.sql.orc.impl=hive;
CREATE TABLE t (i INT, c GEOMETRY(4326)) USING ORC;
INSERT INTO t VALUES (1,
ST_GeomFromWKB(X'0101000000000000000000F03F0000000000000040', 4326));
```
```
org.apache.spark.SparkException: Job aborted due to stage failure: ...
java.lang.IllegalArgumentException: Error: type expected at the position 4 of
'int:geometry(4326)' but 'geometry' is found.
```
In addition, Hive generic UDFs, UDAFs, and UDTFs fail with
`scala.MatchError` or `INTERNAL_ERROR` for geospatial arguments.
```sql
CREATE TEMPORARY FUNCTION hive_hash AS
'org.apache.hadoop.hive.ql.udf.generic.GenericUDFHash';
SELECT
hive_hash(ST_GeomFromWKB(X'0101000000000000000000F03F0000000000000040', 4326));
SELECT hive_hash(CAST(NULL AS GEOMETRY(4326)));
```
```
[CANNOT_INSTANTIATE_HIVE_FUNCTION] ...
Caused by: scala.MatchError: GeometryType(4326) (of class
org.apache.spark.sql.types.GeometryType)
[CANNOT_INSTANTIATE_HIVE_FUNCTION] ...
Caused by: org.apache.spark.SparkException: [INTERNAL_ERROR] Hive doesn't
support the constant type [GeometryType(4326)]. SQLSTATE: XX000
```
### Does this PR introduce _any_ user-facing change?
Yes. The above cases fail with the following errors without running a job.
- Hive SerDe writes, wrapped by `Failed inserting overwrite directory` as
before:
```
[UNSUPPORTED_DATA_TYPE_FOR_DATASOURCE] The Hive datasource doesn't
support the column `c` of the type "GEOMETRY(4326)". SQLSTATE: 0A000
```
- `CREATE TABLE`, writes, and reads with `spark.sql.orc.impl=hive`, like
the native ORC data source:
```
[UNSUPPORTED_DATA_TYPE_FOR_DATASOURCE] The ORC datasource doesn't support
the column `c` of the type "GEOMETRY(4326)". SQLSTATE: 0A000
```
- Hive generic UDFs, UDAFs, and UDTFs, wrapped by
`CANNOT_INSTANTIATE_HIVE_FUNCTION` as before, like `TIME` and the simple Hive
UDFs:
```
[UNSUPPORTED_DATATYPE] Unsupported data type "GEOMETRY(4326)". SQLSTATE:
0A000
```
### How was this patch tested?
Pass the CIs with the newly added test cases in `HiveInspectorSuite`,
`InsertSuite`, `HiveUDFSuite`, and `HiveOrcQuerySuite`. All of them fail
without this fix.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #59755 from dongjoon-hyun/SPARK-60519.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit 935b76e3fd49ab8682b3c89c734928560f639ec0)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../org/apache/spark/sql/hive/HiveInspectors.scala | 6 +++
.../spark/sql/hive/execution/HiveFileFormat.scala | 5 ++-
.../apache/spark/sql/hive/orc/OrcFileFormat.scala | 1 +
.../apache/spark/sql/hive/HiveInspectorSuite.scala | 20 ++++++++++
.../org/apache/spark/sql/hive/InsertSuite.scala | 36 ++++++++++++++++++
.../spark/sql/hive/execution/HiveUDFSuite.scala | 25 ++++++++++++
.../spark/sql/hive/orc/HiveOrcQuerySuite.scala | 44 +++++++++++++++++++++-
7 files changed, 135 insertions(+), 2 deletions(-)
diff --git
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala
index b8758407877f..2a3f16dfeffb 100644
--- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala
+++ b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala
@@ -1051,6 +1051,8 @@ private[hive] trait HiveInspectors {
// Hive has no nanosecond-precision timestamp type, so it cannot be
represented by any Hive
// object inspector. Reject it instead of silently downgrading to
microsecond precision.
case _: AnyTimestampNanoType => throw unsupportedHiveType(dataType)
+ // Hive has no geospatial types, so they cannot be represented by any Hive
object inspector.
+ case _: GeometryType | _: GeographyType => throw
unsupportedHiveType(dataType)
}
private def unsupportedHiveType(dataType: DataType): AnalysisException = {
@@ -1150,6 +1152,10 @@ private[hive] trait HiveInspectors {
// Hive object inspector.
case Literal(_, dt: AnyTimestampNanoType) =>
throw unsupportedHiveType(dt)
+ // Hive has no geospatial types, so a geospatial constant cannot be mapped
to a Hive object
+ // inspector.
+ case Literal(_, dt @ (_: GeometryType | _: GeographyType)) =>
+ throw unsupportedHiveType(dt)
// We will enumerate all of the possible constant expressions, throw
exception if we missed
case Literal(_, dt) =>
throw SparkException.internalError(s"Hive doesn't support the constant
type [$dt].")
diff --git
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/HiveFileFormat.scala
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/HiveFileFormat.scala
index 4545acc7f203..189d90235bc1 100644
---
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/HiveFileFormat.scala
+++
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/HiveFileFormat.scala
@@ -41,7 +41,7 @@ import
org.apache.spark.sql.execution.datasources.{FileFormat, OutputWriter, Out
import org.apache.spark.sql.hive.{HiveInspectors, HiveTableUtil}
import org.apache.spark.sql.internal.SessionStateHelper
import org.apache.spark.sql.sources.DataSourceRegister
-import org.apache.spark.sql.types.{AnyTimestampNanoType, ArrayType, DataType,
MapType, StructType, TimeType, UserDefinedType}
+import org.apache.spark.sql.types.{AnyTimestampNanoType, ArrayType, DataType,
GeographyType, GeometryType, MapType, StructType, TimeType, UserDefinedType}
import org.apache.spark.util.SerializableJobConf
/**
@@ -124,6 +124,9 @@ case class HiveFileFormat(fileSinkConf: FileSinkDesc)
// downgrading to microsecond precision (which the Hive serde would
otherwise do).
case _: AnyTimestampNanoType => false
+ // Hive has no geospatial types, so they cannot be stored in a Hive serde
table.
+ case _: GeometryType | _: GeographyType => false
+
case st: StructType => st.forall { f => supportDataType(f.dataType) }
case ArrayType(elementType, _) => supportDataType(elementType)
diff --git
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFileFormat.scala
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFileFormat.scala
index 10469833c0cc..255f2fdc5b2e 100644
--- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFileFormat.scala
+++ b/sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFileFormat.scala
@@ -201,6 +201,7 @@ case class OrcFileFormat() extends FileFormat
case _: AnsiIntervalType => false
case _: TimeType => false
+ case _: GeometryType | _: GeographyType => false
case _: AtomicType => true
case st: StructType => st.forall { f => supportDataType(f.dataType) }
diff --git
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveInspectorSuite.scala
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveInspectorSuite.scala
index 9f135bf6bc73..6cdf6b15e2be 100644
--- a/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveInspectorSuite.scala
+++ b/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveInspectorSuite.scala
@@ -613,4 +613,24 @@ class HiveInspectorSuite extends SparkFunSuite with
HiveInspectors {
parameters = expectedParams)
}
}
+
+ test("SPARK-60519: geospatial types are unsupported in Hive object
inspectors") {
+ Seq(
+ GeometryType(4326), GeometryType("ANY"),
+ GeographyType(4326), GeographyType("ANY")).foreach { geoType =>
+ val expectedParams = Map("typeName" -> s"\"${geoType.sql}\"")
+ checkError(
+ exception = intercept[AnalysisException](toInspector(geoType)),
+ condition = "UNSUPPORTED_DATATYPE",
+ parameters = expectedParams)
+ checkError(
+ exception =
intercept[AnalysisException](toInspector(Literal.create(null, geoType))),
+ condition = "UNSUPPORTED_DATATYPE",
+ parameters = expectedParams)
+ checkError(
+ exception = intercept[AnalysisException](geoType.toTypeInfo),
+ condition = "UNSUPPORTED_DATATYPE",
+ parameters = expectedParams)
+ }
+ }
}
diff --git
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/InsertSuite.scala
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/InsertSuite.scala
index ba768db68867..1367041b18fd 100644
--- a/sql/hive/src/test/scala/org/apache/spark/sql/hive/InsertSuite.scala
+++ b/sql/hive/src/test/scala/org/apache/spark/sql/hive/InsertSuite.scala
@@ -764,6 +764,42 @@ class InsertSuite extends QueryTest with TestHiveSingleton
with BeforeAndAfter {
}
}
+ test("SPARK-60519: geospatial types are unsupported when writing to a Hive
serde directory") {
+ // WKB of POINT(1 2).
+ val wkb = "0101000000000000000000F03F0000000000000040"
+ // Disable native data source conversion so that PARQUET and ORC also go
through the Hive
+ // serde path (HiveFileFormat) like TEXTFILE.
+ withSQLConf(HiveUtils.CONVERT_METASTORE_INSERT_DIR.key -> "false") {
+ Seq("TEXTFILE", "PARQUET", "ORC").foreach { fileFormat =>
+ Seq(
+ s"ST_GeomFromWKB(X'$wkb', 4326)" -> GeometryType(4326),
+ s"ST_GeogFromWKB(X'$wkb')" -> GeographyType(4326),
+ s"ARRAY(ST_GeomFromWKB(X'$wkb', 4326))" ->
ArrayType(GeometryType(4326))
+ ).foreach { case (expr, dt) =>
+ withTempDir { dir =>
+ // InsertIntoHiveDirCommand wraps the failure in a SparkException,
so assert on the
+ // cause.
+ val e = intercept[SparkException] {
+ sql(
+ s"""
+ |INSERT OVERWRITE LOCAL DIRECTORY '${dir.toURI.getPath}'
+ |STORED AS $fileFormat
+ |SELECT $expr AS c
+ """.stripMargin)
+ }
+ checkError(
+ exception = e.getCause.asInstanceOf[AnalysisException],
+ condition = "UNSUPPORTED_DATA_TYPE_FOR_DATASOURCE",
+ parameters = Map(
+ "columnName" -> "`c`",
+ "columnType" -> s"\"${dt.sql}\"",
+ "format" -> "Hive"))
+ }
+ }
+ }
+ }
+ }
+
test("insert overwrite to dir from temp table") {
withTempView("test_insert_table") {
spark.range(10).selectExpr("id", "id AS
str").createOrReplaceTempView("test_insert_table")
diff --git
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveUDFSuite.scala
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveUDFSuite.scala
index efba2248819a..89edb53048a3 100644
---
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveUDFSuite.scala
+++
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveUDFSuite.scala
@@ -455,6 +455,31 @@ class HiveUDFSuite extends QueryTest with
TestHiveSingleton {
}
}
+ test("SPARK-60519: geospatial types are unsupported as Hive UDF arguments") {
+ withUserDefinedFunction("testGenericUDFHash" -> true) {
+ sql(s"CREATE TEMPORARY FUNCTION testGenericUDFHash AS
'${classOf[GenericUDFHash].getName}'")
+ // WKB of POINT(1 2).
+ val wkb = "0101000000000000000000F03F0000000000000040"
+ Seq(
+ // Non-foldable arguments, which are mapped to Hive object inspectors
by their data types.
+ s"ST_GeomFromWKB(X'$wkb', 4326)" -> GeometryType(4326),
+ s"ST_GeogFromWKB(X'$wkb')" -> GeographyType(4326),
+ // Foldable arguments, which are mapped to constant Hive object
inspectors.
+ "CAST(NULL AS GEOMETRY(4326))" -> GeometryType(4326),
+ "CAST(NULL AS GEOGRAPHY(4326))" -> GeographyType(4326)
+ ).foreach { case (arg, dt) =>
+ // The Hive UDF resolver wraps the failure in
CANNOT_INSTANTIATE_HIVE_FUNCTION.
+ val e = intercept[AnalysisException] {
+ sql(s"SELECT testGenericUDFHash($arg)").collect()
+ }
+ checkError(
+ exception = e.getCause.asInstanceOf[AnalysisException],
+ condition = "UNSUPPORTED_DATATYPE",
+ parameters = Map("typeName" -> s"\"${dt.sql}\""))
+ }
+ }
+ }
+
test("Hive UDFs with insufficient number of input arguments should trigger
an analysis error") {
withTempView("testUDF") {
Seq((1, 2)).toDF("a", "b").createOrReplaceTempView("testUDF")
diff --git
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/HiveOrcQuerySuite.scala
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/HiveOrcQuerySuite.scala
index 45f5d2f273e3..8427c25c77f2 100644
---
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/HiveOrcQuerySuite.scala
+++
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/HiveOrcQuerySuite.scala
@@ -30,7 +30,7 @@ import
org.apache.spark.sql.execution.datasources.orc.OrcQueryTest
import org.apache.spark.sql.hive.{HiveSessionCatalog, HiveUtils}
import org.apache.spark.sql.hive.test.TestHiveSingleton
import org.apache.spark.sql.internal.SQLConf
-import org.apache.spark.sql.types.TimeType
+import org.apache.spark.sql.types.{GeographyType, GeometryType, TimeType}
import org.apache.spark.util.Utils
class HiveOrcQuerySuite extends OrcQueryTest with TestHiveSingleton {
@@ -442,4 +442,46 @@ class HiveOrcQuerySuite extends OrcQueryTest with
TestHiveSingleton {
}
}
}
+
+ test("SPARK-60519: geospatial types are unsupported in Hive ORC") {
+ withSQLConf(SQLConf.ORC_IMPLEMENTATION.key -> "hive") {
+ withTempDir { dir =>
+ val tempDir = new File(dir, "files").getCanonicalPath
+ // WKB of POINT(1 2).
+ val wkb = "0101000000000000000000F03F0000000000000040"
+ Seq(
+ s"ST_GeomFromWKB(X'$wkb', 4326)" -> GeometryType(4326),
+ s"ST_GeogFromWKB(X'$wkb')" -> GeographyType(4326)).foreach { case
(expr, dt) =>
+ val expectedParams = Map(
+ "columnName" -> "`g`",
+ "columnType" -> s"\"${dt.sql}\"",
+ "format" -> "ORC")
+ // write path
+ checkError(
+ exception = intercept[AnalysisException] {
+ sql(s"select $expr as
g").write.format("orc").mode("overwrite").save(tempDir)
+ },
+ condition = "UNSUPPORTED_DATA_TYPE_FOR_DATASOURCE",
+ parameters = expectedParams)
+ // read path
+ checkError(
+ exception = intercept[AnalysisException] {
+ sql(s"select X'$wkb' as
g").write.format("orc").mode("overwrite").save(tempDir)
+ spark.read.schema(s"g
${dt.sql}").format("orc").load(tempDir).collect()
+ },
+ condition = "UNSUPPORTED_DATA_TYPE_FOR_DATASOURCE",
+ parameters = expectedParams)
+ // CREATE TABLE
+ withTable("t") {
+ checkError(
+ exception = intercept[AnalysisException] {
+ sql(s"CREATE TABLE t (g ${dt.sql}) USING ORC")
+ },
+ condition = "UNSUPPORTED_DATA_TYPE_FOR_DATASOURCE",
+ parameters = expectedParams)
+ }
+ }
+ }
+ }
+ }
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]