This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new 53d6711db13c [SPARK-60519][SQL] Raise a clear error for `GEOMETRY` and 
`GEOGRAPHY` in Hive SerDe interop
53d6711db13c is described below

commit 53d6711db13cf281dff22f27a12a96264cf15b5e
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Sat Oct 24 09:54:37 2026 -0700

    [SPARK-60519][SQL] Raise a clear error for `GEOMETRY` and `GEOGRAPHY` in 
Hive SerDe interop
    
    ### What changes were proposed in this pull request?
    
    This PR aims to raise a clear error for the geospatial types, `GEOMETRY` 
and `GEOGRAPHY`, in Hive SerDe interop like `TIME` (SPARK-57556) because Hive 
has no geospatial types.
    
    - `HiveFileFormat.supportDataType` rejects `GeometryType` and 
`GeographyType`, including nested ones, so `INSERT OVERWRITE [LOCAL] DIRECTORY 
... STORED AS` through the Hive SerDe fails with 
`UNSUPPORTED_DATA_TYPE_FOR_DATASOURCE` before running a job.
    - The Hive ORC `OrcFileFormat.supportDataType` (`spark.sql.orc.impl=hive`) 
rejects them like the native ORC data source (SPARK-55262).
    - `HiveInspectors.toInspector` throws `UNSUPPORTED_DATATYPE` for them 
instead of `scala.MatchError` and `INTERNAL_ERROR`, like 
`HiveInspectors.toTypeInfo` already does.
    
    `INSERT OVERWRITE [LOCAL] DIRECTORY ... STORED AS PARQUET` with the default 
configurations, `spark.sql.hive.convertMetastoreInsertDir=true` and 
`spark.sql.hive.convertMetastoreParquet=true`, is not affected because it uses 
the native Parquet data source, which supports the geospatial types.
    
    ### Why are the changes needed?
    
    Hive SerDe cannot parse `geometry(<srid>)` and `geography(<srid>)`. So, the 
following fails inside a task after the job starts.
    
    ```sql
    INSERT OVERWRITE LOCAL DIRECTORY '/tmp/geo' STORED AS TEXTFILE
    SELECT 1 AS i, 
ST_GeomFromWKB(X'0101000000000000000000F03F0000000000000040', 4326) AS c;
    ```
    
    ```
    org.apache.spark.SparkException: Failed inserting overwrite directory 
/tmp/geo
    ...
    Caused by: java.lang.IllegalArgumentException: Error: type expected at the 
position 4 of 'int:geometry(4326)' but 'geometry' is found.
    ```
    
    With `spark.sql.orc.impl=hive`, a data source table with geospatial columns 
is created while the native ORC data source rejects it. However, writing to it 
fails inside a task in the same way. Reading ORC files with a user-specified 
geospatial schema also fails inside a task with `ClassCastException`.
    
    ```sql
    SET spark.sql.orc.impl=hive;
    CREATE TABLE t (i INT, c GEOMETRY(4326)) USING ORC;
    INSERT INTO t VALUES (1, 
ST_GeomFromWKB(X'0101000000000000000000F03F0000000000000040', 4326));
    ```
    
    ```
    org.apache.spark.SparkException: Job aborted due to stage failure: ... 
java.lang.IllegalArgumentException: Error: type expected at the position 4 of 
'int:geometry(4326)' but 'geometry' is found.
    ```
    
    In addition, Hive generic UDFs, UDAFs, and UDTFs fail with 
`scala.MatchError` or `INTERNAL_ERROR` for geospatial arguments.
    
    ```sql
    CREATE TEMPORARY FUNCTION hive_hash AS 
'org.apache.hadoop.hive.ql.udf.generic.GenericUDFHash';
    SELECT 
hive_hash(ST_GeomFromWKB(X'0101000000000000000000F03F0000000000000040', 4326));
    SELECT hive_hash(CAST(NULL AS GEOMETRY(4326)));
    ```
    
    ```
    [CANNOT_INSTANTIATE_HIVE_FUNCTION] ...
    Caused by: scala.MatchError: GeometryType(4326) (of class 
org.apache.spark.sql.types.GeometryType)
    
    [CANNOT_INSTANTIATE_HIVE_FUNCTION] ...
    Caused by: org.apache.spark.SparkException: [INTERNAL_ERROR] Hive doesn't 
support the constant type [GeometryType(4326)]. SQLSTATE: XX000
    ```
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. The above cases fail with the following errors without running a job.
    
    - Hive SerDe writes, wrapped by `Failed inserting overwrite directory` as 
before:
      ```
      [UNSUPPORTED_DATA_TYPE_FOR_DATASOURCE] The Hive datasource doesn't 
support the column `c` of the type "GEOMETRY(4326)". SQLSTATE: 0A000
      ```
    - `CREATE TABLE`, writes, and reads with `spark.sql.orc.impl=hive`, like 
the native ORC data source:
      ```
      [UNSUPPORTED_DATA_TYPE_FOR_DATASOURCE] The ORC datasource doesn't support 
the column `c` of the type "GEOMETRY(4326)". SQLSTATE: 0A000
      ```
    - Hive generic UDFs, UDAFs, and UDTFs, wrapped by 
`CANNOT_INSTANTIATE_HIVE_FUNCTION` as before, like `TIME` and the simple Hive 
UDFs:
      ```
      [UNSUPPORTED_DATATYPE] Unsupported data type "GEOMETRY(4326)". SQLSTATE: 
0A000
      ```
    
    ### How was this patch tested?
    
    Pass the CIs with the newly added test cases in `HiveInspectorSuite`, 
`InsertSuite`, `HiveUDFSuite`, and `HiveOrcQuerySuite`. All of them fail 
without this fix.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    Closes #59755 from dongjoon-hyun/SPARK-60519.
    
    Authored-by: Dongjoon Hyun <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
    (cherry picked from commit 935b76e3fd49ab8682b3c89c734928560f639ec0)
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 .../org/apache/spark/sql/hive/HiveInspectors.scala |  6 +++
 .../spark/sql/hive/execution/HiveFileFormat.scala  |  5 ++-
 .../apache/spark/sql/hive/orc/OrcFileFormat.scala  |  1 +
 .../apache/spark/sql/hive/HiveInspectorSuite.scala | 20 ++++++++++
 .../org/apache/spark/sql/hive/InsertSuite.scala    | 36 ++++++++++++++++++
 .../spark/sql/hive/execution/HiveUDFSuite.scala    | 25 ++++++++++++
 .../spark/sql/hive/orc/HiveOrcQuerySuite.scala     | 44 +++++++++++++++++++++-
 7 files changed, 135 insertions(+), 2 deletions(-)

diff --git 
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala 
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala
index b8758407877f..2a3f16dfeffb 100644
--- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala
+++ b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala
@@ -1051,6 +1051,8 @@ private[hive] trait HiveInspectors {
     // Hive has no nanosecond-precision timestamp type, so it cannot be 
represented by any Hive
     // object inspector. Reject it instead of silently downgrading to 
microsecond precision.
     case _: AnyTimestampNanoType => throw unsupportedHiveType(dataType)
+    // Hive has no geospatial types, so they cannot be represented by any Hive 
object inspector.
+    case _: GeometryType | _: GeographyType => throw 
unsupportedHiveType(dataType)
   }
 
   private def unsupportedHiveType(dataType: DataType): AnalysisException = {
@@ -1150,6 +1152,10 @@ private[hive] trait HiveInspectors {
     // Hive object inspector.
     case Literal(_, dt: AnyTimestampNanoType) =>
       throw unsupportedHiveType(dt)
+    // Hive has no geospatial types, so a geospatial constant cannot be mapped 
to a Hive object
+    // inspector.
+    case Literal(_, dt @ (_: GeometryType | _: GeographyType)) =>
+      throw unsupportedHiveType(dt)
     // We will enumerate all of the possible constant expressions, throw 
exception if we missed
     case Literal(_, dt) =>
       throw SparkException.internalError(s"Hive doesn't support the constant 
type [$dt].")
diff --git 
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/HiveFileFormat.scala
 
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/HiveFileFormat.scala
index 4545acc7f203..189d90235bc1 100644
--- 
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/HiveFileFormat.scala
+++ 
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/HiveFileFormat.scala
@@ -41,7 +41,7 @@ import 
org.apache.spark.sql.execution.datasources.{FileFormat, OutputWriter, Out
 import org.apache.spark.sql.hive.{HiveInspectors, HiveTableUtil}
 import org.apache.spark.sql.internal.SessionStateHelper
 import org.apache.spark.sql.sources.DataSourceRegister
-import org.apache.spark.sql.types.{AnyTimestampNanoType, ArrayType, DataType, 
MapType, StructType, TimeType, UserDefinedType}
+import org.apache.spark.sql.types.{AnyTimestampNanoType, ArrayType, DataType, 
GeographyType, GeometryType, MapType, StructType, TimeType, UserDefinedType}
 import org.apache.spark.util.SerializableJobConf
 
 /**
@@ -124,6 +124,9 @@ case class HiveFileFormat(fileSinkConf: FileSinkDesc)
     // downgrading to microsecond precision (which the Hive serde would 
otherwise do).
     case _: AnyTimestampNanoType => false
 
+    // Hive has no geospatial types, so they cannot be stored in a Hive serde 
table.
+    case _: GeometryType | _: GeographyType => false
+
     case st: StructType => st.forall { f => supportDataType(f.dataType) }
 
     case ArrayType(elementType, _) => supportDataType(elementType)
diff --git 
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFileFormat.scala 
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFileFormat.scala
index 10469833c0cc..255f2fdc5b2e 100644
--- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFileFormat.scala
+++ b/sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFileFormat.scala
@@ -201,6 +201,7 @@ case class OrcFileFormat() extends FileFormat
 
     case _: AnsiIntervalType => false
     case _: TimeType => false
+    case _: GeometryType | _: GeographyType => false
     case _: AtomicType => true
 
     case st: StructType => st.forall { f => supportDataType(f.dataType) }
diff --git 
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveInspectorSuite.scala 
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveInspectorSuite.scala
index 9f135bf6bc73..6cdf6b15e2be 100644
--- a/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveInspectorSuite.scala
+++ b/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveInspectorSuite.scala
@@ -613,4 +613,24 @@ class HiveInspectorSuite extends SparkFunSuite with 
HiveInspectors {
         parameters = expectedParams)
     }
   }
+
+  test("SPARK-60519: geospatial types are unsupported in Hive object 
inspectors") {
+    Seq(
+      GeometryType(4326), GeometryType("ANY"),
+      GeographyType(4326), GeographyType("ANY")).foreach { geoType =>
+      val expectedParams = Map("typeName" -> s"\"${geoType.sql}\"")
+      checkError(
+        exception = intercept[AnalysisException](toInspector(geoType)),
+        condition = "UNSUPPORTED_DATATYPE",
+        parameters = expectedParams)
+      checkError(
+        exception = 
intercept[AnalysisException](toInspector(Literal.create(null, geoType))),
+        condition = "UNSUPPORTED_DATATYPE",
+        parameters = expectedParams)
+      checkError(
+        exception = intercept[AnalysisException](geoType.toTypeInfo),
+        condition = "UNSUPPORTED_DATATYPE",
+        parameters = expectedParams)
+    }
+  }
 }
diff --git 
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/InsertSuite.scala 
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/InsertSuite.scala
index ba768db68867..1367041b18fd 100644
--- a/sql/hive/src/test/scala/org/apache/spark/sql/hive/InsertSuite.scala
+++ b/sql/hive/src/test/scala/org/apache/spark/sql/hive/InsertSuite.scala
@@ -764,6 +764,42 @@ class InsertSuite extends QueryTest with TestHiveSingleton 
with BeforeAndAfter {
     }
   }
 
+  test("SPARK-60519: geospatial types are unsupported when writing to a Hive 
serde directory") {
+    // WKB of POINT(1 2).
+    val wkb = "0101000000000000000000F03F0000000000000040"
+    // Disable native data source conversion so that PARQUET and ORC also go 
through the Hive
+    // serde path (HiveFileFormat) like TEXTFILE.
+    withSQLConf(HiveUtils.CONVERT_METASTORE_INSERT_DIR.key -> "false") {
+      Seq("TEXTFILE", "PARQUET", "ORC").foreach { fileFormat =>
+        Seq(
+          s"ST_GeomFromWKB(X'$wkb', 4326)" -> GeometryType(4326),
+          s"ST_GeogFromWKB(X'$wkb')" -> GeographyType(4326),
+          s"ARRAY(ST_GeomFromWKB(X'$wkb', 4326))" -> 
ArrayType(GeometryType(4326))
+        ).foreach { case (expr, dt) =>
+          withTempDir { dir =>
+            // InsertIntoHiveDirCommand wraps the failure in a SparkException, 
so assert on the
+            // cause.
+            val e = intercept[SparkException] {
+              sql(
+                s"""
+                   |INSERT OVERWRITE LOCAL DIRECTORY '${dir.toURI.getPath}'
+                   |STORED AS $fileFormat
+                   |SELECT $expr AS c
+                 """.stripMargin)
+            }
+            checkError(
+              exception = e.getCause.asInstanceOf[AnalysisException],
+              condition = "UNSUPPORTED_DATA_TYPE_FOR_DATASOURCE",
+              parameters = Map(
+                "columnName" -> "`c`",
+                "columnType" -> s"\"${dt.sql}\"",
+                "format" -> "Hive"))
+          }
+        }
+      }
+    }
+  }
+
   test("insert overwrite to dir from temp table") {
     withTempView("test_insert_table") {
       spark.range(10).selectExpr("id", "id AS 
str").createOrReplaceTempView("test_insert_table")
diff --git 
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveUDFSuite.scala
 
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveUDFSuite.scala
index efba2248819a..89edb53048a3 100644
--- 
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveUDFSuite.scala
+++ 
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveUDFSuite.scala
@@ -455,6 +455,31 @@ class HiveUDFSuite extends QueryTest with 
TestHiveSingleton {
     }
   }
 
+  test("SPARK-60519: geospatial types are unsupported as Hive UDF arguments") {
+    withUserDefinedFunction("testGenericUDFHash" -> true) {
+      sql(s"CREATE TEMPORARY FUNCTION testGenericUDFHash AS 
'${classOf[GenericUDFHash].getName}'")
+      // WKB of POINT(1 2).
+      val wkb = "0101000000000000000000F03F0000000000000040"
+      Seq(
+        // Non-foldable arguments, which are mapped to Hive object inspectors 
by their data types.
+        s"ST_GeomFromWKB(X'$wkb', 4326)" -> GeometryType(4326),
+        s"ST_GeogFromWKB(X'$wkb')" -> GeographyType(4326),
+        // Foldable arguments, which are mapped to constant Hive object 
inspectors.
+        "CAST(NULL AS GEOMETRY(4326))" -> GeometryType(4326),
+        "CAST(NULL AS GEOGRAPHY(4326))" -> GeographyType(4326)
+      ).foreach { case (arg, dt) =>
+        // The Hive UDF resolver wraps the failure in 
CANNOT_INSTANTIATE_HIVE_FUNCTION.
+        val e = intercept[AnalysisException] {
+          sql(s"SELECT testGenericUDFHash($arg)").collect()
+        }
+        checkError(
+          exception = e.getCause.asInstanceOf[AnalysisException],
+          condition = "UNSUPPORTED_DATATYPE",
+          parameters = Map("typeName" -> s"\"${dt.sql}\""))
+      }
+    }
+  }
+
   test("Hive UDFs with insufficient number of input arguments should trigger 
an analysis error") {
     withTempView("testUDF") {
       Seq((1, 2)).toDF("a", "b").createOrReplaceTempView("testUDF")
diff --git 
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/HiveOrcQuerySuite.scala 
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/HiveOrcQuerySuite.scala
index 45f5d2f273e3..8427c25c77f2 100644
--- 
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/HiveOrcQuerySuite.scala
+++ 
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/HiveOrcQuerySuite.scala
@@ -30,7 +30,7 @@ import 
org.apache.spark.sql.execution.datasources.orc.OrcQueryTest
 import org.apache.spark.sql.hive.{HiveSessionCatalog, HiveUtils}
 import org.apache.spark.sql.hive.test.TestHiveSingleton
 import org.apache.spark.sql.internal.SQLConf
-import org.apache.spark.sql.types.TimeType
+import org.apache.spark.sql.types.{GeographyType, GeometryType, TimeType}
 import org.apache.spark.util.Utils
 
 class HiveOrcQuerySuite extends OrcQueryTest with TestHiveSingleton {
@@ -442,4 +442,46 @@ class HiveOrcQuerySuite extends OrcQueryTest with 
TestHiveSingleton {
       }
     }
   }
+
+  test("SPARK-60519: geospatial types are unsupported in Hive ORC") {
+    withSQLConf(SQLConf.ORC_IMPLEMENTATION.key -> "hive") {
+      withTempDir { dir =>
+        val tempDir = new File(dir, "files").getCanonicalPath
+        // WKB of POINT(1 2).
+        val wkb = "0101000000000000000000F03F0000000000000040"
+        Seq(
+          s"ST_GeomFromWKB(X'$wkb', 4326)" -> GeometryType(4326),
+          s"ST_GeogFromWKB(X'$wkb')" -> GeographyType(4326)).foreach { case 
(expr, dt) =>
+          val expectedParams = Map(
+            "columnName" -> "`g`",
+            "columnType" -> s"\"${dt.sql}\"",
+            "format" -> "ORC")
+          // write path
+          checkError(
+            exception = intercept[AnalysisException] {
+              sql(s"select $expr as 
g").write.format("orc").mode("overwrite").save(tempDir)
+            },
+            condition = "UNSUPPORTED_DATA_TYPE_FOR_DATASOURCE",
+            parameters = expectedParams)
+          // read path
+          checkError(
+            exception = intercept[AnalysisException] {
+              sql(s"select X'$wkb' as 
g").write.format("orc").mode("overwrite").save(tempDir)
+              spark.read.schema(s"g 
${dt.sql}").format("orc").load(tempDir).collect()
+            },
+            condition = "UNSUPPORTED_DATA_TYPE_FOR_DATASOURCE",
+            parameters = expectedParams)
+          // CREATE TABLE
+          withTable("t") {
+            checkError(
+              exception = intercept[AnalysisException] {
+                sql(s"CREATE TABLE t (g ${dt.sql}) USING ORC")
+              },
+              condition = "UNSUPPORTED_DATA_TYPE_FOR_DATASOURCE",
+              parameters = expectedParams)
+          }
+        }
+      }
+    }
+  }
 }


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to