This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.2 by this push:
     new 17b408fabd3f [SPARK-60519][SQL] Raise a clear error for `GEOMETRY` and 
`GEOGRAPHY` in Hive SerDe interop
17b408fabd3f is described below

commit 17b408fabd3f78d4d13994792a83dd842457b5c4
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Sat Oct 24 09:54:37 2026 -0700

    [SPARK-60519][SQL] Raise a clear error for `GEOMETRY` and `GEOGRAPHY` in 
Hive SerDe interop
    
    This PR aims to raise a clear error for the geospatial types, `GEOMETRY` 
and `GEOGRAPHY`, in Hive SerDe interop like `TIME` (SPARK-57556) because Hive 
has no geospatial types.
    
    - `HiveFileFormat.supportDataType` rejects `GeometryType` and 
`GeographyType`, including nested ones, so `INSERT OVERWRITE [LOCAL] DIRECTORY 
... STORED AS` through the Hive SerDe fails with 
`UNSUPPORTED_DATA_TYPE_FOR_DATASOURCE` before running a job.
    - The Hive ORC `OrcFileFormat.supportDataType` (`spark.sql.orc.impl=hive`) 
rejects them like the native ORC data source (SPARK-55262).
    - `HiveInspectors.toInspector` throws `UNSUPPORTED_DATATYPE` for them 
instead of `scala.MatchError` and `INTERNAL_ERROR`, like 
`HiveInspectors.toTypeInfo` already does.
    
    `INSERT OVERWRITE [LOCAL] DIRECTORY ... STORED AS PARQUET` with the default 
configurations, `spark.sql.hive.convertMetastoreInsertDir=true` and 
`spark.sql.hive.convertMetastoreParquet=true`, is not affected because it uses 
the native Parquet data source, which supports the geospatial types.
    
    Hive SerDe cannot parse `geometry(<srid>)` and `geography(<srid>)`. So, the 
following fails inside a task after the job starts.
    
    ```sql
    INSERT OVERWRITE LOCAL DIRECTORY '/tmp/geo' STORED AS TEXTFILE
    SELECT 1 AS i, 
ST_GeomFromWKB(X'0101000000000000000000F03F0000000000000040', 4326) AS c;
    ```
    
    ```
    org.apache.spark.SparkException: Failed inserting overwrite directory 
/tmp/geo
    ...
    Caused by: java.lang.IllegalArgumentException: Error: type expected at the 
position 4 of 'int:geometry(4326)' but 'geometry' is found.
    ```
    
    With `spark.sql.orc.impl=hive`, a data source table with geospatial columns 
is created while the native ORC data source rejects it. However, writing to it 
fails inside a task in the same way. Reading ORC files with a user-specified 
geospatial schema also fails inside a task with `ClassCastException`.
    
    ```sql
    SET spark.sql.orc.impl=hive;
    CREATE TABLE t (i INT, c GEOMETRY(4326)) USING ORC;
    INSERT INTO t VALUES (1, 
ST_GeomFromWKB(X'0101000000000000000000F03F0000000000000040', 4326));
    ```
    
    ```
    org.apache.spark.SparkException: Job aborted due to stage failure: ... 
java.lang.IllegalArgumentException: Error: type expected at the position 4 of 
'int:geometry(4326)' but 'geometry' is found.
    ```
    
    In addition, Hive generic UDFs, UDAFs, and UDTFs fail with 
`scala.MatchError` or `INTERNAL_ERROR` for geospatial arguments.
    
    ```sql
    CREATE TEMPORARY FUNCTION hive_hash AS 
'org.apache.hadoop.hive.ql.udf.generic.GenericUDFHash';
    SELECT 
hive_hash(ST_GeomFromWKB(X'0101000000000000000000F03F0000000000000040', 4326));
    SELECT hive_hash(CAST(NULL AS GEOMETRY(4326)));
    ```
    
    ```
    [CANNOT_INSTANTIATE_HIVE_FUNCTION] ...
    Caused by: scala.MatchError: GeometryType(4326) (of class 
org.apache.spark.sql.types.GeometryType)
    
    [CANNOT_INSTANTIATE_HIVE_FUNCTION] ...
    Caused by: org.apache.spark.SparkException: [INTERNAL_ERROR] Hive doesn't 
support the constant type [GeometryType(4326)]. SQLSTATE: XX000
    ```
    
    Yes. The above cases fail with the following errors without running a job.
    
    - Hive SerDe writes, wrapped by `Failed inserting overwrite directory` as 
before:
      ```
      [UNSUPPORTED_DATA_TYPE_FOR_DATASOURCE] The Hive datasource doesn't 
support the column `c` of the type "GEOMETRY(4326)". SQLSTATE: 0A000
      ```
    - `CREATE TABLE`, writes, and reads with `spark.sql.orc.impl=hive`, like 
the native ORC data source:
      ```
      [UNSUPPORTED_DATA_TYPE_FOR_DATASOURCE] The ORC datasource doesn't support 
the column `c` of the type "GEOMETRY(4326)". SQLSTATE: 0A000
      ```
    - Hive generic UDFs, UDAFs, and UDTFs, wrapped by 
`CANNOT_INSTANTIATE_HIVE_FUNCTION` as before, like `TIME` and the simple Hive 
UDFs:
      ```
      [UNSUPPORTED_DATATYPE] Unsupported data type "GEOMETRY(4326)". SQLSTATE: 
0A000
      ```
    
    Pass the CIs with the newly added test cases in `HiveInspectorSuite`, 
`InsertSuite`, `HiveUDFSuite`, and `HiveOrcQuerySuite`. All of them fail 
without this fix.
    
    Generated-by: Claude Opus 5.5
    
    Closes #59755 from dongjoon-hyun/SPARK-60519.
    
    Authored-by: Dongjoon Hyun <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
    (cherry picked from commit 935b76e3fd49ab8682b3c89c734928560f639ec0)
    Signed-off-by: Dongjoon Hyun <[email protected]>
    (cherry picked from commit 215dda27a3907a7aeb6248738ca6a334837cb03b)
---
 .../org/apache/spark/sql/hive/HiveInspectors.scala | 12 ++++++
 .../spark/sql/hive/execution/HiveFileFormat.scala  | 18 ++++++++-
 .../apache/spark/sql/hive/orc/OrcFileFormat.scala  |  1 +
 .../apache/spark/sql/hive/HiveInspectorSuite.scala | 22 ++++++++++-
 .../org/apache/spark/sql/hive/InsertSuite.scala    | 36 ++++++++++++++++++
 .../spark/sql/hive/execution/HiveUDFSuite.scala    | 31 +++++++++++++++
 .../spark/sql/hive/orc/HiveOrcQuerySuite.scala     | 44 +++++++++++++++++++++-
 7 files changed, 161 insertions(+), 3 deletions(-)

diff --git 
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala 
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala
index bf40327bd991..f5fbc3f63443 100644
--- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala
+++ b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala
@@ -849,6 +849,14 @@ private[hive] trait HiveInspectors {
     case _: UserDefinedType[_] =>
       val sqlType = dataType.asInstanceOf[UserDefinedType[_]].sqlType
       toInspector(sqlType)
+    // Hive has no geospatial types, so they cannot be represented by any Hive 
object inspector.
+    case _: GeometryType | _: GeographyType => throw 
unsupportedHiveType(dataType)
+  }
+
+  private def unsupportedHiveType(dataType: DataType): AnalysisException = {
+    new AnalysisException(
+      errorClass = "UNSUPPORTED_DATATYPE",
+      messageParameters = Map("typeName" -> toSQLType(dataType)))
   }
 
   /**
@@ -917,6 +925,10 @@ private[hive] trait HiveInspectors {
       toInspector(dt)
     case Literal(_, dt: UserDefinedType[_]) =>
       toInspector(dt.sqlType)
+    // Hive has no geospatial types, so a geospatial constant cannot be mapped 
to a Hive object
+    // inspector.
+    case Literal(_, dt @ (_: GeometryType | _: GeographyType)) =>
+      throw unsupportedHiveType(dt)
     // We will enumerate all of the possible constant expressions, throw 
exception if we missed
     case Literal(_, dt) =>
       throw SparkException.internalError(s"Hive doesn't support the constant 
type [$dt].")
diff --git 
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/HiveFileFormat.scala
 
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/HiveFileFormat.scala
index 5506cf8dae07..b4b56974b003 100644
--- 
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/HiveFileFormat.scala
+++ 
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/HiveFileFormat.scala
@@ -41,7 +41,7 @@ import 
org.apache.spark.sql.execution.datasources.{FileFormat, OutputWriter, Out
 import org.apache.spark.sql.hive.{HiveInspectors, HiveTableUtil}
 import org.apache.spark.sql.internal.SessionStateHelper
 import org.apache.spark.sql.sources.DataSourceRegister
-import org.apache.spark.sql.types.StructType
+import org.apache.spark.sql.types.{ArrayType, DataType, GeographyType, 
GeometryType, MapType, StructType, UserDefinedType}
 import org.apache.spark.util.SerializableJobConf
 
 /**
@@ -115,6 +115,22 @@ case class HiveFileFormat(fileSinkConf: FileSinkDesc)
     }
   }
 
+  override def supportDataType(dataType: DataType): Boolean = dataType match {
+    // Hive has no geospatial types, so they cannot be stored in a Hive serde 
table.
+    case _: GeometryType | _: GeographyType => false
+
+    case st: StructType => st.forall { f => supportDataType(f.dataType) }
+
+    case ArrayType(elementType, _) => supportDataType(elementType)
+
+    case MapType(keyType, valueType, _) =>
+      supportDataType(keyType) && supportDataType(valueType)
+
+    case udt: UserDefinedType[_] => supportDataType(udt.sqlType)
+
+    case _ => true
+  }
+
   override def supportFieldName(name: String): Boolean = {
     fileSinkConf.getTableInfo.getOutputFileFormatClassName match {
       case "org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat" =>
diff --git 
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFileFormat.scala 
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFileFormat.scala
index a16cbca0f3de..035a5bb0faea 100644
--- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFileFormat.scala
+++ b/sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFileFormat.scala
@@ -201,6 +201,7 @@ case class OrcFileFormat() extends FileFormat
 
     case _: AnsiIntervalType => false
     case _: TimeType => false
+    case _: GeometryType | _: GeographyType => false
     case _: AtomicType => true
 
     case st: StructType => st.forall { f => supportDataType(f.dataType) }
diff --git 
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveInspectorSuite.scala 
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveInspectorSuite.scala
index 8acabd579d44..2ae291939639 100644
--- a/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveInspectorSuite.scala
+++ b/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveInspectorSuite.scala
@@ -28,7 +28,7 @@ import org.apache.hadoop.hive.serde2.typeinfo.DecimalTypeInfo
 import org.apache.hadoop.io.LongWritable
 
 import org.apache.spark.SparkFunSuite
-import org.apache.spark.sql.{Row, TestUserClassUDT}
+import org.apache.spark.sql.{AnalysisException, Row, TestUserClassUDT}
 import org.apache.spark.sql.catalyst.InternalRow
 import org.apache.spark.sql.catalyst.expressions.Literal
 import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, 
GenericArrayData, MapData}
@@ -291,4 +291,24 @@ class HiveInspectorSuite extends SparkFunSuite with 
HiveInspectors {
     assert(typeInfo2.precision() === 18)
     assert(typeInfo2.scale() === 10)
   }
+
+  test("SPARK-60519: geospatial types are unsupported in Hive object 
inspectors") {
+    Seq(
+      GeometryType(4326), GeometryType("ANY"),
+      GeographyType(4326), GeographyType("ANY")).foreach { geoType =>
+      val expectedParams = Map("typeName" -> s"\"${geoType.sql}\"")
+      checkError(
+        exception = intercept[AnalysisException](toInspector(geoType)),
+        condition = "UNSUPPORTED_DATATYPE",
+        parameters = expectedParams)
+      checkError(
+        exception = 
intercept[AnalysisException](toInspector(Literal.create(null, geoType))),
+        condition = "UNSUPPORTED_DATATYPE",
+        parameters = expectedParams)
+      checkError(
+        exception = intercept[AnalysisException](geoType.toTypeInfo),
+        condition = "_LEGACY_ERROR_TEMP_3095",
+        parameters = Map("dt" -> s"\"${geoType.sql}\""))
+    }
+  }
 }
diff --git 
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/InsertSuite.scala 
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/InsertSuite.scala
index e50d3533d5d8..0e4745223876 100644
--- a/sql/hive/src/test/scala/org/apache/spark/sql/hive/InsertSuite.scala
+++ b/sql/hive/src/test/scala/org/apache/spark/sql/hive/InsertSuite.scala
@@ -705,6 +705,42 @@ class InsertSuite extends QueryTest with TestHiveSingleton 
with BeforeAndAfter {
     }
   }
 
+  test("SPARK-60519: geospatial types are unsupported when writing to a Hive 
serde directory") {
+    // WKB of POINT(1 2).
+    val wkb = "0101000000000000000000F03F0000000000000040"
+    // Disable native data source conversion so that PARQUET and ORC also go 
through the Hive
+    // serde path (HiveFileFormat) like TEXTFILE.
+    withSQLConf(HiveUtils.CONVERT_METASTORE_INSERT_DIR.key -> "false") {
+      Seq("TEXTFILE", "PARQUET", "ORC").foreach { fileFormat =>
+        Seq(
+          s"ST_GeomFromWKB(X'$wkb', 4326)" -> GeometryType(4326),
+          s"ST_GeogFromWKB(X'$wkb')" -> GeographyType(4326),
+          s"ARRAY(ST_GeomFromWKB(X'$wkb', 4326))" -> 
ArrayType(GeometryType(4326))
+        ).foreach { case (expr, dt) =>
+          withTempDir { dir =>
+            // InsertIntoHiveDirCommand wraps the failure in a SparkException, 
so assert on the
+            // cause.
+            val e = intercept[SparkException] {
+              sql(
+                s"""
+                   |INSERT OVERWRITE LOCAL DIRECTORY '${dir.toURI.getPath}'
+                   |STORED AS $fileFormat
+                   |SELECT $expr AS c
+                 """.stripMargin)
+            }
+            checkError(
+              exception = e.getCause.asInstanceOf[AnalysisException],
+              condition = "UNSUPPORTED_DATA_TYPE_FOR_DATASOURCE",
+              parameters = Map(
+                "columnName" -> "`c`",
+                "columnType" -> s"\"${dt.sql}\"",
+                "format" -> "Hive"))
+          }
+        }
+      }
+    }
+  }
+
   test("insert overwrite to dir from temp table") {
     withTempView("test_insert_table") {
       spark.range(10).selectExpr("id", "id AS 
str").createOrReplaceTempView("test_insert_table")
diff --git 
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveUDFSuite.scala
 
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveUDFSuite.scala
index 6a44e17296c0..405cfa422b05 100644
--- 
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveUDFSuite.scala
+++ 
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveUDFSuite.scala
@@ -41,6 +41,7 @@ import org.apache.spark.sql.execution.WholeStageCodegenExec
 import org.apache.spark.sql.functions.{call_function, max}
 import org.apache.spark.sql.hive.test.{TestHiveSingleton, TestUDTFJar}
 import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{GeographyType, GeometryType}
 import org.apache.spark.tags.SlowHiveTest
 import org.apache.spark.util.Utils
 
@@ -407,6 +408,36 @@ class HiveUDFSuite extends QueryTest with 
TestHiveSingleton {
     }
   }
 
+  test("SPARK-60519: geospatial types are unsupported as Hive UDF arguments") {
+    withUserDefinedFunction("testGenericUDFHash" -> true) {
+      sql(s"CREATE TEMPORARY FUNCTION testGenericUDFHash AS 
'${classOf[GenericUDFHash].getName}'")
+      // WKB of POINT(1 2).
+      val wkb = "0101000000000000000000F03F0000000000000040"
+      Seq(
+        // Non-foldable arguments, which are mapped to Hive object inspectors 
by their data types.
+        s"ST_GeomFromWKB(X'$wkb', 4326)" -> GeometryType(4326),
+        s"ST_GeogFromWKB(X'$wkb')" -> GeographyType(4326),
+        // Foldable arguments, which are mapped to constant Hive object 
inspectors.
+        "CAST(NULL AS GEOMETRY(4326))" -> GeometryType(4326),
+        "CAST(NULL AS GEOGRAPHY(4326))" -> GeographyType(4326)
+      ).foreach { case (arg, dt) =>
+        // The Hive UDF resolver wraps the failure in _LEGACY_ERROR_TEMP_3084.
+        val fragment = s"testGenericUDFHash($arg)"
+        val e = intercept[AnalysisException] {
+          sql(s"SELECT $fragment").collect()
+        }
+        checkError(
+          exception = e,
+          condition = "_LEGACY_ERROR_TEMP_3084",
+          parameters = Map(
+            "clazz" -> classOf[GenericUDFHash].getCanonicalName,
+            "e" -> (s"${classOf[AnalysisException].getName}: 
[UNSUPPORTED_DATATYPE] " +
+              s"Unsupported data type \"${dt.sql}\". SQLSTATE: 0A000")),
+          context = ExpectedContext(fragment = fragment, start = 7, stop = 6 + 
fragment.length))
+      }
+    }
+  }
+
   test("Hive UDFs with insufficient number of input arguments should trigger 
an analysis error") {
     withTempView("testUDF") {
       Seq((1, 2)).toDF("a", "b").createOrReplaceTempView("testUDF")
diff --git 
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/HiveOrcQuerySuite.scala 
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/HiveOrcQuerySuite.scala
index 45f5d2f273e3..8427c25c77f2 100644
--- 
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/HiveOrcQuerySuite.scala
+++ 
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/HiveOrcQuerySuite.scala
@@ -30,7 +30,7 @@ import 
org.apache.spark.sql.execution.datasources.orc.OrcQueryTest
 import org.apache.spark.sql.hive.{HiveSessionCatalog, HiveUtils}
 import org.apache.spark.sql.hive.test.TestHiveSingleton
 import org.apache.spark.sql.internal.SQLConf
-import org.apache.spark.sql.types.TimeType
+import org.apache.spark.sql.types.{GeographyType, GeometryType, TimeType}
 import org.apache.spark.util.Utils
 
 class HiveOrcQuerySuite extends OrcQueryTest with TestHiveSingleton {
@@ -442,4 +442,46 @@ class HiveOrcQuerySuite extends OrcQueryTest with 
TestHiveSingleton {
       }
     }
   }
+
+  test("SPARK-60519: geospatial types are unsupported in Hive ORC") {
+    withSQLConf(SQLConf.ORC_IMPLEMENTATION.key -> "hive") {
+      withTempDir { dir =>
+        val tempDir = new File(dir, "files").getCanonicalPath
+        // WKB of POINT(1 2).
+        val wkb = "0101000000000000000000F03F0000000000000040"
+        Seq(
+          s"ST_GeomFromWKB(X'$wkb', 4326)" -> GeometryType(4326),
+          s"ST_GeogFromWKB(X'$wkb')" -> GeographyType(4326)).foreach { case 
(expr, dt) =>
+          val expectedParams = Map(
+            "columnName" -> "`g`",
+            "columnType" -> s"\"${dt.sql}\"",
+            "format" -> "ORC")
+          // write path
+          checkError(
+            exception = intercept[AnalysisException] {
+              sql(s"select $expr as 
g").write.format("orc").mode("overwrite").save(tempDir)
+            },
+            condition = "UNSUPPORTED_DATA_TYPE_FOR_DATASOURCE",
+            parameters = expectedParams)
+          // read path
+          checkError(
+            exception = intercept[AnalysisException] {
+              sql(s"select X'$wkb' as 
g").write.format("orc").mode("overwrite").save(tempDir)
+              spark.read.schema(s"g 
${dt.sql}").format("orc").load(tempDir).collect()
+            },
+            condition = "UNSUPPORTED_DATA_TYPE_FOR_DATASOURCE",
+            parameters = expectedParams)
+          // CREATE TABLE
+          withTable("t") {
+            checkError(
+              exception = intercept[AnalysisException] {
+                sql(s"CREATE TABLE t (g ${dt.sql}) USING ORC")
+              },
+              condition = "UNSUPPORTED_DATA_TYPE_FOR_DATASOURCE",
+              parameters = expectedParams)
+          }
+        }
+      }
+    }
+  }
 }


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to