This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 508772274502 [SPARK-60514][SQL] Mark `GEOMETRY` and `GEOGRAPHY` as 
Hive incompatible data types
508772274502 is described below

commit 508772274502054fbc832d9a21f31ba71caeab62
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Sun Oct 25 11:38:36 2026 -0700

    [SPARK-60514][SQL] Mark `GEOMETRY` and `GEOGRAPHY` as Hive incompatible 
data types
    
    ### What changes were proposed in this pull request?
    
    This PR aims to mark the geospatial types, `GEOMETRY` and `GEOGRAPHY`, as 
Hive incompatible data types in `HiveExternalCatalog.isHiveCompatibleDataType`, 
like `TIMESTAMP_NTZ`, ANSI intervals, the nanosecond timestamp types, `TIME`, 
and `VARIANT`.
    
    Hive has no geospatial types, and the Hive SerDe cannot parse 
`geometry(<srid>)` and `geography(<srid>)`. However, `isHiveCompatibleDataType` 
returns `true` for `GeometryType` and `GeographyType`, so `HiveExternalCatalog` 
tries to store them in Hive metastore as they are. With this PR,
    - Data source tables with geospatial data columns are stored in the Spark 
SQL specific format without trying the Hive compatible format first.
    - Views with geospatial columns are stored with the empty Hive metastore 
schema without trying the actual schema first.
    
    Geospatial partition columns are not stored in Hive metastore because 
SPARK-60518 (#59754) disallows them at analysis.
    
    ### Why are the changes needed?
    
    A data source table with geospatial data columns, including nested ones 
like `ARRAY<GEOMETRY(4326)>`, is created only after a failed attempt in the 
Hive compatible format because the Hive SerDe cannot parse `geometry(4326)`.
    
    ```sql
    CREATE TABLE t (i INT, c GEOMETRY(4326)) USING PARQUET;
    ```
    
    ```
    WARN HiveExternalCatalog: Could not persist `spark_catalog`.`default`.`t` 
in a Hive compatible way. Persisting it into Hive metastore in Spark SQL 
specific format.
    org.apache.hadoop.hive.ql.metadata.HiveException: 
java.lang.IllegalArgumentException: Error: type expected at the position 4 of 
'int:geometry(4326)' but 'geometry' is found.
    ```
    
    In addition, Hive metastore accepts nested geospatial types of views, so 
the following view is stored with `c array<geometry(4326)>` in Hive metastore, 
which Hive cannot parse.
    
    ```sql
    CREATE VIEW v AS SELECT 
ARRAY(ST_GeomFromWKB(X'0101000000000000000000F03F0000000000000040', 4326)) AS c;
    ```
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes for the geospatial types.
    - For data source tables with geospatial data columns, Spark logs the 
following instead of the above warning with the stack trace. The table metadata 
in Hive metastore is the same as before.
      ```
      WARN HiveExternalCatalog: Hive incompatible types found: geometry(4326). 
Persisting data source table `spark_catalog`.`default`.`t` into Hive metastore 
in Spark SQL specific format, which is NOT compatible with Hive.
      ```
    - Views with nested geospatial types, e.g., `ARRAY<GEOMETRY(4326)>`, have 
the empty Hive metastore schema instead of `array<geometry(4326)>`, which Hive 
cannot parse. Spark restores the view schema from the table properties as 
before.
    - A Hive serde table whose data columns come from the SerDe, e.g., `STORED 
AS AVRO` with `avro.schema.literal` and no column list, fails with 
`CANNOT_RECOGNIZE_HIVE_TYPE` instead of `INVALID_PARTITION_COLUMN_DATA_TYPE` 
when it has a geospatial partition column, like `TIMESTAMP_NTZ`, `TIME`, and 
`VARIANT`. The table is not created either way.
    
    ### How was this patch tested?
    
    Pass the CIs with the newly added test cases in 
`MetastoreDataSourcesSuite`. Both test cases fail without this fix.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    Closes #59752 from dongjoon-hyun/SPARK-60514.
    
    Authored-by: Dongjoon Hyun <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 .../spark/sql/hive/HiveExternalCatalog.scala       |  1 +
 .../spark/sql/hive/MetastoreDataSourcesSuite.scala | 59 ++++++++++++++++++++++
 2 files changed, 60 insertions(+)

diff --git 
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala 
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala
index e7997c7e862f..c936d1633991 100644
--- 
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala
+++ 
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala
@@ -1554,6 +1554,7 @@ object HiveExternalCatalog {
     case _: AnyTimestampNanoType => false
     case _: TimeType => false
     case _: VariantType => false
+    case _: GeometryType | _: GeographyType => false
     case s: StructType =>
       s.forall { f =>
         // Because HMS thrift API passes StructType's catalog string in a 
whole, we can not
diff --git 
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/MetastoreDataSourcesSuite.scala
 
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/MetastoreDataSourcesSuite.scala
index 8730cb178440..c339b8130689 100644
--- 
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/MetastoreDataSourcesSuite.scala
+++ 
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/MetastoreDataSourcesSuite.scala
@@ -1800,6 +1800,65 @@ class MetastoreDataSourcesSuite extends QueryTest
     }
   }
 
+  test("SPARK-60514: GEOMETRY and GEOGRAPHY columns are stored in Spark SQL 
specific format") {
+    withTable("t") {
+      val logAppender =
+        new LogAppender("Check that geospatial types are reported as Hive 
incompatible")
+      logAppender.setThreshold(Level.WARN)
+      withLogAppender(logAppender) {
+        sql(
+          """
+            |CREATE TABLE t(
+            |  c1 GEOMETRY(4326),
+            |  c2 GEOGRAPHY(4326),
+            |  c3 ARRAY<GEOMETRY(0)>,
+            |  c4 STRUCT<a: INT, b: GEOGRAPHY(4326)>,
+            |  c5 MAP<STRING, GEOMETRY(4326)>
+            |) USING Parquet""".stripMargin)
+      }
+      val expectedMsg = "Hive incompatible types found: geometry(4326), 
geography(4326), " +
+        "array<geometry(0)>, struct<a:int,b:geography(4326)>, 
map<string,geometry(4326)>. " +
+        s"Persisting data source table `$SESSION_CATALOG_NAME`.`default`.`t` 
into Hive " +
+        "metastore in Spark SQL specific format, which is NOT compatible with 
Hive."
+      val actualMessages = logAppender.loggingEvents
+        .map(_.getMessage.getFormattedMessage)
+        .filter(_.contains("incompatible"))
+      assert(actualMessages.contains(expectedMsg))
+      assert(hiveClient.getTable("default", "t").schema
+        .forall(_.dataType == ArrayType(StringType)))
+      assert(sql("SELECT * FROM t").schema ===
+        StructType(Seq(
+          StructField("c1", GeometryType(4326)),
+          StructField("c2", GeographyType(4326)),
+          StructField("c3", ArrayType(GeometryType(0))),
+          StructField("c4",
+            StructType(Seq(
+              StructField("a", IntegerType),
+              StructField("b", GeographyType(4326))))),
+          StructField("c5", MapType(StringType, GeometryType(4326))))))
+    }
+  }
+
+  test("SPARK-60514: views with GEOMETRY and GEOGRAPHY columns are stored with 
" +
+    "the empty Hive metastore schema") {
+    // WKB of POINT(1 2).
+    val wkb = "0101000000000000000000F03F0000000000000040"
+    withView("v") {
+      // Hive metastore accepts nested geospatial types of views, which Hive 
cannot parse.
+      sql(s"CREATE VIEW v AS SELECT ARRAY(ST_GeomFromWKB(X'$wkb', 4326)) AS 
c1")
+      assert(hiveClient.getTable("default", "v").schema
+        .forall(_.dataType == ArrayType(StringType)))
+      checkAnswer(sql("SELECT hex(ST_AsBinary(c1[0])), typeof(c1) FROM v"),
+        Row(wkb, "array<geometry(4326)>"))
+
+      sql(s"ALTER VIEW v AS SELECT 1 AS c1, NAMED_STRUCT('x', 
ST_GeogFromWKB(X'$wkb')) AS c2")
+      assert(hiveClient.getTable("default", "v").schema
+        .forall(_.dataType == ArrayType(StringType)))
+      checkAnswer(sql("SELECT c1, hex(ST_AsBinary(c2.x)), typeof(c2) FROM v"),
+        Row(1, wkb, "struct<x:geography(4326)>"))
+    }
+  }
+
   private def withDebugMode(f: => Unit): Unit = {
     val previousValue = sparkSession.sparkContext.conf.get(DEBUG_MODE)
     try {


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to