This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 508772274502 [SPARK-60514][SQL] Mark `GEOMETRY` and `GEOGRAPHY` as
Hive incompatible data types
508772274502 is described below
commit 508772274502054fbc832d9a21f31ba71caeab62
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Sun Oct 25 11:38:36 2026 -0700
[SPARK-60514][SQL] Mark `GEOMETRY` and `GEOGRAPHY` as Hive incompatible
data types
### What changes were proposed in this pull request?
This PR aims to mark the geospatial types, `GEOMETRY` and `GEOGRAPHY`, as
Hive incompatible data types in `HiveExternalCatalog.isHiveCompatibleDataType`,
like `TIMESTAMP_NTZ`, ANSI intervals, the nanosecond timestamp types, `TIME`,
and `VARIANT`.
Hive has no geospatial types, and the Hive SerDe cannot parse
`geometry(<srid>)` and `geography(<srid>)`. However, `isHiveCompatibleDataType`
returns `true` for `GeometryType` and `GeographyType`, so `HiveExternalCatalog`
tries to store them in Hive metastore as they are. With this PR,
- Data source tables with geospatial data columns are stored in the Spark
SQL specific format without trying the Hive compatible format first.
- Views with geospatial columns are stored with the empty Hive metastore
schema without trying the actual schema first.
Geospatial partition columns are not stored in Hive metastore because
SPARK-60518 (#59754) disallows them at analysis.
### Why are the changes needed?
A data source table with geospatial data columns, including nested ones
like `ARRAY<GEOMETRY(4326)>`, is created only after a failed attempt in the
Hive compatible format because the Hive SerDe cannot parse `geometry(4326)`.
```sql
CREATE TABLE t (i INT, c GEOMETRY(4326)) USING PARQUET;
```
```
WARN HiveExternalCatalog: Could not persist `spark_catalog`.`default`.`t`
in a Hive compatible way. Persisting it into Hive metastore in Spark SQL
specific format.
org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.IllegalArgumentException: Error: type expected at the position 4 of
'int:geometry(4326)' but 'geometry' is found.
```
In addition, Hive metastore accepts nested geospatial types of views, so
the following view is stored with `c array<geometry(4326)>` in Hive metastore,
which Hive cannot parse.
```sql
CREATE VIEW v AS SELECT
ARRAY(ST_GeomFromWKB(X'0101000000000000000000F03F0000000000000040', 4326)) AS c;
```
### Does this PR introduce _any_ user-facing change?
Yes for the geospatial types.
- For data source tables with geospatial data columns, Spark logs the
following instead of the above warning with the stack trace. The table metadata
in Hive metastore is the same as before.
```
WARN HiveExternalCatalog: Hive incompatible types found: geometry(4326).
Persisting data source table `spark_catalog`.`default`.`t` into Hive metastore
in Spark SQL specific format, which is NOT compatible with Hive.
```
- Views with nested geospatial types, e.g., `ARRAY<GEOMETRY(4326)>`, have
the empty Hive metastore schema instead of `array<geometry(4326)>`, which Hive
cannot parse. Spark restores the view schema from the table properties as
before.
- A Hive serde table whose data columns come from the SerDe, e.g., `STORED
AS AVRO` with `avro.schema.literal` and no column list, fails with
`CANNOT_RECOGNIZE_HIVE_TYPE` instead of `INVALID_PARTITION_COLUMN_DATA_TYPE`
when it has a geospatial partition column, like `TIMESTAMP_NTZ`, `TIME`, and
`VARIANT`. The table is not created either way.
### How was this patch tested?
Pass the CIs with the newly added test cases in
`MetastoreDataSourcesSuite`. Both test cases fail without this fix.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #59752 from dongjoon-hyun/SPARK-60514.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../spark/sql/hive/HiveExternalCatalog.scala | 1 +
.../spark/sql/hive/MetastoreDataSourcesSuite.scala | 59 ++++++++++++++++++++++
2 files changed, 60 insertions(+)
diff --git
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala
index e7997c7e862f..c936d1633991 100644
---
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala
+++
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala
@@ -1554,6 +1554,7 @@ object HiveExternalCatalog {
case _: AnyTimestampNanoType => false
case _: TimeType => false
case _: VariantType => false
+ case _: GeometryType | _: GeographyType => false
case s: StructType =>
s.forall { f =>
// Because HMS thrift API passes StructType's catalog string in a
whole, we can not
diff --git
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/MetastoreDataSourcesSuite.scala
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/MetastoreDataSourcesSuite.scala
index 8730cb178440..c339b8130689 100644
---
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/MetastoreDataSourcesSuite.scala
+++
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/MetastoreDataSourcesSuite.scala
@@ -1800,6 +1800,65 @@ class MetastoreDataSourcesSuite extends QueryTest
}
}
+ test("SPARK-60514: GEOMETRY and GEOGRAPHY columns are stored in Spark SQL
specific format") {
+ withTable("t") {
+ val logAppender =
+ new LogAppender("Check that geospatial types are reported as Hive
incompatible")
+ logAppender.setThreshold(Level.WARN)
+ withLogAppender(logAppender) {
+ sql(
+ """
+ |CREATE TABLE t(
+ | c1 GEOMETRY(4326),
+ | c2 GEOGRAPHY(4326),
+ | c3 ARRAY<GEOMETRY(0)>,
+ | c4 STRUCT<a: INT, b: GEOGRAPHY(4326)>,
+ | c5 MAP<STRING, GEOMETRY(4326)>
+ |) USING Parquet""".stripMargin)
+ }
+ val expectedMsg = "Hive incompatible types found: geometry(4326),
geography(4326), " +
+ "array<geometry(0)>, struct<a:int,b:geography(4326)>,
map<string,geometry(4326)>. " +
+ s"Persisting data source table `$SESSION_CATALOG_NAME`.`default`.`t`
into Hive " +
+ "metastore in Spark SQL specific format, which is NOT compatible with
Hive."
+ val actualMessages = logAppender.loggingEvents
+ .map(_.getMessage.getFormattedMessage)
+ .filter(_.contains("incompatible"))
+ assert(actualMessages.contains(expectedMsg))
+ assert(hiveClient.getTable("default", "t").schema
+ .forall(_.dataType == ArrayType(StringType)))
+ assert(sql("SELECT * FROM t").schema ===
+ StructType(Seq(
+ StructField("c1", GeometryType(4326)),
+ StructField("c2", GeographyType(4326)),
+ StructField("c3", ArrayType(GeometryType(0))),
+ StructField("c4",
+ StructType(Seq(
+ StructField("a", IntegerType),
+ StructField("b", GeographyType(4326))))),
+ StructField("c5", MapType(StringType, GeometryType(4326))))))
+ }
+ }
+
+ test("SPARK-60514: views with GEOMETRY and GEOGRAPHY columns are stored with
" +
+ "the empty Hive metastore schema") {
+ // WKB of POINT(1 2).
+ val wkb = "0101000000000000000000F03F0000000000000040"
+ withView("v") {
+ // Hive metastore accepts nested geospatial types of views, which Hive
cannot parse.
+ sql(s"CREATE VIEW v AS SELECT ARRAY(ST_GeomFromWKB(X'$wkb', 4326)) AS
c1")
+ assert(hiveClient.getTable("default", "v").schema
+ .forall(_.dataType == ArrayType(StringType)))
+ checkAnswer(sql("SELECT hex(ST_AsBinary(c1[0])), typeof(c1) FROM v"),
+ Row(wkb, "array<geometry(4326)>"))
+
+ sql(s"ALTER VIEW v AS SELECT 1 AS c1, NAMED_STRUCT('x',
ST_GeogFromWKB(X'$wkb')) AS c2")
+ assert(hiveClient.getTable("default", "v").schema
+ .forall(_.dataType == ArrayType(StringType)))
+ checkAnswer(sql("SELECT c1, hex(ST_AsBinary(c2.x)), typeof(c2) FROM v"),
+ Row(1, wkb, "struct<x:geography(4326)>"))
+ }
+ }
+
private def withDebugMode(f: => Unit): Unit = {
val previousValue = sparkSession.sparkContext.conf.get(DEBUG_MODE)
try {
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]