This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.2 by this push:
new 17b408fabd3f [SPARK-60519][SQL] Raise a clear error for `GEOMETRY` and
`GEOGRAPHY` in Hive SerDe interop
17b408fabd3f is described below
commit 17b408fabd3f78d4d13994792a83dd842457b5c4
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Sat Oct 24 09:54:37 2026 -0700
[SPARK-60519][SQL] Raise a clear error for `GEOMETRY` and `GEOGRAPHY` in
Hive SerDe interop
This PR aims to raise a clear error for the geospatial types, `GEOMETRY`
and `GEOGRAPHY`, in Hive SerDe interop like `TIME` (SPARK-57556) because Hive
has no geospatial types.
- `HiveFileFormat.supportDataType` rejects `GeometryType` and
`GeographyType`, including nested ones, so `INSERT OVERWRITE [LOCAL] DIRECTORY
... STORED AS` through the Hive SerDe fails with
`UNSUPPORTED_DATA_TYPE_FOR_DATASOURCE` before running a job.
- The Hive ORC `OrcFileFormat.supportDataType` (`spark.sql.orc.impl=hive`)
rejects them like the native ORC data source (SPARK-55262).
- `HiveInspectors.toInspector` throws `UNSUPPORTED_DATATYPE` for them
instead of `scala.MatchError` and `INTERNAL_ERROR`, like
`HiveInspectors.toTypeInfo` already does.
`INSERT OVERWRITE [LOCAL] DIRECTORY ... STORED AS PARQUET` with the default
configurations, `spark.sql.hive.convertMetastoreInsertDir=true` and
`spark.sql.hive.convertMetastoreParquet=true`, is not affected because it uses
the native Parquet data source, which supports the geospatial types.
Hive SerDe cannot parse `geometry(<srid>)` and `geography(<srid>)`. So, the
following fails inside a task after the job starts.
```sql
INSERT OVERWRITE LOCAL DIRECTORY '/tmp/geo' STORED AS TEXTFILE
SELECT 1 AS i,
ST_GeomFromWKB(X'0101000000000000000000F03F0000000000000040', 4326) AS c;
```
```
org.apache.spark.SparkException: Failed inserting overwrite directory
/tmp/geo
...
Caused by: java.lang.IllegalArgumentException: Error: type expected at the
position 4 of 'int:geometry(4326)' but 'geometry' is found.
```
With `spark.sql.orc.impl=hive`, a data source table with geospatial columns
is created while the native ORC data source rejects it. However, writing to it
fails inside a task in the same way. Reading ORC files with a user-specified
geospatial schema also fails inside a task with `ClassCastException`.
```sql
SET spark.sql.orc.impl=hive;
CREATE TABLE t (i INT, c GEOMETRY(4326)) USING ORC;
INSERT INTO t VALUES (1,
ST_GeomFromWKB(X'0101000000000000000000F03F0000000000000040', 4326));
```
```
org.apache.spark.SparkException: Job aborted due to stage failure: ...
java.lang.IllegalArgumentException: Error: type expected at the position 4 of
'int:geometry(4326)' but 'geometry' is found.
```
In addition, Hive generic UDFs, UDAFs, and UDTFs fail with
`scala.MatchError` or `INTERNAL_ERROR` for geospatial arguments.
```sql
CREATE TEMPORARY FUNCTION hive_hash AS
'org.apache.hadoop.hive.ql.udf.generic.GenericUDFHash';
SELECT
hive_hash(ST_GeomFromWKB(X'0101000000000000000000F03F0000000000000040', 4326));
SELECT hive_hash(CAST(NULL AS GEOMETRY(4326)));
```
```
[CANNOT_INSTANTIATE_HIVE_FUNCTION] ...
Caused by: scala.MatchError: GeometryType(4326) (of class
org.apache.spark.sql.types.GeometryType)
[CANNOT_INSTANTIATE_HIVE_FUNCTION] ...
Caused by: org.apache.spark.SparkException: [INTERNAL_ERROR] Hive doesn't
support the constant type [GeometryType(4326)]. SQLSTATE: XX000
```
Yes. The above cases fail with the following errors without running a job.
- Hive SerDe writes, wrapped by `Failed inserting overwrite directory` as
before:
```
[UNSUPPORTED_DATA_TYPE_FOR_DATASOURCE] The Hive datasource doesn't
support the column `c` of the type "GEOMETRY(4326)". SQLSTATE: 0A000
```
- `CREATE TABLE`, writes, and reads with `spark.sql.orc.impl=hive`, like
the native ORC data source:
```
[UNSUPPORTED_DATA_TYPE_FOR_DATASOURCE] The ORC datasource doesn't support
the column `c` of the type "GEOMETRY(4326)". SQLSTATE: 0A000
```
- Hive generic UDFs, UDAFs, and UDTFs, wrapped by
`CANNOT_INSTANTIATE_HIVE_FUNCTION` as before, like `TIME` and the simple Hive
UDFs:
```
[UNSUPPORTED_DATATYPE] Unsupported data type "GEOMETRY(4326)". SQLSTATE:
0A000
```
Pass the CIs with the newly added test cases in `HiveInspectorSuite`,
`InsertSuite`, `HiveUDFSuite`, and `HiveOrcQuerySuite`. All of them fail
without this fix.
Generated-by: Claude Opus 5.5
Closes #59755 from dongjoon-hyun/SPARK-60519.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit 935b76e3fd49ab8682b3c89c734928560f639ec0)
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit 215dda27a3907a7aeb6248738ca6a334837cb03b)
---
.../org/apache/spark/sql/hive/HiveInspectors.scala | 12 ++++++
.../spark/sql/hive/execution/HiveFileFormat.scala | 18 ++++++++-
.../apache/spark/sql/hive/orc/OrcFileFormat.scala | 1 +
.../apache/spark/sql/hive/HiveInspectorSuite.scala | 22 ++++++++++-
.../org/apache/spark/sql/hive/InsertSuite.scala | 36 ++++++++++++++++++
.../spark/sql/hive/execution/HiveUDFSuite.scala | 31 +++++++++++++++
.../spark/sql/hive/orc/HiveOrcQuerySuite.scala | 44 +++++++++++++++++++++-
7 files changed, 161 insertions(+), 3 deletions(-)
diff --git
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala
index bf40327bd991..f5fbc3f63443 100644
--- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala
+++ b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala
@@ -849,6 +849,14 @@ private[hive] trait HiveInspectors {
case _: UserDefinedType[_] =>
val sqlType = dataType.asInstanceOf[UserDefinedType[_]].sqlType
toInspector(sqlType)
+ // Hive has no geospatial types, so they cannot be represented by any Hive
object inspector.
+ case _: GeometryType | _: GeographyType => throw
unsupportedHiveType(dataType)
+ }
+
+ private def unsupportedHiveType(dataType: DataType): AnalysisException = {
+ new AnalysisException(
+ errorClass = "UNSUPPORTED_DATATYPE",
+ messageParameters = Map("typeName" -> toSQLType(dataType)))
}
/**
@@ -917,6 +925,10 @@ private[hive] trait HiveInspectors {
toInspector(dt)
case Literal(_, dt: UserDefinedType[_]) =>
toInspector(dt.sqlType)
+ // Hive has no geospatial types, so a geospatial constant cannot be mapped
to a Hive object
+ // inspector.
+ case Literal(_, dt @ (_: GeometryType | _: GeographyType)) =>
+ throw unsupportedHiveType(dt)
// We will enumerate all of the possible constant expressions, throw
exception if we missed
case Literal(_, dt) =>
throw SparkException.internalError(s"Hive doesn't support the constant
type [$dt].")
diff --git
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/HiveFileFormat.scala
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/HiveFileFormat.scala
index 5506cf8dae07..b4b56974b003 100644
---
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/HiveFileFormat.scala
+++
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/HiveFileFormat.scala
@@ -41,7 +41,7 @@ import
org.apache.spark.sql.execution.datasources.{FileFormat, OutputWriter, Out
import org.apache.spark.sql.hive.{HiveInspectors, HiveTableUtil}
import org.apache.spark.sql.internal.SessionStateHelper
import org.apache.spark.sql.sources.DataSourceRegister
-import org.apache.spark.sql.types.StructType
+import org.apache.spark.sql.types.{ArrayType, DataType, GeographyType,
GeometryType, MapType, StructType, UserDefinedType}
import org.apache.spark.util.SerializableJobConf
/**
@@ -115,6 +115,22 @@ case class HiveFileFormat(fileSinkConf: FileSinkDesc)
}
}
+ override def supportDataType(dataType: DataType): Boolean = dataType match {
+ // Hive has no geospatial types, so they cannot be stored in a Hive serde
table.
+ case _: GeometryType | _: GeographyType => false
+
+ case st: StructType => st.forall { f => supportDataType(f.dataType) }
+
+ case ArrayType(elementType, _) => supportDataType(elementType)
+
+ case MapType(keyType, valueType, _) =>
+ supportDataType(keyType) && supportDataType(valueType)
+
+ case udt: UserDefinedType[_] => supportDataType(udt.sqlType)
+
+ case _ => true
+ }
+
override def supportFieldName(name: String): Boolean = {
fileSinkConf.getTableInfo.getOutputFileFormatClassName match {
case "org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat" =>
diff --git
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFileFormat.scala
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFileFormat.scala
index a16cbca0f3de..035a5bb0faea 100644
--- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFileFormat.scala
+++ b/sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFileFormat.scala
@@ -201,6 +201,7 @@ case class OrcFileFormat() extends FileFormat
case _: AnsiIntervalType => false
case _: TimeType => false
+ case _: GeometryType | _: GeographyType => false
case _: AtomicType => true
case st: StructType => st.forall { f => supportDataType(f.dataType) }
diff --git
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveInspectorSuite.scala
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveInspectorSuite.scala
index 8acabd579d44..2ae291939639 100644
--- a/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveInspectorSuite.scala
+++ b/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveInspectorSuite.scala
@@ -28,7 +28,7 @@ import org.apache.hadoop.hive.serde2.typeinfo.DecimalTypeInfo
import org.apache.hadoop.io.LongWritable
import org.apache.spark.SparkFunSuite
-import org.apache.spark.sql.{Row, TestUserClassUDT}
+import org.apache.spark.sql.{AnalysisException, Row, TestUserClassUDT}
import org.apache.spark.sql.catalyst.InternalRow
import org.apache.spark.sql.catalyst.expressions.Literal
import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData,
GenericArrayData, MapData}
@@ -291,4 +291,24 @@ class HiveInspectorSuite extends SparkFunSuite with
HiveInspectors {
assert(typeInfo2.precision() === 18)
assert(typeInfo2.scale() === 10)
}
+
+ test("SPARK-60519: geospatial types are unsupported in Hive object
inspectors") {
+ Seq(
+ GeometryType(4326), GeometryType("ANY"),
+ GeographyType(4326), GeographyType("ANY")).foreach { geoType =>
+ val expectedParams = Map("typeName" -> s"\"${geoType.sql}\"")
+ checkError(
+ exception = intercept[AnalysisException](toInspector(geoType)),
+ condition = "UNSUPPORTED_DATATYPE",
+ parameters = expectedParams)
+ checkError(
+ exception =
intercept[AnalysisException](toInspector(Literal.create(null, geoType))),
+ condition = "UNSUPPORTED_DATATYPE",
+ parameters = expectedParams)
+ checkError(
+ exception = intercept[AnalysisException](geoType.toTypeInfo),
+ condition = "_LEGACY_ERROR_TEMP_3095",
+ parameters = Map("dt" -> s"\"${geoType.sql}\""))
+ }
+ }
}
diff --git
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/InsertSuite.scala
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/InsertSuite.scala
index e50d3533d5d8..0e4745223876 100644
--- a/sql/hive/src/test/scala/org/apache/spark/sql/hive/InsertSuite.scala
+++ b/sql/hive/src/test/scala/org/apache/spark/sql/hive/InsertSuite.scala
@@ -705,6 +705,42 @@ class InsertSuite extends QueryTest with TestHiveSingleton
with BeforeAndAfter {
}
}
+ test("SPARK-60519: geospatial types are unsupported when writing to a Hive
serde directory") {
+ // WKB of POINT(1 2).
+ val wkb = "0101000000000000000000F03F0000000000000040"
+ // Disable native data source conversion so that PARQUET and ORC also go
through the Hive
+ // serde path (HiveFileFormat) like TEXTFILE.
+ withSQLConf(HiveUtils.CONVERT_METASTORE_INSERT_DIR.key -> "false") {
+ Seq("TEXTFILE", "PARQUET", "ORC").foreach { fileFormat =>
+ Seq(
+ s"ST_GeomFromWKB(X'$wkb', 4326)" -> GeometryType(4326),
+ s"ST_GeogFromWKB(X'$wkb')" -> GeographyType(4326),
+ s"ARRAY(ST_GeomFromWKB(X'$wkb', 4326))" ->
ArrayType(GeometryType(4326))
+ ).foreach { case (expr, dt) =>
+ withTempDir { dir =>
+ // InsertIntoHiveDirCommand wraps the failure in a SparkException,
so assert on the
+ // cause.
+ val e = intercept[SparkException] {
+ sql(
+ s"""
+ |INSERT OVERWRITE LOCAL DIRECTORY '${dir.toURI.getPath}'
+ |STORED AS $fileFormat
+ |SELECT $expr AS c
+ """.stripMargin)
+ }
+ checkError(
+ exception = e.getCause.asInstanceOf[AnalysisException],
+ condition = "UNSUPPORTED_DATA_TYPE_FOR_DATASOURCE",
+ parameters = Map(
+ "columnName" -> "`c`",
+ "columnType" -> s"\"${dt.sql}\"",
+ "format" -> "Hive"))
+ }
+ }
+ }
+ }
+ }
+
test("insert overwrite to dir from temp table") {
withTempView("test_insert_table") {
spark.range(10).selectExpr("id", "id AS
str").createOrReplaceTempView("test_insert_table")
diff --git
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveUDFSuite.scala
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveUDFSuite.scala
index 6a44e17296c0..405cfa422b05 100644
---
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveUDFSuite.scala
+++
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveUDFSuite.scala
@@ -41,6 +41,7 @@ import org.apache.spark.sql.execution.WholeStageCodegenExec
import org.apache.spark.sql.functions.{call_function, max}
import org.apache.spark.sql.hive.test.{TestHiveSingleton, TestUDTFJar}
import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{GeographyType, GeometryType}
import org.apache.spark.tags.SlowHiveTest
import org.apache.spark.util.Utils
@@ -407,6 +408,36 @@ class HiveUDFSuite extends QueryTest with
TestHiveSingleton {
}
}
+ test("SPARK-60519: geospatial types are unsupported as Hive UDF arguments") {
+ withUserDefinedFunction("testGenericUDFHash" -> true) {
+ sql(s"CREATE TEMPORARY FUNCTION testGenericUDFHash AS
'${classOf[GenericUDFHash].getName}'")
+ // WKB of POINT(1 2).
+ val wkb = "0101000000000000000000F03F0000000000000040"
+ Seq(
+ // Non-foldable arguments, which are mapped to Hive object inspectors
by their data types.
+ s"ST_GeomFromWKB(X'$wkb', 4326)" -> GeometryType(4326),
+ s"ST_GeogFromWKB(X'$wkb')" -> GeographyType(4326),
+ // Foldable arguments, which are mapped to constant Hive object
inspectors.
+ "CAST(NULL AS GEOMETRY(4326))" -> GeometryType(4326),
+ "CAST(NULL AS GEOGRAPHY(4326))" -> GeographyType(4326)
+ ).foreach { case (arg, dt) =>
+ // The Hive UDF resolver wraps the failure in _LEGACY_ERROR_TEMP_3084.
+ val fragment = s"testGenericUDFHash($arg)"
+ val e = intercept[AnalysisException] {
+ sql(s"SELECT $fragment").collect()
+ }
+ checkError(
+ exception = e,
+ condition = "_LEGACY_ERROR_TEMP_3084",
+ parameters = Map(
+ "clazz" -> classOf[GenericUDFHash].getCanonicalName,
+ "e" -> (s"${classOf[AnalysisException].getName}:
[UNSUPPORTED_DATATYPE] " +
+ s"Unsupported data type \"${dt.sql}\". SQLSTATE: 0A000")),
+ context = ExpectedContext(fragment = fragment, start = 7, stop = 6 +
fragment.length))
+ }
+ }
+ }
+
test("Hive UDFs with insufficient number of input arguments should trigger
an analysis error") {
withTempView("testUDF") {
Seq((1, 2)).toDF("a", "b").createOrReplaceTempView("testUDF")
diff --git
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/HiveOrcQuerySuite.scala
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/HiveOrcQuerySuite.scala
index 45f5d2f273e3..8427c25c77f2 100644
---
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/HiveOrcQuerySuite.scala
+++
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/HiveOrcQuerySuite.scala
@@ -30,7 +30,7 @@ import
org.apache.spark.sql.execution.datasources.orc.OrcQueryTest
import org.apache.spark.sql.hive.{HiveSessionCatalog, HiveUtils}
import org.apache.spark.sql.hive.test.TestHiveSingleton
import org.apache.spark.sql.internal.SQLConf
-import org.apache.spark.sql.types.TimeType
+import org.apache.spark.sql.types.{GeographyType, GeometryType, TimeType}
import org.apache.spark.util.Utils
class HiveOrcQuerySuite extends OrcQueryTest with TestHiveSingleton {
@@ -442,4 +442,46 @@ class HiveOrcQuerySuite extends OrcQueryTest with
TestHiveSingleton {
}
}
}
+
+ test("SPARK-60519: geospatial types are unsupported in Hive ORC") {
+ withSQLConf(SQLConf.ORC_IMPLEMENTATION.key -> "hive") {
+ withTempDir { dir =>
+ val tempDir = new File(dir, "files").getCanonicalPath
+ // WKB of POINT(1 2).
+ val wkb = "0101000000000000000000F03F0000000000000040"
+ Seq(
+ s"ST_GeomFromWKB(X'$wkb', 4326)" -> GeometryType(4326),
+ s"ST_GeogFromWKB(X'$wkb')" -> GeographyType(4326)).foreach { case
(expr, dt) =>
+ val expectedParams = Map(
+ "columnName" -> "`g`",
+ "columnType" -> s"\"${dt.sql}\"",
+ "format" -> "ORC")
+ // write path
+ checkError(
+ exception = intercept[AnalysisException] {
+ sql(s"select $expr as
g").write.format("orc").mode("overwrite").save(tempDir)
+ },
+ condition = "UNSUPPORTED_DATA_TYPE_FOR_DATASOURCE",
+ parameters = expectedParams)
+ // read path
+ checkError(
+ exception = intercept[AnalysisException] {
+ sql(s"select X'$wkb' as
g").write.format("orc").mode("overwrite").save(tempDir)
+ spark.read.schema(s"g
${dt.sql}").format("orc").load(tempDir).collect()
+ },
+ condition = "UNSUPPORTED_DATA_TYPE_FOR_DATASOURCE",
+ parameters = expectedParams)
+ // CREATE TABLE
+ withTable("t") {
+ checkError(
+ exception = intercept[AnalysisException] {
+ sql(s"CREATE TABLE t (g ${dt.sql}) USING ORC")
+ },
+ condition = "UNSUPPORTED_DATA_TYPE_FOR_DATASOURCE",
+ parameters = expectedParams)
+ }
+ }
+ }
+ }
+ }
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]