This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new 2346a34b315a [SPARK-60518][SQL] Disallow `GEOMETRY` and `GEOGRAPHY` 
partition columns
2346a34b315a is described below

commit 2346a34b315a9f26de950d561246a586968fce84
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Sat Oct 24 11:07:46 2026 -0700

    [SPARK-60518][SQL] Disallow `GEOMETRY` and `GEOGRAPHY` partition columns
    
    ### What changes were proposed in this pull request?
    
    This PR aims to disallow the geospatial types, `GEOMETRY` and `GEOGRAPHY`, 
as partition column types in `PartitioningUtils.canPartitionOn`, like `VARIANT` 
(SPARK-45827) and non-default collated strings (SPARK-47015).
    
    After this PR, `CREATE TABLE`, CTAS, `saveAsTable`, and path-based writes 
with a geospatial partition column fail at analysis with 
`INVALID_PARTITION_COLUMN_DATA_TYPE` for all file data sources and Hive serde 
tables.
    
    ### Why are the changes needed?
    
    `canPartitionOn` accepts `GeometryType` and `GeographyType` because they 
are `AtomicType`s. However, Spark stores partition values as strings in 
directory names and in the catalog, and geospatial values cannot be cast to or 
from strings. So, geospatial partition values can be neither written nor read.
    
    - A write can succeed, e.g., when the partition value is a constant 
expression, but the directory name is the `Object.toString` result of the 
internal value, so the value is lost.
      ```scala
      scala> sql("SELECT 1 AS i, 
ST_GeomFromWKB(X'0101000000000000000000F03F0000000000000040', 4326) AS 
p").write.partitionBy("p").parquet("/tmp/geo")
      ```
      ```
      $ ls /tmp/geo
      _SUCCESS
      p=org.apache.spark.unsafe.types.BinaryView15dc1861
      ```
      Reading it back gives `p` as a `STRING` column with this text, or fails 
with `INVALID_PARTITION_VALUE` with the user-specified schema, `i INT, p 
GEOMETRY(4326)`.
    - Otherwise, e.g., `INSERT INTO` a table with a geospatial partition 
column, a write fails with an internal error because `V1Writes` adds a `Sort` 
on the non-orderable geospatial column.
      ```
      org.apache.spark.SparkException: [PLAN_VALIDATION_FAILED_RULE_IN_BATCH] 
Rule org.apache.spark.sql.execution.datasources.V1Writes in batch Early Filter 
and Projection Push-Down generated an invalid plan: The plan becomes 
unresolved: ...
      ```
    - `INSERT INTO` with a static partition spec, and reading a table created 
by CTAS or `saveAsTable`, fail with `INTERNAL_ERROR` (`scala.MatchError: 
StringType`) because there is no cast from `STRING` to the geospatial types.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. Geospatial partition columns are rejected at analysis instead of 
creating tables and directories which cannot be used.
    
    ```
    org.apache.spark.sql.AnalysisException: 
[INVALID_PARTITION_COLUMN_DATA_TYPE] Cannot use "GEOMETRY(4326)" for partition 
column. SQLSTATE: 0A000
    ```
    
    This includes Hive serde tables and data source tables in Hive metastore, 
which previously failed with the following error from Hive metastore.
    
    ```
    org.apache.spark.sql.AnalysisException: 
org.apache.hadoop.hive.ql.metadata.HiveException: 
InvalidObjectException(message:Invalid partition column type: geometry(4326))
    ```
    
    The geospatial types are enabled by default since Apache Spark 4.2.0 
([v4.2.0](https://github.com/apache/spark/releases/tag/v4.2.0), 2026-07-11).
    
    ### How was this patch tested?
    
    Pass the CIs with the newly added test case in `FileBasedDataSourceSuite`, 
which fails without this fix.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    Closes #59754 from dongjoon-hyun/SPARK-60518.
    
    Authored-by: Dongjoon Hyun <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
    (cherry picked from commit 5b562f0665096129126ed3a161ed1943de03154f)
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 .../execution/datasources/PartitioningUtils.scala  |  2 +
 .../spark/sql/FileBasedDataSourceSuite.scala       | 50 ++++++++++++++++++++++
 2 files changed, 52 insertions(+)

diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala
index 43584858c329..ef45e49021ed 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala
@@ -631,6 +631,8 @@ object PartitioningUtils extends SQLConfHelper {
     // non default collated strings should not be used as partition columns
     // as we cannot implement string collation semantic with directory names
     case st: StringType => st.supportsBinaryOrdering
+    // geospatial values cannot be cast to and from strings, which partition 
values are stored as
+    case _: GeometryType | _: GeographyType => false
     case a: AtomicType => !a.isInstanceOf[VariantType]
     case _ => false
   }
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/FileBasedDataSourceSuite.scala 
b/sql/core/src/test/scala/org/apache/spark/sql/FileBasedDataSourceSuite.scala
index 1251d78818b2..c8d6a0145540 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/FileBasedDataSourceSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/FileBasedDataSourceSuite.scala
@@ -1338,6 +1338,56 @@ class FileBasedDataSourceSuite extends SharedSparkSession
     }
   }
 
+  test("SPARK-60518: Geospatial types cannot be used as partition columns") {
+    // All of these file formats, including Parquet, do NOT support geospatial 
partition columns.
+    val dataSources = Seq("csv", "json", "orc", "parquet", "text", "xml")
+    // Test data: WKB representation of POINT(1 2).
+    val wkb = "0101000000000000000000F03F0000000000000040"
+    // Test GEOMETRY and GEOGRAPHY data types.
+    val geoTestCases = Seq(
+      (s"ST_GeomFromWKB(X'$wkb', 4326)", "GEOMETRY(4326)"),
+      (s"ST_GeogFromWKB(X'$wkb')", "GEOGRAPHY(4326)")
+    )
+    // Test both v1 and v2 data sources.
+    Seq(true, false).foreach { useV1 =>
+      val useV1List = if (useV1) {
+        dataSources.mkString(",")
+      } else {
+        ""
+      }
+      withSQLConf(SQLConf.USE_V1_SOURCE_LIST.key -> useV1List) {
+        dataSources.foreach { format =>
+          geoTestCases.foreach { case (expr, geoType) =>
+            def checkPartitionColumnError(f: => Unit): Unit = {
+              checkError(
+                exception = intercept[AnalysisException](f),
+                condition = "INVALID_PARTITION_COLUMN_DATA_TYPE",
+                parameters = Map("type" -> s""""$geoType""""))
+            }
+            // `TEXT` data source always has a single column whose name is 
`value`.
+            val query = s"SELECT 'a' AS value, $expr AS p"
+            withTable("t") {
+              checkPartitionColumnError {
+                sql(s"CREATE TABLE t (value STRING, p $geoType) USING $format 
PARTITIONED BY (p)")
+              }
+              checkPartitionColumnError {
+                sql(s"CREATE TABLE t USING $format PARTITIONED BY (p) AS 
$query")
+              }
+              checkPartitionColumnError {
+                
sql(query).write.format(format).partitionBy("p").saveAsTable("t")
+              }
+            }
+            withTempPath { dir =>
+              checkPartitionColumnError {
+                
sql(query).write.format(format).partitionBy("p").save(dir.getCanonicalPath)
+              }
+            }
+          }
+        }
+      }
+    }
+  }
+
   test("SPARK-57166: ORC supports nanosecond timestamp types in v1 and v2") {
     withSQLConf(SQLConf.TIMESTAMP_NANOS_TYPES_ENABLED.key -> "true") {
       // Validate both v1 and v2 ORC paths.


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to