This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.2 by this push:
     new bfd26e43c599 [SPARK-60518][SQL] Disallow `GEOMETRY` and `GEOGRAPHY` 
partition columns
bfd26e43c599 is described below

commit bfd26e43c599e8753979cac1a2e4e9ac7afaad18
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Sat Oct 24 11:07:46 2026 -0700

    [SPARK-60518][SQL] Disallow `GEOMETRY` and `GEOGRAPHY` partition columns
    
    ### What changes were proposed in this pull request?
    
    This PR aims to disallow the geospatial types, `GEOMETRY` and `GEOGRAPHY`, 
as partition column types in `PartitioningUtils.canPartitionOn`, like `VARIANT` 
(SPARK-45827) and non-default collated strings (SPARK-47015).
    
    After this PR, `CREATE TABLE`, CTAS, `saveAsTable`, and path-based writes 
with a geospatial partition column fail at analysis with 
`INVALID_PARTITION_COLUMN_DATA_TYPE` for all file data sources and Hive serde 
tables.
    
    ### Why are the changes needed?
    
    `canPartitionOn` accepts `GeometryType` and `GeographyType` because they 
are `AtomicType`s. However, Spark stores partition values as strings in 
directory names and in the catalog, and geospatial values cannot be cast to or 
from strings. So, geospatial partition values can be neither written nor read.
    
    - A write can succeed, e.g., when the partition value is a constant 
expression, but the directory name is the `Object.toString` result of the 
internal value, so the value is lost.
      ```scala
      scala> sql("SELECT 1 AS i, 
ST_GeomFromWKB(X'0101000000000000000000F03F0000000000000040', 4326) AS 
p").write.partitionBy("p").parquet("/tmp/geo")
      ```
      ```
      $ ls /tmp/geo
      _SUCCESS
      p=org.apache.spark.unsafe.types.BinaryView15dc1861
      ```
      Reading it back gives `p` as a `STRING` column with this text, or fails 
with `INVALID_PARTITION_VALUE` with the user-specified schema, `i INT, p 
GEOMETRY(4326)`.
    - Otherwise, e.g., `INSERT INTO` a table with a geospatial partition 
column, a write fails with an internal error because `V1Writes` adds a `Sort` 
on the non-orderable geospatial column.
      ```
      org.apache.spark.SparkException: [PLAN_VALIDATION_FAILED_RULE_IN_BATCH] 
Rule org.apache.spark.sql.execution.datasources.V1Writes in batch Early Filter 
and Projection Push-Down generated an invalid plan: The plan becomes 
unresolved: ...
      ```
    - `INSERT INTO` with a static partition spec, and reading a table created 
by CTAS or `saveAsTable`, fail with `INTERNAL_ERROR` (`scala.MatchError: 
StringType`) because there is no cast from `STRING` to the geospatial types.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. Geospatial partition columns are rejected at analysis instead of 
creating tables and directories which cannot be used.
    
    ```
    org.apache.spark.sql.AnalysisException: 
[INVALID_PARTITION_COLUMN_DATA_TYPE] Cannot use "GEOMETRY(4326)" for partition 
column. SQLSTATE: 0A000
    ```
    
    This includes Hive serde tables and data source tables in Hive metastore, 
which previously failed with the following error from Hive metastore.
    
    ```
    org.apache.spark.sql.AnalysisException: 
org.apache.hadoop.hive.ql.metadata.HiveException: 
InvalidObjectException(message:Invalid partition column type: geometry(4326))
    ```
    
    The geospatial types are enabled by default since Apache Spark 4.2.0 
([v4.2.0](https://github.com/apache/spark/releases/tag/v4.2.0), 2026-07-11).
    
    ### How was this patch tested?
    
    Pass the CIs with the newly added test case in `FileBasedDataSourceSuite`, 
which fails without this fix.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    Closes #59754 from dongjoon-hyun/SPARK-60518.
    
    Authored-by: Dongjoon Hyun <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
    (cherry picked from commit 5b562f0665096129126ed3a161ed1943de03154f)
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 .../execution/datasources/PartitioningUtils.scala  |  2 +
 .../spark/sql/FileBasedDataSourceSuite.scala       | 50 ++++++++++++++++++++++
 2 files changed, 52 insertions(+)

diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala
index 16ae22d611f7..6712b5e72aaf 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala
@@ -631,6 +631,8 @@ object PartitioningUtils extends SQLConfHelper {
     // non default collated strings should not be used as partition columns
     // as we cannot implement string collation semantic with directory names
     case st: StringType => st.supportsBinaryOrdering
+    // geospatial values cannot be cast to and from strings, which partition 
values are stored as
+    case _: GeometryType | _: GeographyType => false
     case a: AtomicType => !a.isInstanceOf[VariantType]
     case _ => false
   }
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/FileBasedDataSourceSuite.scala 
b/sql/core/src/test/scala/org/apache/spark/sql/FileBasedDataSourceSuite.scala
index 1fc45e9703f9..214cf2c1bb9f 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/FileBasedDataSourceSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/FileBasedDataSourceSuite.scala
@@ -1334,6 +1334,56 @@ class FileBasedDataSourceSuite extends SharedSparkSession
       }
     }
   }
+
+  test("SPARK-60518: Geospatial types cannot be used as partition columns") {
+    // All of these file formats, including Parquet, do NOT support geospatial 
partition columns.
+    val dataSources = Seq("csv", "json", "orc", "parquet", "text", "xml")
+    // Test data: WKB representation of POINT(1 2).
+    val wkb = "0101000000000000000000F03F0000000000000040"
+    // Test GEOMETRY and GEOGRAPHY data types.
+    val geoTestCases = Seq(
+      (s"ST_GeomFromWKB(X'$wkb', 4326)", "GEOMETRY(4326)"),
+      (s"ST_GeogFromWKB(X'$wkb')", "GEOGRAPHY(4326)")
+    )
+    // Test both v1 and v2 data sources.
+    Seq(true, false).foreach { useV1 =>
+      val useV1List = if (useV1) {
+        dataSources.mkString(",")
+      } else {
+        ""
+      }
+      withSQLConf(SQLConf.USE_V1_SOURCE_LIST.key -> useV1List) {
+        dataSources.foreach { format =>
+          geoTestCases.foreach { case (expr, geoType) =>
+            def checkPartitionColumnError(f: => Unit): Unit = {
+              checkError(
+                exception = intercept[AnalysisException](f),
+                condition = "INVALID_PARTITION_COLUMN_DATA_TYPE",
+                parameters = Map("type" -> s""""$geoType""""))
+            }
+            // `TEXT` data source always has a single column whose name is 
`value`.
+            val query = s"SELECT 'a' AS value, $expr AS p"
+            withTable("t") {
+              checkPartitionColumnError {
+                sql(s"CREATE TABLE t (value STRING, p $geoType) USING $format 
PARTITIONED BY (p)")
+              }
+              checkPartitionColumnError {
+                sql(s"CREATE TABLE t USING $format PARTITIONED BY (p) AS 
$query")
+              }
+              checkPartitionColumnError {
+                
sql(query).write.format(format).partitionBy("p").saveAsTable("t")
+              }
+            }
+            withTempPath { dir =>
+              checkPartitionColumnError {
+                
sql(query).write.format(format).partitionBy("p").save(dir.getCanonicalPath)
+              }
+            }
+          }
+        }
+      }
+    }
+  }
 }
 
 object TestingUDT {


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to