dongjoon-hyun opened a new pull request, #59754:
URL: https://github.com/apache/spark/pull/59754
### What changes were proposed in this pull request?
This PR aims to disallow the geospatial types, `GEOMETRY` and `GEOGRAPHY`,
as partition column types in `PartitioningUtils.canPartitionOn`, like `VARIANT`
(SPARK-45827) and non-default collated strings (SPARK-47015).
After this PR, `CREATE TABLE`, CTAS, `saveAsTable`, and path-based writes
with a geospatial partition column fail at analysis with
`INVALID_PARTITION_COLUMN_DATA_TYPE` for all file data sources and Hive serde
tables.
### Why are the changes needed?
`canPartitionOn` accepts `GeometryType` and `GeographyType` because they are
`AtomicType`s. However, Spark stores partition values as strings in directory
names and in the catalog, and geospatial values cannot be cast to or from
strings. So, geospatial partition values can be neither written nor read.
- A write can succeed, e.g., when the partition value is a constant
expression, but the directory name is the `Object.toString` result of the
internal value, so the value is lost.
```scala
scala> sql("SELECT 1 AS i,
ST_GeomFromWKB(X'0101000000000000000000F03F0000000000000040', 4326) AS
p").write.partitionBy("p").parquet("/tmp/geo")
```
```
$ ls /tmp/geo
_SUCCESS
p=org.apache.spark.unsafe.types.BinaryView@15dc1861
```
Reading it back gives `p` as a `STRING` column with this text, or fails
with `INVALID_PARTITION_VALUE` with the user-specified schema, `i INT, p
GEOMETRY(4326)`.
- Otherwise, e.g., `INSERT INTO` a table with a geospatial partition column,
a write fails with an internal error because `V1Writes` adds a `Sort` on the
non-orderable geospatial column.
```
org.apache.spark.SparkException: [PLAN_VALIDATION_FAILED_RULE_IN_BATCH]
Rule org.apache.spark.sql.execution.datasources.V1Writes in batch Early Filter
and Projection Push-Down generated an invalid plan: The plan becomes
unresolved: ...
```
- `INSERT INTO` with a static partition spec, and reading a table created by
CTAS or `saveAsTable`, fail with `INTERNAL_ERROR` (`scala.MatchError:
StringType`) because there is no cast from `STRING` to the geospatial types.
### Does this PR introduce _any_ user-facing change?
Yes. Geospatial partition columns are rejected at analysis instead of
creating tables and directories which cannot be used.
```
org.apache.spark.sql.AnalysisException: [INVALID_PARTITION_COLUMN_DATA_TYPE]
Cannot use "GEOMETRY(4326)" for partition column. SQLSTATE: 0A000
```
This includes Hive serde tables and data source tables in Hive metastore,
which previously failed with the following error from Hive metastore.
```
org.apache.spark.sql.AnalysisException:
org.apache.hadoop.hive.ql.metadata.HiveException:
InvalidObjectException(message:Invalid partition column type: geometry(4326))
```
The geospatial types are enabled by default since Apache Spark 4.2.0
([v4.2.0](https://github.com/apache/spark/releases/tag/v4.2.0), 2026-07-11).
### How was this patch tested?
Pass the CIs with the newly added test case in `FileBasedDataSourceSuite`,
which fails without this fix.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
