This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-connect-swift.git
The following commit(s) were added to refs/heads/main by this push:
new f62880e [SPARK-58498] Support `geometry/geography` types in
`DataType.simpleString`
f62880e is described below
commit f62880e0c2b9d0a6a1ea30a550235a321e6d0384
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Sat Aug 1 20:06:16 2026 -0700
[SPARK-58498] Support `geometry/geography` types in `DataType.simpleString`
### What changes were proposed in this pull request?
This PR aims to support `GEOMETRY` and `GEOGRAPHY` types (added in Apache
Spark 4.1) in
`DataType.simpleString` so that `DataFrame.dtypes` works on geospatial
columns.
Following Apache Spark's `GeometryType`/`GeographyType.typeName`
convention, the mapping uses
`geometry(<srid>)`/`geography(<srid>)` for fixed-SRID types and
`geometry(any)`/`geography(any)`
for the mixed-SRID case (`srid == -1`).
### Why are the changes needed?
`Spark_Connect_DataType`'s `geometry` and `geography` kinds fell through to
the `default:` branch
of `DataType.simpleString` and threw `SparkConnectError.InvalidType`. As a
result,
`DataFrame.dtypes` failed entirely on any DataFrame containing a geospatial
column.
```swift
let df = try await spark.sql("SELECT
st_geomfromwkb(X'0101000000000000000000F03F0000000000000040')")
try await df.dtypes // Before: throws SparkConnectError.InvalidType
// After: [("st_geomfromwkb(...)", "geometry(0)")]
```
### Does this PR introduce _any_ user-facing change?
Yes, `DataFrame.dtypes` no longer throws on geospatial columns and returns
Spark-style type strings instead. This is a bug fix from the unreleased
perspective.
### How was this patch tested?
Pass the CIs with a newly added test case.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Fable 5
Closes #469 from dongjoon-hyun/SPARK-58498.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
Sources/SparkConnect/Extension.swift | 12 ++++++++++++
Tests/SparkConnectTests/DataFrameTests.swift | 19 +++++++++++++++++++
2 files changed, 31 insertions(+)
diff --git a/Sources/SparkConnect/Extension.swift
b/Sources/SparkConnect/Extension.swift
index 5684783..ab88213 100644
--- a/Sources/SparkConnect/Extension.swift
+++ b/Sources/SparkConnect/Extension.swift
@@ -343,6 +343,18 @@ extension DataType {
try self.map.toString()
case .variant:
"variant"
+ case .geometry:
+ if self.geometry.srid == -1 {
+ "geometry(any)"
+ } else {
+ "geometry(\(self.geometry.srid))"
+ }
+ case .geography:
+ if self.geography.srid == -1 {
+ "geography(any)"
+ } else {
+ "geography(\(self.geography.srid))"
+ }
case .udt:
self.udt.type
case .unparsed:
diff --git a/Tests/SparkConnectTests/DataFrameTests.swift
b/Tests/SparkConnectTests/DataFrameTests.swift
index a23b4b4..34a5108 100644
--- a/Tests/SparkConnectTests/DataFrameTests.swift
+++ b/Tests/SparkConnectTests/DataFrameTests.swift
@@ -135,6 +135,25 @@ struct DataFrameTests {
await spark.stop()
}
+ @Test
+ func dtypesGeospatial() async throws {
+ let spark = try await SparkSession.builder.getOrCreate()
+ if await spark.version >= "4.2" {
+ let expected = [
+ ("st_geomfromwkb(X'0101000000000000000000F03F0000000000000040')",
"geometry(0)"),
+ (
+
"st_setsrid(st_geomfromwkb(X'0101000000000000000000F03F0000000000000040'),
4326)",
+ "geometry(4326)"
+ ),
+ ("st_geogfromwkb(X'0101000000000000000000F03F0000000000000040')",
"geography(4326)"),
+ ]
+ for pair in expected {
+ #expect(try await spark.sql("SELECT \(pair.0)").dtypes[0].1 == pair.1)
+ }
+ }
+ await spark.stop()
+ }
+
@Test
func array() async throws {
let spark = try await SparkSession.builder.getOrCreate()
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]