This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-connect-swift.git


The following commit(s) were added to refs/heads/main by this push:
     new f62880e  [SPARK-58498] Support `geometry/geography` types in 
`DataType.simpleString`
f62880e is described below

commit f62880e0c2b9d0a6a1ea30a550235a321e6d0384
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Sat Aug 1 20:06:16 2026 -0700

    [SPARK-58498] Support `geometry/geography` types in `DataType.simpleString`
    
    ### What changes were proposed in this pull request?
    
    This PR aims to support `GEOMETRY` and `GEOGRAPHY` types (added in Apache 
Spark 4.1) in
    `DataType.simpleString` so that `DataFrame.dtypes` works on geospatial 
columns.
    
    Following Apache Spark's `GeometryType`/`GeographyType.typeName` 
convention, the mapping uses
    `geometry(<srid>)`/`geography(<srid>)` for fixed-SRID types and 
`geometry(any)`/`geography(any)`
    for the mixed-SRID case (`srid == -1`).
    
    ### Why are the changes needed?
    
    `Spark_Connect_DataType`'s `geometry` and `geography` kinds fell through to 
the `default:` branch
    of `DataType.simpleString` and threw `SparkConnectError.InvalidType`. As a 
result,
    `DataFrame.dtypes` failed entirely on any DataFrame containing a geospatial 
column.
    
    ```swift
    let df = try await spark.sql("SELECT 
st_geomfromwkb(X'0101000000000000000000F03F0000000000000040')")
    try await df.dtypes  // Before: throws SparkConnectError.InvalidType
                         // After: [("st_geomfromwkb(...)", "geometry(0)")]
    ```
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes, `DataFrame.dtypes` no longer throws on geospatial columns and returns 
Spark-style type strings instead. This is a bug fix from the unreleased 
perspective.
    
    ### How was this patch tested?
    
    Pass the CIs with a newly added test case.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Fable 5
    
    Closes #469 from dongjoon-hyun/SPARK-58498.
    
    Authored-by: Dongjoon Hyun <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 Sources/SparkConnect/Extension.swift         | 12 ++++++++++++
 Tests/SparkConnectTests/DataFrameTests.swift | 19 +++++++++++++++++++
 2 files changed, 31 insertions(+)

diff --git a/Sources/SparkConnect/Extension.swift 
b/Sources/SparkConnect/Extension.swift
index 5684783..ab88213 100644
--- a/Sources/SparkConnect/Extension.swift
+++ b/Sources/SparkConnect/Extension.swift
@@ -343,6 +343,18 @@ extension DataType {
         try self.map.toString()
       case .variant:
         "variant"
+      case .geometry:
+        if self.geometry.srid == -1 {
+          "geometry(any)"
+        } else {
+          "geometry(\(self.geometry.srid))"
+        }
+      case .geography:
+        if self.geography.srid == -1 {
+          "geography(any)"
+        } else {
+          "geography(\(self.geography.srid))"
+        }
       case .udt:
         self.udt.type
       case .unparsed:
diff --git a/Tests/SparkConnectTests/DataFrameTests.swift 
b/Tests/SparkConnectTests/DataFrameTests.swift
index a23b4b4..34a5108 100644
--- a/Tests/SparkConnectTests/DataFrameTests.swift
+++ b/Tests/SparkConnectTests/DataFrameTests.swift
@@ -135,6 +135,25 @@ struct DataFrameTests {
     await spark.stop()
   }
 
+  @Test
+  func dtypesGeospatial() async throws {
+    let spark = try await SparkSession.builder.getOrCreate()
+    if await spark.version >= "4.2" {
+      let expected = [
+        ("st_geomfromwkb(X'0101000000000000000000F03F0000000000000040')", 
"geometry(0)"),
+        (
+          
"st_setsrid(st_geomfromwkb(X'0101000000000000000000F03F0000000000000040'), 
4326)",
+          "geometry(4326)"
+        ),
+        ("st_geogfromwkb(X'0101000000000000000000F03F0000000000000040')", 
"geography(4326)"),
+      ]
+      for pair in expected {
+        #expect(try await spark.sql("SELECT \(pair.0)").dtypes[0].1 == pair.1)
+      }
+    }
+    await spark.stop()
+  }
+
   @Test
   func array() async throws {
     let spark = try await SparkSession.builder.getOrCreate()


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to