This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new 9cfbe3ea7bf7 [SPARK-60470][SQL] Fix `CatalystTypeConverters` to read 
`GEOMETRY` and `GEOGRAPHY` fields at the given ordinal
9cfbe3ea7bf7 is described below

commit 9cfbe3ea7bf730b4d920e761bb3c7ca4981bcb6b
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Thu Oct 22 13:48:32 2026 -0700

    [SPARK-60470][SQL] Fix `CatalystTypeConverters` to read `GEOMETRY` and 
`GEOGRAPHY` fields at the given ordinal
    
    ### What changes were proposed in this pull request?
    
    This PR aims to fix `CatalystTypeConverters` to read `GEOMETRY` and 
`GEOGRAPHY` fields at the given ordinal when it converts an `InternalRow` to an 
external `Row`.
    
    `GeometryConverter.toScalaImpl(row, column)` and 
`GeographyConverter.toScalaImpl(row, column)` read `row.getBinaryView(0)` 
instead of `row.getBinaryView(column)`. Since `StructConverter` calls 
`toScala(row, idx)` for every field, a `GEOMETRY` or `GEOGRAPHY` field at a 
non-zero ordinal of a row or a struct is read from the first field. The null 
check in `toScala(row, column)` already uses the given ordinal, so only 
non-null values are affected.
    
    ### Why are the changes needed?
    
    A `GEOMETRY` or `GEOGRAPHY` field at a non-zero ordinal silently gets the 
value of the first field if the first field has the same type. Otherwise, the 
first field is read as a geospatial value, which can fail, e.g., with 
`IndexOutOfBoundsException`.
    
    `CatalystTypeConverters.createToScalaConverter` with a `StructType` is used 
for
    - the result rows of the `spark-sql` CLI (`HiveResult.hiveResultString` via 
`SparkPlan.executeCollectPublic`)
    - `Row` inputs of Scala and Java UDFs
    - the inputs of `UserDefinedAggregateFunction`
    - the metrics of `Dataset.observe`
    
    `Dataset.collect` is not affected because it uses an encoder.
    
    For example, Apache Spark 4.2.0 shows `POINT(1 2)` for both columns of the 
following query, although the second one is `POINT(3 4)`.
    
    ```
    spark-sql (default)> SELECT 
ST_GeomFromWKB(X'0101000000000000000000F03F0000000000000040') a, 
ST_GeomFromWKB(X'010100000000000000000008400000000000001040') b;
    POINT(1 2)      POINT(1 2)
    ```
    
    SPARK-54110 (#52813) introduced this bug as `row.getGeometry(0)` and 
`row.getGeography(0)` in Apache Spark 4.1.0 
([v4.1.0](https://github.com/apache/spark/releases/tag/v4.1.0), 2025-12-11), 
and SPARK-57058 (#56104) kept the ordinal `0` when it changed them to 
`row.getBinaryView(0)`. Apache Spark 4.1.x disables the geospatial types by 
default, so this bug affects users by default since Apache Spark 4.2.0 
([v4.2.0](https://github.com/apache/spark/releases/tag/v4.2.0), 2026-07-11).
    - https://github.com/apache/spark/pull/52813
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. `GEOMETRY` and `GEOGRAPHY` fields at non-zero ordinals of rows and 
structs are converted correctly.
    
    **BEFORE (Apache Spark 4.2.0)**
    
    ```
    spark-sql (default)> SELECT 
ST_GeomFromWKB(X'0101000000000000000000F03F0000000000000040') a, 
ST_GeomFromWKB(X'010100000000000000000008400000000000001040') b;
    POINT(1 2)      POINT(1 2)
    spark-sql (default)> SELECT 
ST_GeogFromWKB(X'0101000000000000000000F03F0000000000000040') a, 
ST_GeogFromWKB(X'010100000000000000000008400000000000001040') b;
    SRID=4326;POINT(1 2)    SRID=4326;POINT(1 2)
    spark-sql (default)> SELECT 1 i, 
ST_GeomFromWKB(X'010100000000000000000008400000000000001040') g;
    org.apache.spark.sql.execution.QueryExecutionException: 
java.lang.IndexOutOfBoundsException
    ```
    
    **AFTER**
    
    ```
    spark-sql (default)> SELECT 
ST_GeomFromWKB(X'0101000000000000000000F03F0000000000000040') a, 
ST_GeomFromWKB(X'010100000000000000000008400000000000001040') b;
    POINT(1 2)      POINT(3 4)
    spark-sql (default)> SELECT 
ST_GeogFromWKB(X'0101000000000000000000F03F0000000000000040') a, 
ST_GeogFromWKB(X'010100000000000000000008400000000000001040') b;
    SRID=4326;POINT(1 2)    SRID=4326;POINT(3 4)
    spark-sql (default)> SELECT 1 i, 
ST_GeomFromWKB(X'010100000000000000000008400000000000001040') g;
    1       POINT(3 4)
    ```
    
    ### How was this patch tested?
    
    Pass the CIs with the newly added test case in 
`CatalystTypeConvertersSuite`. It converts rows whose two `GEOMETRY` or 
`GEOGRAPHY` fields have different values. Without the fix, it fails.
    
    ```
    [info] - SPARK-60470: createToScalaConverter for geospatial fields at 
non-zero ordinals *** FAILED *** (8 milliseconds)
    [info]   
[org.apache.spark.sql.types.Geometry6f7338ee,org.apache.spark.sql.types.Geometry6f7338ee]
 did not equal 
[org.apache.spark.sql.types.Geometry6f7338ee,org.apache.spark.sql.types.Geometry96d26af9]
 (CatalystTypeConvertersSuite.scala:822)
    ```
    
    The related test suites also pass.
    
    ```
    $ build/sbt "catalyst/testOnly *CatalystTypeConvertersSuite"
    $ build/sbt "sql/testOnly *HiveResultSuite *GeometryDataFrameSuite 
*GeographyDataFrameSuite *STExpressionsSuite *STFunctionsSuite 
*GeometryTypeSuite *GeographyTypeSuite"
    $ build/sbt "sql/testOnly org.apache.spark.sql.SQLQueryTestSuite -- -z 
st-functions.sql"
    ```
    
    The `spark-sql` examples above were also checked manually with Apache Spark 
4.2.0 and this PR.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    Closes #59711 from dongjoon-hyun/SPARK-60470.
    
    Authored-by: Dongjoon Hyun <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
    (cherry picked from commit 6f257a187bcafe8e4a0f7dacba09d25c76748bc8)
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 .../spark/sql/catalyst/CatalystTypeConverters.scala     |  4 ++--
 .../sql/catalyst/CatalystTypeConvertersSuite.scala      | 17 +++++++++++++++++
 2 files changed, 19 insertions(+), 2 deletions(-)

diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/CatalystTypeConverters.scala
 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/CatalystTypeConverters.scala
index cab367b4badf..1074f13c3473 100644
--- 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/CatalystTypeConverters.scala
+++ 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/CatalystTypeConverters.scala
@@ -391,7 +391,7 @@ object CatalystTypeConverters {
     override def toScalaImpl(row: InternalRow, column: Int):
         org.apache.spark.sql.types.Geometry = {
       assertGeospatialEnabled()
-      STUtils.deserializeGeom(row.getBinaryView(0), dataType)
+      STUtils.deserializeGeom(row.getBinaryView(column), dataType)
     }
   }
 
@@ -411,7 +411,7 @@ object CatalystTypeConverters {
     override def toScalaImpl(row: InternalRow, column: Int):
         org.apache.spark.sql.types.Geography = {
       assertGeospatialEnabled()
-      STUtils.deserializeGeog(row.getBinaryView(0), dataType)
+      STUtils.deserializeGeog(row.getBinaryView(column), dataType)
     }
   }
 
diff --git 
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/CatalystTypeConvertersSuite.scala
 
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/CatalystTypeConvertersSuite.scala
index 988a8d4510cb..4fc47e5a03da 100644
--- 
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/CatalystTypeConvertersSuite.scala
+++ 
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/CatalystTypeConvertersSuite.scala
@@ -806,6 +806,23 @@ class CatalystTypeConvertersSuite extends SparkFunSuite 
with SQLHelper {
     assert(result === geog)
   }
 
+  test("SPARK-60470: createToScalaConverter for geospatial fields at non-zero 
ordinals") {
+    // WKB bytes for POINT (3 4), distinct from `pointWkb`.
+    val otherPointWkb: Array[Byte] = 
"010100000000000000000008400000000000001040"
+      .grouped(2).map(Integer.parseInt(_, 16).toByte).toArray
+    Seq(
+      (GeometryType(0), Geometry.fromWKB(pointWkb, 0), 
Geometry.fromWKB(otherPointWkb, 0)),
+      (GeographyType(4326), Geography.fromWKB(pointWkb, 4326),
+        Geography.fromWKB(otherPointWkb, 4326))
+    ).foreach { case (dt, first, second) =>
+      val schema = new StructType().add("first", dt).add("second", dt)
+      val toCatalyst = CatalystTypeConverters.createToCatalystConverter(schema)
+      val toScala = CatalystTypeConverters.createToScalaConverter(schema)
+      val row = Row(first, second)
+      assert(toScala(toCatalyst(row)) === row)
+    }
+  }
+
   test("null handling for geospatial individual values") {
     
assert(CatalystTypeConverters.createToScalaConverter(GeometryType(0))(null) === 
null)
     
assert(CatalystTypeConverters.createToScalaConverter(GeographyType(4326))(null) 
=== null)


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to