dongjoon-hyun opened a new pull request, #59711: URL: https://github.com/apache/spark/pull/59711
### What changes were proposed in this pull request? This PR aims to fix `CatalystTypeConverters` to read `GEOMETRY` and `GEOGRAPHY` fields at the given ordinal when it converts an `InternalRow` to an external `Row`. `GeometryConverter.toScalaImpl(row, column)` and `GeographyConverter.toScalaImpl(row, column)` read `row.getBinaryView(0)` instead of `row.getBinaryView(column)`. Since `StructConverter` calls `toScala(row, idx)` for every field, a `GEOMETRY` or `GEOGRAPHY` field at a non-zero ordinal of a row or a struct is read from the first field. The null check in `toScala(row, column)` already uses the given ordinal, so only non-null values are affected. ### Why are the changes needed? A `GEOMETRY` or `GEOGRAPHY` field at a non-zero ordinal silently gets the value of the first field if the first field has the same type. Otherwise, the first field is read as a geospatial value, which can fail, e.g., with `IndexOutOfBoundsException`. `CatalystTypeConverters.createToScalaConverter` with a `StructType` is used for - the result rows of the `spark-sql` CLI (`HiveResult.hiveResultString` via `SparkPlan.executeCollectPublic`) - `Row` inputs of Scala and Java UDFs - the inputs of `UserDefinedAggregateFunction` - the metrics of `Dataset.observe` `Dataset.collect` is not affected because it uses an encoder. For example, Apache Spark 4.2.0 shows `POINT(1 2)` for both columns of the following query, although the second one is `POINT(3 4)`. ``` spark-sql (default)> SELECT ST_GeomFromWKB(X'0101000000000000000000F03F0000000000000040') a, ST_GeomFromWKB(X'010100000000000000000008400000000000001040') b; POINT(1 2) POINT(1 2) ``` SPARK-54110 (#52813) introduced this bug as `row.getGeometry(0)` and `row.getGeography(0)` in Apache Spark 4.1.0 ([v4.1.0](https://github.com/apache/spark/releases/tag/v4.1.0), 2025-12-11), and SPARK-57058 (#56104) kept the ordinal `0` when it changed them to `row.getBinaryView(0)`. Apache Spark 4.1.x disables the geospatial types by default, so this bug affects users by default since Apache Spark 4.2.0 ([v4.2.0](https://github.com/apache/spark/releases/tag/v4.2.0), 2026-07-11). ### Does this PR introduce _any_ user-facing change? Yes. `GEOMETRY` and `GEOGRAPHY` fields at non-zero ordinals of rows and structs are converted correctly. **BEFORE (Apache Spark 4.2.0)** ``` spark-sql (default)> SELECT ST_GeomFromWKB(X'0101000000000000000000F03F0000000000000040') a, ST_GeomFromWKB(X'010100000000000000000008400000000000001040') b; POINT(1 2) POINT(1 2) spark-sql (default)> SELECT ST_GeogFromWKB(X'0101000000000000000000F03F0000000000000040') a, ST_GeogFromWKB(X'010100000000000000000008400000000000001040') b; SRID=4326;POINT(1 2) SRID=4326;POINT(1 2) spark-sql (default)> SELECT 1 i, ST_GeomFromWKB(X'010100000000000000000008400000000000001040') g; org.apache.spark.sql.execution.QueryExecutionException: java.lang.IndexOutOfBoundsException ``` **AFTER** ``` spark-sql (default)> SELECT ST_GeomFromWKB(X'0101000000000000000000F03F0000000000000040') a, ST_GeomFromWKB(X'010100000000000000000008400000000000001040') b; POINT(1 2) POINT(3 4) spark-sql (default)> SELECT ST_GeogFromWKB(X'0101000000000000000000F03F0000000000000040') a, ST_GeogFromWKB(X'010100000000000000000008400000000000001040') b; SRID=4326;POINT(1 2) SRID=4326;POINT(3 4) spark-sql (default)> SELECT 1 i, ST_GeomFromWKB(X'010100000000000000000008400000000000001040') g; 1 POINT(3 4) ``` ### How was this patch tested? Pass the CIs with the newly added test case in `CatalystTypeConvertersSuite`. It converts rows whose two `GEOMETRY` or `GEOGRAPHY` fields have different values. Without the fix, it fails. ``` [info] - SPARK-60470: createToScalaConverter for geospatial fields at non-zero ordinals *** FAILED *** (8 milliseconds) [info] [org.apache.spark.sql.types.Geometry@6f7338ee,org.apache.spark.sql.types.Geometry@6f7338ee] did not equal [org.apache.spark.sql.types.Geometry@6f7338ee,org.apache.spark.sql.types.Geometry@96d26af9] (CatalystTypeConvertersSuite.scala:822) ``` The related test suites also pass. ``` $ build/sbt "catalyst/testOnly *CatalystTypeConvertersSuite" $ build/sbt "sql/testOnly *HiveResultSuite *GeometryDataFrameSuite *GeographyDataFrameSuite *STExpressionsSuite *STFunctionsSuite *GeometryTypeSuite *GeographyTypeSuite" $ build/sbt "sql/testOnly org.apache.spark.sql.SQLQueryTestSuite -- -z st-functions.sql" ``` The `spark-sql` examples above were also checked manually with Apache Spark 4.2.0 and this PR. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
