huan233usc opened a new pull request, #17149:
URL: https://github.com/apache/iceberg/pull/17149
Adds end-to-end Spark SQL DML tests for `geometry` and `geography` columns,
and
fixes a vectorized-read bug those tests surfaced. This completes the Phase-1
geo end-to-end item (row-level DML with deletion vectors / nested geo /
nulls)
and builds on #17073 (Spark geo value path) and #16982 (Parquet geo WKB
values).
## Tests
`TestSparkGeospatial` previously covered only `INSERT` → `SELECT` read-back.
This
adds row-level DML on a geo table under **merge-on-read + format version 3**
(deletion vectors):
- `testDeleteGeospatialMergeOnRead` — `DELETE` leaves a survivor in a single
data
file, and the snapshot summary confirms a deletion vector was written
(`added-dvs = 1`); the surviving null-geo row round-trips.
- `testUpdateGeospatialMergeOnRead` — `UPDATE` swaps a row's geometry; others
unchanged.
- `testMergeGeospatialMergeOnRead` — `MERGE` updates a matched row and
inserts a
not-matched row, both carrying geo values.
- `testDeleteNestedGeometryMergeOnRead` — geometry nested in a `STRUCT`;
delete by
a nested non-geo field, surviving nested geometry (and a null nested
geometry)
still round-trip.
Geo values are asserted via `hex(st_asbinary(...))` round-trips and DML
filters on
non-geo columns, since Spark 4.1 has no spatial predicates.
## Vectorization fallback (production fix)
`geometry`/`geography` are primitive types, so they passed
`SparkBatch.supportsParquetBatchReads(NestedField)` and were routed to the
vectorized Parquet reader — but there is no Arrow geo vector yet, so
scanning a geo
column with vectorization enabled threw
`UnsupportedOperationException: Unsupported primitive type: geometry`. This
excludes
the two geo types from vectorized reads so they fall back to the row-based
reader.
With this fix, `testGeospatialWkbReadBack` now runs (no longer skipped) with
vectorization both **off and on**, proving the fallback. A dedicated Arrow
geo
vector remains future work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
