jiayuasu opened a new pull request, #3274: URL: https://github.com/apache/sedona/pull/3274
## What changes This is the native foundation for #3273. - Add a focused JTS wrapper that validates one target geometry against an explicit array of adjacent coverage members. - Add a hidden Catalyst expression for the per-target validator. - Add an operation-scoped row identifier that retains complete source lineage across independently optimized Spark plan branches. - Register only `__sedona_internal_*` aliases and enforce registry-wide name uniqueness. This layer adds no public `ST_*`, Python, or GeoPandas API and requires no public documentation. A stacked follow-up will add distributed neighbor discovery and the GeoPandas methods. ## Why Coverage validation is collection-wide, so the public distributed implementation cannot collect every polygon into one task. The native validator lets the follow-up spatially join candidate neighbors, group them per target, and validate each target on executors. The operation row identifier is needed because duplicate indexes cannot identify physical rows, while ordinary partition-based IDs can change when Catalyst prunes a window-derived index from one branch but retains it in another. Its opaque dependency keeps geometry, index, and ordering lineage below the identifier without evaluating or hashing complex index values. ## Validation - `CoverageValidationTest`: 6 tests passed. - Spark 3.4 / Scala 2.12: 9 focused tests passed. - Spark 4.1 / Scala 2.13: 9 focused tests passed. - Tests cover valid edges, overlaps, narrow gaps, null handling, invalid widths, hidden registration, catalog uniqueness, Window-derived indexes, and MapType dependencies. - Repository commit hooks, Spotless, and `git diff --check` passed. ## Performance The lineage-safe identifier adds approximately 6–8 ns per row in a synthetic 10-million-row scan. On a fair 500,000-row window-derived source, its runtime was indistinguishable from the ordinary identifier. It adds no action, cache, checkpoint, sort, or shuffle. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
