jiayuasu commented on issue #3153: URL: https://github.com/apache/sedona/issues/3153#issuecomment-5127618647
@james-willis After exploring the implementation in #3194, I think the alignment direction should be reversed: Sedona Spark should retain its established behavior, and SedonaDB should align with it. Sedona Spark currently applies a simple rule when raster CRS or geometry SRID metadata is missing: - A missing raster CRS is interpreted as WGS84. - Geometry SRID 0 is interpreted as WGS84. - Operands with known, different CRSs are transformed to a common CRS before comparison. This rule is imperfect because missing metadata may describe data that is not actually WGS84. However, it gives every operand an independent coordinate interpretation. That property is particularly important for optimized spatial joins. Raster CRS is stored per value in Spark, so one raster column may contain rows using EPSG:4326, EPSG:3857, custom non-EPSG CRSs, and no CRS. A strict “exactly one operand has a CRS must error” rule cannot generally be validated once at planning time. If validation occurs only after envelope filtering, the error becomes data-dependent because the index may discard the incompatible pair before the scalar predicate evaluates it. Guaranteeing the error would require a preflight scan and classification of both inputs. We also considered treating an SRID-0 geometry as native raster coordinates when the raster has a custom non-EPSG CRS. This keeps custom rasters usable, but introduces pair-dependent semantics: the same geometry row can be interpreted in a different CRS for every raster row. It also prevents the geometry from having one independently computable envelope for the spatial index. Supporting this correctly requires conservative filtering, multiple join paths, or significant loss of pruning. The existing WGS84 default avoids these problems: - Every operand can be normalized independently. - Mixed per-row raster CRSs are supported naturally. - Scalar predicates, range joins, and broadcast-index joins use the same interpretation. - No preflight validation job is required. - Join optimization does not change whether a query returns a result or raises a CRS error. The strict one-sided-CRS behavior and optimized raster–vector join are relatively recent in SedonaDB, including the work in apache/sedona-db#1073. Sedona Spark’s behavior is older and has a much larger compatibility surface across existing raster functions and applications. Changing Spark would therefore introduce substantially more risk than aligning the newer SedonaDB behavior with Spark. I propose that both engines use the established Sedona Spark rule: 1. Missing CRS or SRID metadata is interpreted as WGS84. 2. Known CRSs are respected and transformed when necessary. 3. Scalar predicates and optimized joins must use the same interpretation. 4. Documentation should strongly recommend assigning explicit CRS/SRID metadata whenever it is known. This intentionally favors backward compatibility and consistent query execution over strict detection of missing metadata. A strict mode could be reconsidered later as an opt-in or coordinated breaking change, ideally with schema-level CRS information and a migration path. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
