HyukjinKwon opened a new pull request, #258: URL: https://github.com/apache/spark-connect-rust/pull/258
### What changes were proposed in this pull request? The inference of a `Geometry` or `Geography` value in `createDataFrame` validates its SRID with the validator that the type constructors already use (`spatial_type_srid`, now `pub(crate)`), as PySpark's `_infer_type` does (`crates/pyspark-rs/src/infer.rs`): Geometry takes 0, 3857 and 4326, Geography only 4326, and any other SRID raises `IllegalArgumentException` `ST_INVALID_SRID_VALUE` (`srid`) on the client, at the row of the value, top-level or nested. Explicit schemas are unchanged: PySpark leaves their SRIDs to the server, and ours already matched it there. ### Why are the changes needed? [SPARK-61068](https://issues.apache.org/jira/browse/SPARK-61068): the raw SRID reached the inferred type. Against a local Spark 4.3.0 server: | Input | Current master | This change and PySpark 4.3.0 | | --- | --- | --- | | `[(Geography.fromWKB(wkb, 4269),)]` (also Geometry 4269, 99999) | server `ParseException [PARSE_SYNTAX_ERROR]` | `IllegalArgumentException [ST_INVALID_SRID_VALUE]` (4269) | | Geography 4326 and 4269 in two rows | silently `geography(any)` | `ST_INVALID_SRID_VALUE` (4269) | | Geography 4326 and 0 | the server's error, after sending | the client's `ST_INVALID_SRID_VALUE` (0) | | Geometry 0, 3857 and 4326 | `geometry(any)` | `geometry(any)` (unchanged) | ### Does this PR introduce _any_ user-facing change? Yes, invalid SRIDs raise PySpark's error. ### How was this patch tested? - New offline tests in `python/tests/test_dropin_offline.py`: each invalid SRID of each type, top-level, in an array and after a null row; mixed Geography SRIDs; the inferred types of valid SRIDs, from the local relation's schema; and an explicit `GeographyType("ANY")` still taking any SRID. - The offline suite (967 passed, 8 skipped) and the live spatial test; a probe of these cases against both clients gives the same output. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
