[
https://issues.apache.org/jira/browse/SPARK-61895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61895:
-----------------------------------
Labels: pull-request-available (was: )
> Load the SRS registry PySpark ships in GeometryType and GeographyType
> ---------------------------------------------------------------------
>
> Key: SPARK-61895
> URL: https://issues.apache.org/jira/browse/SPARK-61895
> Project: Spark
> Issue Type: Improvement
> Components: Connect
> Affects Versions: 4.2.0, 4.3.0, 5.0.0, 4.2.1, 4.4.0, 4.2.2, 4.3.1, 4.5.0
> Reporter: James Willis
> Priority: Major
> Labels: pull-request-available
>
> SPARK-55790 committed the PROJ-based SRS registry to both {{sql/api}}
> resources and {{python/pyspark/sql/srs_registry.csv}}, but only the JVM reads
> it. PySpark's {{SpatialReferenceSystemCache}} ({{geo_utils.py}}) still
> hard-codes {{SRID:0}}, {{EPSG:3857}} and {{OGC:CRS84}}, so
> {{GeometryType(32610)}} raises {{ST_INVALID_SRID_VALUE}} while the server
> accepts the SRID. For a Spark Connect client a result schema with
> {{geometry(32610)}} therefore fails to decode; every released
> {{pyspark-client}} is affected.
> {{python/pyspark/sql/srs_registry.csv}} already exists, written by the same
> generator as the JVM copy. Load it in the Python cache with
> {{importlib.resources}} (as {{pyspark.errors}} loads its JSON), parsing rows
> like the Java class and adding {{SRID:0}} and the OGC overrides on top; add
> {{**/*.csv}} to {{package_data}} so the classic and client wheels ship it;
> mirror {{SpatialReferenceSystemMapperSuite}} in Python.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]