[ 
https://issues.apache.org/jira/browse/SPARK-61895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Willis updated SPARK-61895:
---------------------------------
    Description: 
SPARK-55790 committed the PROJ-based SRS registry to both {{sql/api}} resources 
and {{python/pyspark/sql/srs_registry.csv}}, but only the JVM reads it. 
PySpark's {{SpatialReferenceSystemCache}} ({{geo_utils.py}}) still hard-codes 
{{SRID:0}}, {{EPSG:3857}} and {{OGC:CRS84}}, so {{GeometryType(32610)}} raises 
{{ST_INVALID_SRID_VALUE}} while the server accepts the SRID. For a Spark 
Connect client a result schema with {{geometry(32610)}} therefore fails to 
decode; every released {{pyspark-client}} is affected.

Load the shipped CSV in the Python cache the way the Java class does (plus 
{{SRID:0}} and the OGC overrides), ship the CSV in the classic and client 
wheels, and mirror {{SpatialReferenceSystemMapperSuite}} in Python.

  was:
SPARK-55790 built the complete spatial reference system (SRS) registry from 
PROJ and committed it twice: to 
{{sql/api/src/main/resources/org/apache/spark/sql/srs_registry.csv}}, which the 
JVM's {{SpatialReferenceSystemCache}} loads, and to 
{{python/pyspark/sql/srs_registry.csv}}, which nothing reads. PySpark's 
{{SpatialReferenceSystemCache}} in {{python/pyspark/sql/geo_utils.py}} still 
carries the Spark 4.1 table of three entries ({{SRID:0}}, {{EPSG:3857}}, 
{{OGC:CRS84}}), so the two sides disagree on which SRIDs are valid:

{code}
>>> from pyspark.sql.types import GeometryType
>>> GeometryType(32610)
pyspark.errors.exceptions.base.IllegalArgumentException: 
[ST_INVALID_SRID_VALUE] Invalid or unsupported SRID (spatial reference 
identifier) value: 32610.
{code}

while {{SELECT ST_SetSRID(..., 32610)}} is accepted by the server since the 
registry landed.

For a Spark Connect client the mismatch is fatal before any data moves: a 
result schema carrying {{geometry(32610)}} is decoded into 
{{GeometryType(32610)}} on the client and the query fails with the error above, 
even though the server produced the type. Every released {{pyspark-client}} 
(4.1.x, 4.2.1, 4.3.0) behaves this way, so a server cannot return a projected 
geometry or geography column to a Python client unless it drops the CRS and 
sends {{GEOMETRY(ANY)}}.

Proposed change: have the Python cache load the shipped CSV the way the Java 
class does (comment and header lines skipped; {{srid,string_id,is_geographic}} 
rows), then add the Spark-specific entries on top ({{SRID:0}}, and the OGC 
overrides {{OGC:CRS84}} / {{OGC:CRS27}} / {{OGC:CRS83}} for 4326 / 4267 / 4269, 
with the EPSG spellings still resolving); ship the CSV in the classic and 
client wheels via {{package_data}}; add a Python counterpart of 
{{SpatialReferenceSystemMapperSuite}} and extend the {{GeometryType}} / 
{{GeographyType}} tests with registry SRIDs and CRS string IDs. 
{{GeometryType}} and {{GeographyType}} need no change: they already consult the 
mappers.



> Load the SRS registry PySpark ships in GeometryType and GeographyType
> ---------------------------------------------------------------------
>
>                 Key: SPARK-61895
>                 URL: https://issues.apache.org/jira/browse/SPARK-61895
>             Project: Spark
>          Issue Type: Improvement
>          Components: Connect
>    Affects Versions: 4.2.0, 4.3.0, 5.0.0, 4.2.1, 4.4.0, 4.2.2, 4.3.1, 4.5.0
>            Reporter: James Willis
>            Priority: Major
>
> SPARK-55790 committed the PROJ-based SRS registry to both {{sql/api}} 
> resources and {{python/pyspark/sql/srs_registry.csv}}, but only the JVM reads 
> it. PySpark's {{SpatialReferenceSystemCache}} ({{geo_utils.py}}) still 
> hard-codes {{SRID:0}}, {{EPSG:3857}} and {{OGC:CRS84}}, so 
> {{GeometryType(32610)}} raises {{ST_INVALID_SRID_VALUE}} while the server 
> accepts the SRID. For a Spark Connect client a result schema with 
> {{geometry(32610)}} therefore fails to decode; every released 
> {{pyspark-client}} is affected.
> Load the shipped CSV in the Python cache the way the Java class does (plus 
> {{SRID:0}} and the OGC overrides), ship the CSV in the classic and client 
> wheels, and mirror {{SpatialReferenceSystemMapperSuite}} in Python.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to