[ 
https://issues.apache.org/jira/browse/SPARK-61895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Willis updated SPARK-61895:
---------------------------------
    Description: 
SPARK-55790 built the complete spatial reference system (SRS) registry from 
PROJ and committed it twice: to 
{{sql/api/src/main/resources/org/apache/spark/sql/srs_registry.csv}}, which the 
JVM's {{SpatialReferenceSystemCache}} loads, and to 
{{python/pyspark/sql/srs_registry.csv}}, which nothing reads. PySpark's 
{{SpatialReferenceSystemCache}} in {{python/pyspark/sql/geo_utils.py}} still 
carries the Spark 4.1 table of three entries ({{SRID:0}}, {{EPSG:3857}}, 
{{OGC:CRS84}}), so the two sides disagree on which SRIDs are valid:

{code}
>>> from pyspark.sql.types import GeometryType
>>> GeometryType(32610)
pyspark.errors.exceptions.base.IllegalArgumentException: 
[ST_INVALID_SRID_VALUE] Invalid or unsupported SRID (spatial reference 
identifier) value: 32610.
{code}

while {{SELECT ST_SetSRID(..., 32610)}} is accepted by the server since the 
registry landed.

For a Spark Connect client the mismatch is fatal before any data moves: a 
result schema carrying {{geometry(32610)}} is decoded into 
{{GeometryType(32610)}} on the client and the query fails with the error above, 
even though the server produced the type. Every released {{pyspark-client}} 
(4.1.x, 4.2.1, 4.3.0) behaves this way, so a server cannot return a projected 
geometry or geography column to a Python client unless it drops the CRS and 
sends {{GEOMETRY(ANY)}}.

Proposed change: have the Python cache load the shipped CSV the way the Java 
class does (comment and header lines skipped; {{srid,string_id,is_geographic}} 
rows), then add the Spark-specific entries on top ({{SRID:0}}, and the OGC 
overrides {{OGC:CRS84}} / {{OGC:CRS27}} / {{OGC:CRS83}} for 4326 / 4267 / 4269, 
with the EPSG spellings still resolving); ship the CSV in the classic and 
client wheels via {{package_data}}; add a Python counterpart of 
{{SpatialReferenceSystemMapperSuite}} and extend the {{GeometryType}} / 
{{GeographyType}} tests with registry SRIDs and CRS string IDs. 
{{GeometryType}} and {{GeographyType}} need no change: they already consult the 
mappers.


> Load the SRS registry PySpark ships in GeometryType and GeographyType
> ---------------------------------------------------------------------
>
>                 Key: SPARK-61895
>                 URL: https://issues.apache.org/jira/browse/SPARK-61895
>             Project: Spark
>          Issue Type: Improvement
>          Components: Connect
>    Affects Versions: 4.2.0, 4.3.0, 5.0.0, 4.2.1, 4.4.0, 4.2.2, 4.3.1, 4.5.0
>            Reporter: James Willis
>            Priority: Major
>
> SPARK-55790 built the complete spatial reference system (SRS) registry from 
> PROJ and committed it twice: to 
> {{sql/api/src/main/resources/org/apache/spark/sql/srs_registry.csv}}, which 
> the JVM's {{SpatialReferenceSystemCache}} loads, and to 
> {{python/pyspark/sql/srs_registry.csv}}, which nothing reads. PySpark's 
> {{SpatialReferenceSystemCache}} in {{python/pyspark/sql/geo_utils.py}} still 
> carries the Spark 4.1 table of three entries ({{SRID:0}}, {{EPSG:3857}}, 
> {{OGC:CRS84}}), so the two sides disagree on which SRIDs are valid:
> {code}
> >>> from pyspark.sql.types import GeometryType
> >>> GeometryType(32610)
> pyspark.errors.exceptions.base.IllegalArgumentException: 
> [ST_INVALID_SRID_VALUE] Invalid or unsupported SRID (spatial reference 
> identifier) value: 32610.
> {code}
> while {{SELECT ST_SetSRID(..., 32610)}} is accepted by the server since the 
> registry landed.
> For a Spark Connect client the mismatch is fatal before any data moves: a 
> result schema carrying {{geometry(32610)}} is decoded into 
> {{GeometryType(32610)}} on the client and the query fails with the error 
> above, even though the server produced the type. Every released 
> {{pyspark-client}} (4.1.x, 4.2.1, 4.3.0) behaves this way, so a server cannot 
> return a projected geometry or geography column to a Python client unless it 
> drops the CRS and sends {{GEOMETRY(ANY)}}.
> Proposed change: have the Python cache load the shipped CSV the way the Java 
> class does (comment and header lines skipped; 
> {{srid,string_id,is_geographic}} rows), then add the Spark-specific entries 
> on top ({{SRID:0}}, and the OGC overrides {{OGC:CRS84}} / {{OGC:CRS27}} / 
> {{OGC:CRS83}} for 4326 / 4267 / 4269, with the EPSG spellings still 
> resolving); ship the CSV in the classic and client wheels via 
> {{package_data}}; add a Python counterpart of 
> {{SpatialReferenceSystemMapperSuite}} and extend the {{GeometryType}} / 
> {{GeographyType}} tests with registry SRIDs and CRS string IDs. 
> {{GeometryType}} and {{GeographyType}} need no change: they already consult 
> the mappers.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to