[
https://issues.apache.org/jira/browse/SPARK-61895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Willis updated SPARK-61895:
---------------------------------
Description:
SPARK-55790 built the complete spatial reference system (SRS) registry from
PROJ and committed it twice: to
{{sql/api/src/main/resources/org/apache/spark/sql/srs_registry.csv}}, which the
JVM's {{SpatialReferenceSystemCache}} loads, and to
{{python/pyspark/sql/srs_registry.csv}}, which nothing reads. PySpark's
{{SpatialReferenceSystemCache}} in {{python/pyspark/sql/geo_utils.py}} still
carries the Spark 4.1 table of three entries ({{SRID:0}}, {{EPSG:3857}},
{{OGC:CRS84}}), so the two sides disagree on which SRIDs are valid:
{code}
>>> from pyspark.sql.types import GeometryType
>>> GeometryType(32610)
pyspark.errors.exceptions.base.IllegalArgumentException:
[ST_INVALID_SRID_VALUE] Invalid or unsupported SRID (spatial reference
identifier) value: 32610.
{code}
while {{SELECT ST_SetSRID(..., 32610)}} is accepted by the server since the
registry landed.
For a Spark Connect client the mismatch is fatal before any data moves: a
result schema carrying {{geometry(32610)}} is decoded into
{{GeometryType(32610)}} on the client and the query fails with the error above,
even though the server produced the type. Every released {{pyspark-client}}
(4.1.x, 4.2.1, 4.3.0) behaves this way, so a server cannot return a projected
geometry or geography column to a Python client unless it drops the CRS and
sends {{GEOMETRY(ANY)}}.
Proposed change: have the Python cache load the shipped CSV the way the Java
class does (comment and header lines skipped; {{srid,string_id,is_geographic}}
rows), then add the Spark-specific entries on top ({{SRID:0}}, and the OGC
overrides {{OGC:CRS84}} / {{OGC:CRS27}} / {{OGC:CRS83}} for 4326 / 4267 / 4269,
with the EPSG spellings still resolving); ship the CSV in the classic and
client wheels via {{package_data}}; add a Python counterpart of
{{SpatialReferenceSystemMapperSuite}} and extend the {{GeometryType}} /
{{GeographyType}} tests with registry SRIDs and CRS string IDs.
{{GeometryType}} and {{GeographyType}} need no change: they already consult the
mappers.
> Load the SRS registry PySpark ships in GeometryType and GeographyType
> ---------------------------------------------------------------------
>
> Key: SPARK-61895
> URL: https://issues.apache.org/jira/browse/SPARK-61895
> Project: Spark
> Issue Type: Improvement
> Components: Connect
> Affects Versions: 4.2.0, 4.3.0, 5.0.0, 4.2.1, 4.4.0, 4.2.2, 4.3.1, 4.5.0
> Reporter: James Willis
> Priority: Major
>
> SPARK-55790 built the complete spatial reference system (SRS) registry from
> PROJ and committed it twice: to
> {{sql/api/src/main/resources/org/apache/spark/sql/srs_registry.csv}}, which
> the JVM's {{SpatialReferenceSystemCache}} loads, and to
> {{python/pyspark/sql/srs_registry.csv}}, which nothing reads. PySpark's
> {{SpatialReferenceSystemCache}} in {{python/pyspark/sql/geo_utils.py}} still
> carries the Spark 4.1 table of three entries ({{SRID:0}}, {{EPSG:3857}},
> {{OGC:CRS84}}), so the two sides disagree on which SRIDs are valid:
> {code}
> >>> from pyspark.sql.types import GeometryType
> >>> GeometryType(32610)
> pyspark.errors.exceptions.base.IllegalArgumentException:
> [ST_INVALID_SRID_VALUE] Invalid or unsupported SRID (spatial reference
> identifier) value: 32610.
> {code}
> while {{SELECT ST_SetSRID(..., 32610)}} is accepted by the server since the
> registry landed.
> For a Spark Connect client the mismatch is fatal before any data moves: a
> result schema carrying {{geometry(32610)}} is decoded into
> {{GeometryType(32610)}} on the client and the query fails with the error
> above, even though the server produced the type. Every released
> {{pyspark-client}} (4.1.x, 4.2.1, 4.3.0) behaves this way, so a server cannot
> return a projected geometry or geography column to a Python client unless it
> drops the CRS and sends {{GEOMETRY(ANY)}}.
> Proposed change: have the Python cache load the shipped CSV the way the Java
> class does (comment and header lines skipped;
> {{srid,string_id,is_geographic}} rows), then add the Spark-specific entries
> on top ({{SRID:0}}, and the OGC overrides {{OGC:CRS84}} / {{OGC:CRS27}} /
> {{OGC:CRS83}} for 4326 / 4267 / 4269, with the EPSG spellings still
> resolving); ship the CSV in the classic and client wheels via
> {{package_data}}; add a Python counterpart of
> {{SpatialReferenceSystemMapperSuite}} and extend the {{GeometryType}} /
> {{GeographyType}} tests with registry SRIDs and CRS string IDs.
> {{GeometryType}} and {{GeographyType}} need no change: they already consult
> the mappers.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]