This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new f4a6d028 [SPARK-61135][CONNECT][PYTHON] Parse the JSON of GeometryType 
and GeographyType, such as bare geometry and geography(SPHERICAL), in the 
Python client as PySpark does
f4a6d028 is described below

commit f4a6d0288f8b8285eb306fb871fa8874a998a6bc
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 12:43:16 2026 +0900

    [SPARK-61135][CONNECT][PYTHON] Parse the JSON of GeometryType and 
GeographyType, such as bare geometry and geography(SPHERICAL), in the Python 
client as PySpark does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-61135](https://issues.apache.org/jira/browse/SPARK-61135): The 
client now parses the JSON of `GeometryType` and `GeographyType` as PySpark's 
`_parse_datatype_json_value` does. It uses PySpark's patterns and 
`GeometryType._from_crs` and `GeographyType._from_crs`:
    
    - **Defaults:** bare `geometry` and `geography` are of the default CRS 
`OGC:CRS84` (SRID 4326). `geography(SPHERICAL)` is of the default CRS with that 
algorithm.
    - **CRSs:** a CRS is one of `geo_utils` (`SRID:0`, `EPSG:3857` and 
`OGC:CRS84` for a geometry, `OGC:CRS84` for a geography), or `SRID:ANY` of any 
case, the mixed SRID.
    - **Errors:**
      - an unknown CRS raises `IllegalArgumentException` `ST_INVALID_CRS_VALUE`
      - an algorithm other than `SPHERICAL` raises `IllegalArgumentException` 
`ST_INVALID_ALGORITHM_VALUE`
      - another form, such as `geometry(4326)`, raises `PySparkValueError` 
`CANNOT_PARSE_DATATYPE`
    
    The JSON strings of the spatial types are parsed this way in Python. The 
others still go to the Rust grammar, whose DDL form of a spatial type, such as 
`geometry(4326)`, is not changed.
    
    `GeometryType` and `GeographyType` also gain PySpark's classmethods 
`_from_crs(crs)` and `_from_crs(crs, alg)`, of the same logic.
    
    ### Why are the changes needed?
    
    The Rust grammar parsed the JSON of the spatial types more loosely than 
PySpark does:
    - It rejected bare `geometry`, bare `geography` and `geography(SPHERICAL)` 
with `CANNOT_PARSE_DATATYPE`.
    - It accepted 21 of the forms that PySpark rejects, such as 
`geography(4326)`, `geography(ANY)` and `geometry(SRID:1)`.
    
    Apache Spark's `test_geometry_json_serde` and `test_geography_json_serde` 
failed on this client, and so did 5 tests of `test_parity_geometrytype.py` and 
`test_parity_geographytype.py`, which call `_from_crs`.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. The JSON of the spatial types parses, and fails, as with PySpark 4.3.0.
    
    ### How was this patch tested?
    
    - A new offline test checks:
      - the types of the valid forms and the round trips of their JSON
      - the error classes of the invalid forms
      - the classmethods `_from_crs`
    
      It passes with official PySpark 4.3.0, and fails without this change. All 
60 valid and invalid forms of Apache Spark's tests, and a few more, give the 
same types and errors, with the same parameters, as official PySpark 4.3.0.
    - Apache Spark's spatial tests of `test_parity_types.py` run on this client 
against a local server: 7 pass and 1 is skipped, including the two tests above.
    - Apache Spark's `test_parity_geometrytype.py` and 
`test_parity_geographytype.py` pass on this client against a local server, all 
19 tests.
    - The lib tests of both crates (590 passed), the build of all tests, also 
with `wasm-udf`, the offline Python suites (2202 passed, 8 skipped), and the 
live drop-in and Arrow suites (184 passed) and live cases (10 passed) against a 
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #330 from HyukjinKwon/SPARK-61135-spatial-json.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/types.rs      | 28 +++++++++++++++
 python/pyspark/sql/types.py         | 70 +++++++++++++++++++++++++++++++++++++
 python/tests/test_dropin_offline.py | 51 +++++++++++++++++++++++++++
 3 files changed, 149 insertions(+)

diff --git a/crates/pyspark-rs/src/types.rs b/crates/pyspark-rs/src/types.rs
index 6c58da2e..f75050b6 100644
--- a/crates/pyspark-rs/src/types.rs
+++ b/crates/pyspark-rs/src/types.rs
@@ -3146,6 +3146,19 @@ impl PyGeometryType {
     fn __reduce__(&self, py: Python<'_>) -> PyResult<(Py<PyAny>, (String,))> {
         type_reduce(py, &DataType::Geometry { srid: self.srid }.json())
     }
+    /// As PySpark's `GeometryType._from_crs(crs)`, of the JSON of a geometry 
type: the mixed
+    /// SRID of `SRID:ANY` of any case, or else the SRID of a CRS of 
`geo_utils`, and
+    /// `ST_INVALID_CRS_VALUE` of another.
+    #[classmethod]
+    fn _from_crs<'py>(
+        _cls: &Bound<'py, pyo3::types::PyType>,
+        py: Python<'py>,
+        crs: &Bound<'py, PyAny>,
+    ) -> PyResult<Bound<'py, PyAny>> {
+        py.import("pyspark.sql.types")?
+            .getattr("_geometry_type_from_crs")?
+            .call1((crs,))
+    }
     fn __repr__(&self) -> String {
         match self.srid {
             -1 => "GeometryType(ANY)".to_string(),
@@ -3194,6 +3207,21 @@ impl PyGeographyType {
     fn __reduce__(&self, py: Python<'_>) -> PyResult<(Py<PyAny>, (String,))> {
         type_reduce(py, &DataType::Geography { srid: self.srid }.json())
     }
+    /// As PySpark's `GeographyType._from_crs(crs, alg)`, of the JSON of a 
geography type:
+    /// `ST_INVALID_ALGORITHM_VALUE` of another algorithm than `SPHERICAL`, 
the mixed SRID of
+    /// `SRID:ANY` of any case, or else the SRID of a geographic CRS of 
`geo_utils`, and
+    /// `ST_INVALID_CRS_VALUE` of another.
+    #[classmethod]
+    fn _from_crs<'py>(
+        _cls: &Bound<'py, pyo3::types::PyType>,
+        py: Python<'py>,
+        crs: &Bound<'py, PyAny>,
+        alg: &Bound<'py, PyAny>,
+    ) -> PyResult<Bound<'py, PyAny>> {
+        py.import("pyspark.sql.types")?
+            .getattr("_geography_type_from_crs")?
+            .call1((crs, alg))
+    }
     fn __repr__(&self) -> String {
         match self.srid {
             -1 => "GeographyType(ANY)".to_string(),
diff --git a/python/pyspark/sql/types.py b/python/pyspark/sql/types.py
index 75b4b7bb..33a92e30 100644
--- a/python/pyspark/sql/types.py
+++ b/python/pyspark/sql/types.py
@@ -2,6 +2,7 @@
 Minimal pyspark.sql.types module for the Spark Connect client.
 Provides DataType classes and conversion functions.
 """
+import re
 from typing import cast  # noqa: F401  (upstream leaks typing.cast from this 
module)
 
 # Import from _pyspark for the actual DataType implementations
@@ -341,6 +342,8 @@ def _parse_datatype_json_value(json_value, fieldPath="", 
collationsMap=None):
         ):
             _assert_valid_type_for_collation(fieldPath, json_value, 
collationsMap)
             return StringType(collationsMap[fieldPath])
+        if isinstance(json_value, str) and json_value.startswith(("geometry", 
"geography")):
+            return _spatial_type_from_json(json_value)
         from pyspark._pyspark import parse_datatype_json
 
         try:
@@ -365,6 +368,73 @@ def _parse_datatype_json_value(json_value, fieldPath="", 
collationsMap=None):
     raise _pyspark_value_error("UNSUPPORTED_DATA_TYPE", {"data_type": 
str(tpe)})
 
 
+# The patterns of the JSON of the spatial types, as PySpark's.
+_GEOMETRY = re.compile(r"^geometry$")
+_GEOMETRY_CRS = re.compile(r"geometry\(\s*([\w]+:-?[\w]+)\s*\)")
+_GEOGRAPHY = re.compile(r"^geography$")
+_GEOGRAPHY_CRS = re.compile(r"geography\(\s*([\w]+:-?[\w]+)\s*\)")
+_GEOGRAPHY_CRS_ALG = 
re.compile(r"geography\(\s*([\w]+:-?[\w]+)\s*,\s*(\w+)\s*\)")
+_GEOGRAPHY_ALG = re.compile(r"geography\(\s*(\w+)\s*\)")
+
+
+def _spatial_type_from_json(json_value):
+    """The GeometryType or GeographyType of the JSON string ``json_value``, as 
PySpark's
+    ``_parse_datatype_json_value`` parses it, of its patterns: of the default 
CRS
+    ``OGC:CRS84`` and algorithm ``SPHERICAL`` without them, and 
``CANNOT_PARSE_DATATYPE`` of
+    another form."""
+    if _GEOMETRY.match(json_value):
+        return _geometry_type_from_crs("OGC:CRS84")
+    elif _GEOMETRY_CRS.match(json_value):
+        return 
_geometry_type_from_crs(_GEOMETRY_CRS.match(json_value).group(1))
+    elif _GEOGRAPHY.match(json_value):
+        return _geography_type_from_crs("OGC:CRS84", "SPHERICAL")
+    elif _GEOGRAPHY_CRS.match(json_value):
+        return 
_geography_type_from_crs(_GEOGRAPHY_CRS.match(json_value).group(1), "SPHERICAL")
+    elif _GEOGRAPHY_CRS_ALG.match(json_value):
+        crs_alg = _GEOGRAPHY_CRS_ALG.match(json_value)
+        return _geography_type_from_crs(crs_alg.group(1), crs_alg.group(2))
+    elif _GEOGRAPHY_ALG.match(json_value):
+        return _geography_type_from_crs("OGC:CRS84", 
_GEOGRAPHY_ALG.match(json_value).group(1))
+    raise _pyspark_value_error("CANNOT_PARSE_DATATYPE", {"msg": 
str(json_value)})
+
+
+def _geometry_type_from_crs(crs):
+    """``GeometryType._from_crs`` of PySpark: the mixed SRID of the CRS 
``SRID:ANY`` of any
+    case, or else the SRID of a CRS of ``geo_utils``, and 
``ST_INVALID_CRS_VALUE`` of another."""
+    from pyspark.errors import IllegalArgumentException
+    from pyspark.sql.geo_utils import CartesianSpatialReferenceSystemMapper
+
+    if crs.lower() == "srid:any":
+        return GeometryType("ANY")
+    srid = CartesianSpatialReferenceSystemMapper.get_srid(crs)
+    if srid is None:
+        raise IllegalArgumentException(
+            errorClass="ST_INVALID_CRS_VALUE", messageParameters={"crs": 
str(crs)}
+        )
+    return GeometryType(srid)
+
+
+def _geography_type_from_crs(crs, alg):
+    """``GeographyType._from_crs`` of PySpark: ``ST_INVALID_ALGORITHM_VALUE`` 
of another
+    algorithm than ``SPHERICAL``, the mixed SRID of the CRS ``SRID:ANY`` of 
any case, or else
+    the SRID of a geographic CRS of ``geo_utils``, and 
``ST_INVALID_CRS_VALUE`` of another."""
+    from pyspark.errors import IllegalArgumentException
+    from pyspark.sql.geo_utils import GeographicSpatialReferenceSystemMapper
+
+    if alg != "SPHERICAL":
+        raise IllegalArgumentException(
+            errorClass="ST_INVALID_ALGORITHM_VALUE", messageParameters={"alg": 
str(alg)}
+        )
+    if crs.lower() == "srid:any":
+        return GeographyType("ANY")
+    srid = GeographicSpatialReferenceSystemMapper.get_srid(crs)
+    if srid is None:
+        raise IllegalArgumentException(
+            errorClass="ST_INVALID_CRS_VALUE", messageParameters={"crs": 
str(crs)}
+        )
+    return GeographyType(srid)
+
+
 def _array_type_from_json(json_value, fieldPath="", collationsMap=None):
     """``ArrayType.fromJson``: its element at ``element`` of ``fieldPath``, as 
PySpark's."""
     elementType = _parse_datatype_json_value(
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index 2612868b..599a8daa 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -2862,6 +2862,57 @@ def 
test_collected_spatial_values_are_their_objects_as_pyspark():
     assert g(None) is None
 
 
+def test_spatial_types_are_parsed_of_their_json_as_pyspark():
+    # As PySpark's _parse_datatype_json_value, of the patterns of the JSON of 
the spatial types:
+    # the default CRS OGC:CRS84 and algorithm SPHERICAL without them, the CRSs 
of geo_utils and
+    # SRID:ANY of any case, and the errors of PySpark of the other forms.
+    from pyspark.errors import IllegalArgumentException, PySparkValueError
+    from pyspark.sql.types import _parse_datatype_json_string, 
_parse_datatype_json_value
+
+    valid = [
+        ("geometry", T.GeometryType(4326)),
+        ("geometry(OGC:CRS84)", T.GeometryType(4326)),
+        ("geometry(EPSG:3857)", T.GeometryType(3857)),
+        ("geometry(SRID:0)", T.GeometryType(0)),
+        ("geometry(srid:any)", T.GeometryType("ANY")),
+        ("geography", T.GeographyType(4326)),
+        ("geography(SPHERICAL)", T.GeographyType(4326)),
+        ("geography(OGC:CRS84, SPHERICAL)", T.GeographyType(4326)),
+        ("geography(SRID:ANY)", T.GeographyType("ANY")),
+    ]
+    for json, expected in valid:
+        assert _parse_datatype_json_value(json) == expected
+        assert _parse_datatype_json_string(expected.json()) == expected
+    invalid = [
+        ("geometry(4326)", PySparkValueError, "CANNOT_PARSE_DATATYPE"),
+        ("geometry()", PySparkValueError, "CANNOT_PARSE_DATATYPE"),
+        ("geometry(SRID:1)", IllegalArgumentException, "ST_INVALID_CRS_VALUE"),
+        ("geometry(ocg:crs84)", IllegalArgumentException, 
"ST_INVALID_CRS_VALUE"),
+        ("geography(4326)", IllegalArgumentException, 
"ST_INVALID_ALGORITHM_VALUE"),
+        ("geography(EPSG:3857)", IllegalArgumentException, 
"ST_INVALID_CRS_VALUE"),
+        ("geography(OGC:CRS84, spherical)", IllegalArgumentException, 
"ST_INVALID_ALGORITHM_VALUE"),
+        ("geography(SRID:0,)", PySparkValueError, "CANNOT_PARSE_DATATYPE"),
+    ]
+    for json, error, condition in invalid:
+        with pytest.raises(error) as e:
+            _parse_datatype_json_value(json)
+        assert e.value.getCondition() == condition
+    # Of the classmethods `_from_crs` of the types, of which PySpark's JSON 
parses them.
+    assert T.GeometryType._from_crs("EPSG:3857") == T.GeometryType(3857)
+    assert T.GeometryType._from_crs("srid:ANY") == T.GeometryType("ANY")
+    assert T.GeographyType._from_crs("OGC:CRS84", "SPHERICAL") == 
T.GeographyType(4326)
+    for make, condition in [
+        (lambda: T.GeometryType._from_crs("EPSG:4326"), 
"ST_INVALID_CRS_VALUE"),
+        (lambda: T.GeographyType._from_crs("SRID:0", "SPHERICAL"), 
"ST_INVALID_CRS_VALUE"),
+        (lambda: T.GeographyType._from_crs("OGC:CRS84", "PLANAR"), 
"ST_INVALID_ALGORITHM_VALUE"),
+    ]:
+        with pytest.raises(IllegalArgumentException) as e:
+            make()
+        assert e.value.getCondition() == condition
+
+
+
+
 def test_variant_and_geo_values():
     from pyspark.sql.types import VariantVal, Geometry, Geography
     v = VariantVal.parseJson('{"a": 1, "b": [2, 3]}')


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to