This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new f4a6d028 [SPARK-61135][CONNECT][PYTHON] Parse the JSON of GeometryType
and GeographyType, such as bare geometry and geography(SPHERICAL), in the
Python client as PySpark does
f4a6d028 is described below
commit f4a6d0288f8b8285eb306fb871fa8874a998a6bc
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 12:43:16 2026 +0900
[SPARK-61135][CONNECT][PYTHON] Parse the JSON of GeometryType and
GeographyType, such as bare geometry and geography(SPHERICAL), in the Python
client as PySpark does
### What changes were proposed in this pull request?
[SPARK-61135](https://issues.apache.org/jira/browse/SPARK-61135): The
client now parses the JSON of `GeometryType` and `GeographyType` as PySpark's
`_parse_datatype_json_value` does. It uses PySpark's patterns and
`GeometryType._from_crs` and `GeographyType._from_crs`:
- **Defaults:** bare `geometry` and `geography` are of the default CRS
`OGC:CRS84` (SRID 4326). `geography(SPHERICAL)` is of the default CRS with that
algorithm.
- **CRSs:** a CRS is one of `geo_utils` (`SRID:0`, `EPSG:3857` and
`OGC:CRS84` for a geometry, `OGC:CRS84` for a geography), or `SRID:ANY` of any
case, the mixed SRID.
- **Errors:**
- an unknown CRS raises `IllegalArgumentException` `ST_INVALID_CRS_VALUE`
- an algorithm other than `SPHERICAL` raises `IllegalArgumentException`
`ST_INVALID_ALGORITHM_VALUE`
- another form, such as `geometry(4326)`, raises `PySparkValueError`
`CANNOT_PARSE_DATATYPE`
The JSON strings of the spatial types are parsed this way in Python. The
others still go to the Rust grammar, whose DDL form of a spatial type, such as
`geometry(4326)`, is not changed.
`GeometryType` and `GeographyType` also gain PySpark's classmethods
`_from_crs(crs)` and `_from_crs(crs, alg)`, of the same logic.
### Why are the changes needed?
The Rust grammar parsed the JSON of the spatial types more loosely than
PySpark does:
- It rejected bare `geometry`, bare `geography` and `geography(SPHERICAL)`
with `CANNOT_PARSE_DATATYPE`.
- It accepted 21 of the forms that PySpark rejects, such as
`geography(4326)`, `geography(ANY)` and `geometry(SRID:1)`.
Apache Spark's `test_geometry_json_serde` and `test_geography_json_serde`
failed on this client, and so did 5 tests of `test_parity_geometrytype.py` and
`test_parity_geographytype.py`, which call `_from_crs`.
### Does this PR introduce _any_ user-facing change?
Yes. The JSON of the spatial types parses, and fails, as with PySpark 4.3.0.
### How was this patch tested?
- A new offline test checks:
- the types of the valid forms and the round trips of their JSON
- the error classes of the invalid forms
- the classmethods `_from_crs`
It passes with official PySpark 4.3.0, and fails without this change. All
60 valid and invalid forms of Apache Spark's tests, and a few more, give the
same types and errors, with the same parameters, as official PySpark 4.3.0.
- Apache Spark's spatial tests of `test_parity_types.py` run on this client
against a local server: 7 pass and 1 is skipped, including the two tests above.
- Apache Spark's `test_parity_geometrytype.py` and
`test_parity_geographytype.py` pass on this client against a local server, all
19 tests.
- The lib tests of both crates (590 passed), the build of all tests, also
with `wasm-udf`, the offline Python suites (2202 passed, 8 skipped), and the
live drop-in and Arrow suites (184 passed) and live cases (10 passed) against a
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #330 from HyukjinKwon/SPARK-61135-spatial-json.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/types.rs | 28 +++++++++++++++
python/pyspark/sql/types.py | 70 +++++++++++++++++++++++++++++++++++++
python/tests/test_dropin_offline.py | 51 +++++++++++++++++++++++++++
3 files changed, 149 insertions(+)
diff --git a/crates/pyspark-rs/src/types.rs b/crates/pyspark-rs/src/types.rs
index 6c58da2e..f75050b6 100644
--- a/crates/pyspark-rs/src/types.rs
+++ b/crates/pyspark-rs/src/types.rs
@@ -3146,6 +3146,19 @@ impl PyGeometryType {
fn __reduce__(&self, py: Python<'_>) -> PyResult<(Py<PyAny>, (String,))> {
type_reduce(py, &DataType::Geometry { srid: self.srid }.json())
}
+ /// As PySpark's `GeometryType._from_crs(crs)`, of the JSON of a geometry
type: the mixed
+ /// SRID of `SRID:ANY` of any case, or else the SRID of a CRS of
`geo_utils`, and
+ /// `ST_INVALID_CRS_VALUE` of another.
+ #[classmethod]
+ fn _from_crs<'py>(
+ _cls: &Bound<'py, pyo3::types::PyType>,
+ py: Python<'py>,
+ crs: &Bound<'py, PyAny>,
+ ) -> PyResult<Bound<'py, PyAny>> {
+ py.import("pyspark.sql.types")?
+ .getattr("_geometry_type_from_crs")?
+ .call1((crs,))
+ }
fn __repr__(&self) -> String {
match self.srid {
-1 => "GeometryType(ANY)".to_string(),
@@ -3194,6 +3207,21 @@ impl PyGeographyType {
fn __reduce__(&self, py: Python<'_>) -> PyResult<(Py<PyAny>, (String,))> {
type_reduce(py, &DataType::Geography { srid: self.srid }.json())
}
+ /// As PySpark's `GeographyType._from_crs(crs, alg)`, of the JSON of a
geography type:
+ /// `ST_INVALID_ALGORITHM_VALUE` of another algorithm than `SPHERICAL`,
the mixed SRID of
+ /// `SRID:ANY` of any case, or else the SRID of a geographic CRS of
`geo_utils`, and
+ /// `ST_INVALID_CRS_VALUE` of another.
+ #[classmethod]
+ fn _from_crs<'py>(
+ _cls: &Bound<'py, pyo3::types::PyType>,
+ py: Python<'py>,
+ crs: &Bound<'py, PyAny>,
+ alg: &Bound<'py, PyAny>,
+ ) -> PyResult<Bound<'py, PyAny>> {
+ py.import("pyspark.sql.types")?
+ .getattr("_geography_type_from_crs")?
+ .call1((crs, alg))
+ }
fn __repr__(&self) -> String {
match self.srid {
-1 => "GeographyType(ANY)".to_string(),
diff --git a/python/pyspark/sql/types.py b/python/pyspark/sql/types.py
index 75b4b7bb..33a92e30 100644
--- a/python/pyspark/sql/types.py
+++ b/python/pyspark/sql/types.py
@@ -2,6 +2,7 @@
Minimal pyspark.sql.types module for the Spark Connect client.
Provides DataType classes and conversion functions.
"""
+import re
from typing import cast # noqa: F401 (upstream leaks typing.cast from this
module)
# Import from _pyspark for the actual DataType implementations
@@ -341,6 +342,8 @@ def _parse_datatype_json_value(json_value, fieldPath="",
collationsMap=None):
):
_assert_valid_type_for_collation(fieldPath, json_value,
collationsMap)
return StringType(collationsMap[fieldPath])
+ if isinstance(json_value, str) and json_value.startswith(("geometry",
"geography")):
+ return _spatial_type_from_json(json_value)
from pyspark._pyspark import parse_datatype_json
try:
@@ -365,6 +368,73 @@ def _parse_datatype_json_value(json_value, fieldPath="",
collationsMap=None):
raise _pyspark_value_error("UNSUPPORTED_DATA_TYPE", {"data_type":
str(tpe)})
+# The patterns of the JSON of the spatial types, as PySpark's.
+_GEOMETRY = re.compile(r"^geometry$")
+_GEOMETRY_CRS = re.compile(r"geometry\(\s*([\w]+:-?[\w]+)\s*\)")
+_GEOGRAPHY = re.compile(r"^geography$")
+_GEOGRAPHY_CRS = re.compile(r"geography\(\s*([\w]+:-?[\w]+)\s*\)")
+_GEOGRAPHY_CRS_ALG =
re.compile(r"geography\(\s*([\w]+:-?[\w]+)\s*,\s*(\w+)\s*\)")
+_GEOGRAPHY_ALG = re.compile(r"geography\(\s*(\w+)\s*\)")
+
+
+def _spatial_type_from_json(json_value):
+ """The GeometryType or GeographyType of the JSON string ``json_value``, as
PySpark's
+ ``_parse_datatype_json_value`` parses it, of its patterns: of the default
CRS
+ ``OGC:CRS84`` and algorithm ``SPHERICAL`` without them, and
``CANNOT_PARSE_DATATYPE`` of
+ another form."""
+ if _GEOMETRY.match(json_value):
+ return _geometry_type_from_crs("OGC:CRS84")
+ elif _GEOMETRY_CRS.match(json_value):
+ return
_geometry_type_from_crs(_GEOMETRY_CRS.match(json_value).group(1))
+ elif _GEOGRAPHY.match(json_value):
+ return _geography_type_from_crs("OGC:CRS84", "SPHERICAL")
+ elif _GEOGRAPHY_CRS.match(json_value):
+ return
_geography_type_from_crs(_GEOGRAPHY_CRS.match(json_value).group(1), "SPHERICAL")
+ elif _GEOGRAPHY_CRS_ALG.match(json_value):
+ crs_alg = _GEOGRAPHY_CRS_ALG.match(json_value)
+ return _geography_type_from_crs(crs_alg.group(1), crs_alg.group(2))
+ elif _GEOGRAPHY_ALG.match(json_value):
+ return _geography_type_from_crs("OGC:CRS84",
_GEOGRAPHY_ALG.match(json_value).group(1))
+ raise _pyspark_value_error("CANNOT_PARSE_DATATYPE", {"msg":
str(json_value)})
+
+
+def _geometry_type_from_crs(crs):
+ """``GeometryType._from_crs`` of PySpark: the mixed SRID of the CRS
``SRID:ANY`` of any
+ case, or else the SRID of a CRS of ``geo_utils``, and
``ST_INVALID_CRS_VALUE`` of another."""
+ from pyspark.errors import IllegalArgumentException
+ from pyspark.sql.geo_utils import CartesianSpatialReferenceSystemMapper
+
+ if crs.lower() == "srid:any":
+ return GeometryType("ANY")
+ srid = CartesianSpatialReferenceSystemMapper.get_srid(crs)
+ if srid is None:
+ raise IllegalArgumentException(
+ errorClass="ST_INVALID_CRS_VALUE", messageParameters={"crs":
str(crs)}
+ )
+ return GeometryType(srid)
+
+
+def _geography_type_from_crs(crs, alg):
+ """``GeographyType._from_crs`` of PySpark: ``ST_INVALID_ALGORITHM_VALUE``
of another
+ algorithm than ``SPHERICAL``, the mixed SRID of the CRS ``SRID:ANY`` of
any case, or else
+ the SRID of a geographic CRS of ``geo_utils``, and
``ST_INVALID_CRS_VALUE`` of another."""
+ from pyspark.errors import IllegalArgumentException
+ from pyspark.sql.geo_utils import GeographicSpatialReferenceSystemMapper
+
+ if alg != "SPHERICAL":
+ raise IllegalArgumentException(
+ errorClass="ST_INVALID_ALGORITHM_VALUE", messageParameters={"alg":
str(alg)}
+ )
+ if crs.lower() == "srid:any":
+ return GeographyType("ANY")
+ srid = GeographicSpatialReferenceSystemMapper.get_srid(crs)
+ if srid is None:
+ raise IllegalArgumentException(
+ errorClass="ST_INVALID_CRS_VALUE", messageParameters={"crs":
str(crs)}
+ )
+ return GeographyType(srid)
+
+
def _array_type_from_json(json_value, fieldPath="", collationsMap=None):
"""``ArrayType.fromJson``: its element at ``element`` of ``fieldPath``, as
PySpark's."""
elementType = _parse_datatype_json_value(
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index 2612868b..599a8daa 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -2862,6 +2862,57 @@ def
test_collected_spatial_values_are_their_objects_as_pyspark():
assert g(None) is None
+def test_spatial_types_are_parsed_of_their_json_as_pyspark():
+ # As PySpark's _parse_datatype_json_value, of the patterns of the JSON of
the spatial types:
+ # the default CRS OGC:CRS84 and algorithm SPHERICAL without them, the CRSs
of geo_utils and
+ # SRID:ANY of any case, and the errors of PySpark of the other forms.
+ from pyspark.errors import IllegalArgumentException, PySparkValueError
+ from pyspark.sql.types import _parse_datatype_json_string,
_parse_datatype_json_value
+
+ valid = [
+ ("geometry", T.GeometryType(4326)),
+ ("geometry(OGC:CRS84)", T.GeometryType(4326)),
+ ("geometry(EPSG:3857)", T.GeometryType(3857)),
+ ("geometry(SRID:0)", T.GeometryType(0)),
+ ("geometry(srid:any)", T.GeometryType("ANY")),
+ ("geography", T.GeographyType(4326)),
+ ("geography(SPHERICAL)", T.GeographyType(4326)),
+ ("geography(OGC:CRS84, SPHERICAL)", T.GeographyType(4326)),
+ ("geography(SRID:ANY)", T.GeographyType("ANY")),
+ ]
+ for json, expected in valid:
+ assert _parse_datatype_json_value(json) == expected
+ assert _parse_datatype_json_string(expected.json()) == expected
+ invalid = [
+ ("geometry(4326)", PySparkValueError, "CANNOT_PARSE_DATATYPE"),
+ ("geometry()", PySparkValueError, "CANNOT_PARSE_DATATYPE"),
+ ("geometry(SRID:1)", IllegalArgumentException, "ST_INVALID_CRS_VALUE"),
+ ("geometry(ocg:crs84)", IllegalArgumentException,
"ST_INVALID_CRS_VALUE"),
+ ("geography(4326)", IllegalArgumentException,
"ST_INVALID_ALGORITHM_VALUE"),
+ ("geography(EPSG:3857)", IllegalArgumentException,
"ST_INVALID_CRS_VALUE"),
+ ("geography(OGC:CRS84, spherical)", IllegalArgumentException,
"ST_INVALID_ALGORITHM_VALUE"),
+ ("geography(SRID:0,)", PySparkValueError, "CANNOT_PARSE_DATATYPE"),
+ ]
+ for json, error, condition in invalid:
+ with pytest.raises(error) as e:
+ _parse_datatype_json_value(json)
+ assert e.value.getCondition() == condition
+ # Of the classmethods `_from_crs` of the types, of which PySpark's JSON
parses them.
+ assert T.GeometryType._from_crs("EPSG:3857") == T.GeometryType(3857)
+ assert T.GeometryType._from_crs("srid:ANY") == T.GeometryType("ANY")
+ assert T.GeographyType._from_crs("OGC:CRS84", "SPHERICAL") ==
T.GeographyType(4326)
+ for make, condition in [
+ (lambda: T.GeometryType._from_crs("EPSG:4326"),
"ST_INVALID_CRS_VALUE"),
+ (lambda: T.GeographyType._from_crs("SRID:0", "SPHERICAL"),
"ST_INVALID_CRS_VALUE"),
+ (lambda: T.GeographyType._from_crs("OGC:CRS84", "PLANAR"),
"ST_INVALID_ALGORITHM_VALUE"),
+ ]:
+ with pytest.raises(IllegalArgumentException) as e:
+ make()
+ assert e.value.getCondition() == condition
+
+
+
+
def test_variant_and_geo_values():
from pyspark.sql.types import VariantVal, Geometry, Geography
v = VariantVal.parseJson('{"a": 1, "b": [2, 3]}')
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]