This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new a8d6c76  [SPARK-60907][CONNECT] Send reader schemas as PySpark does, 
and accept a StructType in schema()
a8d6c76 is described below

commit a8d6c76a4da1f9f90ae07130a4e8542781e192b3
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Thu Nov 5 19:05:57 2026 +0900

    [SPARK-60907][CONNECT] Send reader schemas as PySpark does, and accept a 
StructType in schema()
    
    ### What changes were proposed in this pull request?
    
    [SPARK-60907](https://issues.apache.org/jira/browse/SPARK-60907): reader 
schemas of the Rust-backed Python client are now sent as PySpark Connect 4.3.0 
sends them.
    
    - `DataFrameReader.schema` and `DataStreamReader.schema` accept a 
StructType as well as a string.
    - As in PySpark, a StructType is sent as its JSON and a string as is, for 
the server to parse. Anything else raises `NOT_EXPECTED_TYPE` with PySpark's 
parameters.
    - The `schema` arguments of `load()`, `csv()`, `json()` and the other 
format methods, batch and streaming, go through the same conversion, as they go 
through `schema()` in PySpark. Before, they were parsed on the client and sent 
as a simple string.
    - The JSON of a spatial type now names its SRID's coordinate reference 
system, as PySpark's does: `geometry(OGC:CRS84)` for 4326, 
`geometry(EPSG:3857)`, `geometry(SRID:0)` and `geometry(SRID:ANY)`, and 
`geography(OGC:CRS84, SPHERICAL)`. It wrote `OGC:CRS84` for any SRID, so that 
`GeometryType(3857)` in a reader schema would reach the server as 4326. The 
JSON parser reads these names back, so spatial types now round-trip through 
JSON.
    
    ### Why are the changes needed?
    
    `spark.read.schema(struct_type)` and `spark.readStream.schema(struct_type)` 
raised `TypeError`. The format methods' `schema` arguments lost what the simple 
string cannot express: a StructType with field metadata, such as a comment, 
failed on the server with `PARSE_SYNTAX_ERROR`, and a DDL string with `COMMENT` 
failed in the client parser with `ValueError`. Data types other than a 
StructType were accepted, while PySpark rejects them.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes, these now match PySpark Connect 4.3.0:
    
    - `spark.read.schema(struct_type)` and 
`spark.readStream.schema(struct_type)` work.
    - A schema with field metadata, given as a StructType or as DDL with 
`COMMENT`, now works and keeps the metadata, through `schema()` and through the 
`schema` arguments.
    - A data type other than a StructType raises `NOT_EXPECTED_TYPE` (a 
`PySparkTypeError`).
    
    ### How was this patch tested?
    
    - Against a local Spark 4.3.0 server, a probe of seven reader calls, with a 
StructType carrying field metadata, DDL with `COMMENT`, and a `LongType`, 
through `read.schema()`, `read.csv(schema=...)` and `readStream.schema()`, 
gives output identical to official PySpark 4.3.0: the resulting field metadata 
and rows, or the error and its parameters.
    - New offline tests check, over the mock server, the schema that both 
readers send for `schema()` and the `schema` arguments of `load()` and `csv()`: 
the JSON of a StructType, as official PySpark 4.3.0 writes it, or a string as 
is. They also check `NOT_EXPECTED_TYPE` for anything else. Against master they 
fail, where the StructType was sent as `struct<a:bigint>` or raised `TypeError`.
    - New unit tests check the JSON of the spatial types, as official PySpark 
4.3.0 writes it, and that it reads back. The JSON round-trip test now covers 
the spatial types too. Against a local server, a reader schema with 
`GeometryType(3857)`, `GeometryType(4326)`, `GeometryType(0)` or 
`GeographyType(4326)` reaches the server with that SRID, as with official 
PySpark.
    - A new live test reads a CSV file through `read.schema()`, 
`read.csv(schema=...)` and `readStream.schema()`, with both a StructType and 
DDL, and checks the field metadata and rows; it also passes with official 
PySpark 4.3.0.
    - The lib tests and every integration test, e2e included, against the local 
server (629 passed), the build of all tests with `wasm-udf`, the offline Python 
suites (426 passed, 8 live cases skipped), and the live drop-in and Arrow 
suites (23 passed) and live cases (9 passed). Clippy reports no diagnostics in 
the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #151 from HyukjinKwon/SPARK-60907-reader-schema.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/readwriter.rs           | 44 ++++++++++---
 crates/pyspark-rs/src/streaming.rs            | 16 +++--
 crates/spark-connect/src/types.rs             | 90 ++++++++++++++++-----------
 crates/spark-connect/tests/types_roundtrip.rs | 41 +++++++++---
 python/tests/_mock_connect_server.py          |  4 ++
 python/tests/test_dropin_e2e.py               | 18 ++++++
 python/tests/test_dropin_offline.py           | 40 ++++++++++++
 7 files changed, 193 insertions(+), 60 deletions(-)

diff --git a/crates/pyspark-rs/src/readwriter.rs 
b/crates/pyspark-rs/src/readwriter.rs
index 3c0db1b..217034a 100644
--- a/crates/pyspark-rs/src/readwriter.rs
+++ b/crates/pyspark-rs/src/readwriter.rs
@@ -62,11 +62,35 @@ fn set_opt(
     }
 }
 
-/// Resolve a reader `schema` arg (a StructType/DataType or a DDL string) to 
the
-/// reader's DDL schema.
+/// A reader `schema` as pyspark's `DataFrameReader.schema` and 
`DataStreamReader.schema`
+/// send it: a StructType as its JSON, which keeps field metadata, and a 
string as is, for
+/// the server to parse. Anything else raises `NOT_EXPECTED_TYPE`.
+pub(crate) fn reader_schema(schema: &Bound<'_, PyAny>) -> PyResult<String> {
+    if let Ok(schema) = schema.extract::<String>() {
+        return Ok(schema);
+    }
+    if let Ok(st) = schema.extract::<PyRef<crate::types::PyStructType>>() {
+        return Ok(spark_connect::types::DataType::Struct {
+            fields: st.fields.clone(),
+        }
+        .json());
+    }
+    let arg_type = schema.get_type().name()?.to_string();
+    Err(crate::column::raise_pyspark(
+        schema.py(),
+        "PySparkTypeError",
+        "NOT_EXPECTED_TYPE",
+        &[
+            ("arg_name", "schema"),
+            ("expected_type", "str or struct type"),
+            ("arg_type", &arg_type),
+        ],
+    ))
+}
+
+/// Apply the `schema` argument of a format method, as pyspark does through 
`schema()`.
 fn apply_reader_schema(r: DataFrameReader, v: &Bound<'_, PyAny>) -> 
PyResult<DataFrameReader> {
-    let dt = crate::types::py_to_data_type(v)?;
-    Ok(r.schema(dt.simple_string()))
+    Ok(r.schema(reader_schema(v)?))
 }
 
 #[pymethods]
@@ -77,10 +101,14 @@ impl PyDataFrameReader {
         slf
     }
 
-    /// Set the schema (a DDL string).
-    fn schema<'py>(mut slf: PyRefMut<'py, Self>, schema: &str) -> 
PyRefMut<'py, Self> {
-        slf.inner = slf.inner.clone().schema(schema.to_string());
-        slf
+    /// Set the schema: a StructType or a DDL string, as in pyspark.
+    fn schema<'py>(
+        mut slf: PyRefMut<'py, Self>,
+        schema: &Bound<'_, PyAny>,
+    ) -> PyResult<PyRefMut<'py, Self>> {
+        let schema = reader_schema(schema)?;
+        slf.inner = slf.inner.clone().schema(schema);
+        Ok(slf)
     }
 
     /// Set a single read option. `None` leaves the option unset (not the 
string
diff --git a/crates/pyspark-rs/src/streaming.rs 
b/crates/pyspark-rs/src/streaming.rs
index 88d3baf..ef080b1 100644
--- a/crates/pyspark-rs/src/streaming.rs
+++ b/crates/pyspark-rs/src/streaming.rs
@@ -57,11 +57,10 @@ fn set_ropt(
     }
 }
 
-/// Apply a `schema` argument (a DDL string or a StructType) to a streaming 
reader,
-/// mirroring the batch `apply_reader_schema`.
+/// Apply a `schema` argument (a StructType or a DDL string) to a streaming 
reader, as the
+/// batch reader does.
 fn apply_stream_schema(r: DataStreamReader, v: &Bound<'_, PyAny>) -> 
PyResult<DataStreamReader> {
-    let dt = crate::types::py_to_data_type(v)?;
-    Ok(r.schema(dt.simple_string()))
+    Ok(r.schema(crate::readwriter::reader_schema(v)?))
 }
 
 /// Coerce a `partitionBy` argument (a single column name or a list of names) 
to a
@@ -96,8 +95,13 @@ impl PyDataStreamReader {
         Ok(slf)
     }
 
-    fn schema<'py>(mut slf: PyRefMut<'py, Self>, schema: &str) -> 
PyResult<PyRefMut<'py, Self>> {
-        slf.inner = slf.inner.clone().schema(schema);
+    /// Set the schema: a StructType or a DDL string, as in pyspark.
+    fn schema<'py>(
+        mut slf: PyRefMut<'py, Self>,
+        schema: &Bound<'_, PyAny>,
+    ) -> PyResult<PyRefMut<'py, Self>> {
+        let schema = crate::readwriter::reader_schema(schema)?;
+        slf.inner = slf.inner.clone().schema(&schema);
         Ok(slf)
     }
 
diff --git a/crates/spark-connect/src/types.rs 
b/crates/spark-connect/src/types.rs
index 976520d..fb026fd 100644
--- a/crates/spark-connect/src/types.rs
+++ b/crates/spark-connect/src/types.rs
@@ -434,19 +434,13 @@ impl DataType {
             }
             DataType::Variant => serde_json::json!("variant"),
             DataType::Geometry { srid } => {
-                if *srid == -1 {
-                    serde_json::json!("geometry(SRID:ANY)")
-                } else {
-                    serde_json::json!(format!("geometry(OGC:CRS84)"))
-                }
-            }
-            DataType::Geography { srid } => {
-                if *srid == -1 {
-                    serde_json::json!("geography(SRID:ANY, SPHERICAL)")
-                } else {
-                    serde_json::json!(format!("geography(OGC:CRS84, 
SPHERICAL)"))
-                }
+                serde_json::json!(format!("geometry({})", spatial_crs(*srid)))
             }
+            DataType::Geography { srid } => serde_json::json!(format!(
+                "geography({}, {})",
+                spatial_crs(*srid),
+                GeographyType::DEFAULT_ALG
+            )),
             DataType::Udt {
                 type_str: _,
                 jvm_class,
@@ -1456,16 +1450,12 @@ fn parse_geometry(s: &str) -> Result<DataType> {
             )
         })?;
 
-    let srid = if inner.to_lowercase() == "any" {
-        -1
-    } else {
-        inner.parse::<i32>().map_err(|_| {
-            SparkError::value(
-                "INVALID_DATATYPE_FORMAT",
-                &[("detail", "Invalid geometry srid")],
-            )
-        })?
-    };
+    let srid = spatial_srid(inner).ok_or_else(|| {
+        SparkError::value(
+            "INVALID_DATATYPE_FORMAT",
+            &[("detail", "Invalid geometry srid")],
+        )
+    })?;
 
     Ok(DataType::Geometry { srid })
 }
@@ -1481,25 +1471,51 @@ fn parse_geography(s: &str) -> Result<DataType> {
             )
         })?;
 
-    let srid = if inner.to_lowercase() == "any" {
-        -1
-    } else {
-        inner
-            .split(',')
-            .next()
-            .unwrap_or("4326")
-            .parse::<i32>()
-            .map_err(|_| {
-                SparkError::value(
-                    "INVALID_DATATYPE_FORMAT",
-                    &[("detail", "Invalid geography srid")],
-                )
-            })?
-    };
+    // The JSON form also names the algorithm, as in `OGC:CRS84, SPHERICAL`.
+    let crs = inner.split(',').next().unwrap_or_default();
+    let srid = spatial_srid(crs).ok_or_else(|| {
+        SparkError::value(
+            "INVALID_DATATYPE_FORMAT",
+            &[("detail", "Invalid geography srid")],
+        )
+    })?;
 
     Ok(DataType::Geography { srid })
 }
 
+/// The coordinate reference system that a spatial type's SRID is named by in 
its JSON, as
+/// PySpark's `geo_utils` names the SRIDs that it supports: `SRID:0`, 
`EPSG:3857` and
+/// `OGC:CRS84` for 4326, and `SRID:ANY` for a mixed SRID. Another SRID, which 
PySpark does not
+/// accept, is named `SRID:<srid>`.
+fn spatial_crs(srid: i32) -> String {
+    match srid {
+        SpatialType::MIXED_SRID => SpatialType::MIXED_CRS.to_string(),
+        3857 => "EPSG:3857".to_string(),
+        GeometryType::DEFAULT_SRID => GeometryType::DEFAULT_CRS.to_string(),
+        other => format!("SRID:{other}"),
+    }
+}
+
+/// The SRID of a spatial type's CRS, as [`spatial_crs`] names it, or of the 
SRID itself, as
+/// in a DDL string such as `geometry(4326)`, or `any`.
+fn spatial_srid(crs: &str) -> Option<i32> {
+    let crs = crs.trim();
+    if crs.eq_ignore_ascii_case("any") || 
crs.eq_ignore_ascii_case(SpatialType::MIXED_CRS) {
+        return Some(SpatialType::MIXED_SRID);
+    }
+    if crs.eq_ignore_ascii_case(GeometryType::DEFAULT_CRS) {
+        return Some(GeometryType::DEFAULT_SRID);
+    }
+    if crs.eq_ignore_ascii_case("EPSG:3857") {
+        return Some(3857);
+    }
+    let number = match crs.get(..5) {
+        Some(prefix) if prefix.eq_ignore_ascii_case("SRID:") => &crs[5..],
+        _ => crs,
+    };
+    number.parse().ok()
+}
+
 /// Main DDL parser: parses DDL strings into DataType.
 /// Handles primitive types, complex types (array, map, struct), and top-level 
schemas.
 fn parse_datatype_string(input: &str) -> Result<DataType> {
diff --git a/crates/spark-connect/tests/types_roundtrip.rs 
b/crates/spark-connect/tests/types_roundtrip.rs
index a0516fc..bffae04 100644
--- a/crates/spark-connect/tests/types_roundtrip.rs
+++ b/crates/spark-connect/tests/types_roundtrip.rs
@@ -108,15 +108,7 @@ fn proto_roundtrip_all_variants() {
 
 #[test]
 fn json_roundtrip_all_variants() {
-    // Geometry/Geography JSON is intentionally excluded: their json encoding 
is
-    // lossy today (emits a fixed CRS token rather than the numeric srid), so 
it does
-    // not round-trip. Their proto encoding round-trips and is covered above.
-    let is_geo = |d: &DataType| matches!(d, DataType::Geometry { .. } | 
DataType::Geography { .. });
-    for dt in all_scalar_variants()
-        .into_iter()
-        .chain(nested_variants())
-        .filter(|d| !is_geo(d))
-    {
+    for dt in all_scalar_variants().into_iter().chain(nested_variants()) {
         let value = dt.json_value();
         let s = dt.json();
         assert!(!s.is_empty());
@@ -347,3 +339,34 @@ fn timestamp_nanos_types() {
         DataType::TimestampLtzNanos { precision: 7 }
     );
 }
+
+/// Spatial types name their SRID by its CRS in JSON, as PySpark does, and 
read it back.
+#[test]
+fn spatial_json_names_the_crs_as_pyspark() {
+    for (dt, json) in [
+        (DataType::Geometry { srid: 4326 }, "\"geometry(OGC:CRS84)\""),
+        (DataType::Geometry { srid: 3857 }, "\"geometry(EPSG:3857)\""),
+        (DataType::Geometry { srid: 0 }, "\"geometry(SRID:0)\""),
+        (DataType::Geometry { srid: -1 }, "\"geometry(SRID:ANY)\""),
+        (
+            DataType::Geography { srid: 4326 },
+            "\"geography(OGC:CRS84, SPHERICAL)\"",
+        ),
+        (
+            DataType::Geography { srid: -1 },
+            "\"geography(SRID:ANY, SPHERICAL)\"",
+        ),
+    ] {
+        assert_eq!(dt.json(), json);
+        assert_eq!(DataType::from_json(&dt.json_value()).unwrap(), dt);
+    }
+    // The DDL forms name the SRID itself.
+    assert_eq!(
+        DataType::from_ddl("geometry(3857)").unwrap(),
+        DataType::Geometry { srid: 3857 }
+    );
+    assert_eq!(
+        DataType::from_ddl("geography(any)").unwrap(),
+        DataType::Geography { srid: -1 }
+    );
+}
diff --git a/python/tests/_mock_connect_server.py 
b/python/tests/_mock_connect_server.py
index 5e88ee3..ae573fa 100644
--- a/python/tests/_mock_connect_server.py
+++ b/python/tests/_mock_connect_server.py
@@ -101,8 +101,12 @@ def serve(connection):
         block_response = False
         response_gate = Event()
 
+        # Each AnalyzePlanRequest, recorded.
+        analyze_requests = []
+
         def analyze(self, request, context):
             self.calls += 1
+            self.analyze_requests.append(request)
             if self.block_response and not self.response_gate.wait(5):
                 context.abort(
                     grpc.StatusCode.DEADLINE_EXCEEDED,
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 682afbd..6137e18 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -258,6 +258,24 @@ def test_create_dataframe_infers_types_past_nulls(session, 
data, simple_string,
     assert [tuple(r) for r in df.collect()] == rows
 
 
[email protected]("ddl", [False, True], ids=["StructType", "DDL"])
+def test_reader_schema_keeps_field_metadata(session, tmp_path, ddl):
+    (tmp_path / "t.csv").write_text("1,x\n2,y\n")
+    path = str(tmp_path / "t.csv")
+    if ddl:
+        schema = "a long COMMENT 'the key', b string"
+    else:
+        schema = StructType(
+            [StructField("a", LongType(), True, {"comment": "the key"}), 
StructField("b", StringType())]
+        )
+    for df in (session.read.schema(schema).csv(path), session.read.csv(path, 
schema=schema)):
+        assert df.schema.fields[0].metadata == {"comment": "the key"}
+        assert [tuple(r) for r in df.collect()] == [(1, "x"), (2, "y")]
+    stream = session.readStream.schema(schema).csv(str(tmp_path))
+    assert stream.isStreaming
+    assert stream.schema.fields[0].metadata == {"comment": "the key"}
+
+
 def test_collect_reads_a_result_across_stream_ends(session):
     # Takes several response streams when the server's senderMaxStreamSize is 
small.
     n = 300_000
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index 01c2c44..68513c4 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -4609,3 +4609,43 @@ def 
test_merge_into_writer_mutates_and_returns_self(offline_spark):
     m.whenNotMatched().insertAll()
     m.withSchemaEvolution()
     assert m.whenMatched().updateAll() is m
+
+
+# --------------------------------------------------------------------------- 
reader schema
+
+def _sent_read_schema(request):
+    """The DataSource.schema of a captured AnalyzePlanRequest: schema=4 -> 
plan=1 -> root=1 ->
+    read=2 -> data_source=2 -> schema=2."""
+    message = parse_message(request)
+    for field in (4, 1, 1, 2, 2):
+        message = parse_message(message[field][0])
+    return message[2][0].decode()
+
+
[email protected]("read", [
+    lambda spark, schema: spark.read.schema(schema).csv("unused.csv"),
+    lambda spark, schema: spark.read.csv("unused.csv", schema=schema),
+    lambda spark, schema: spark.read.load("unused", format="json", 
schema=schema),
+    lambda spark, schema: spark.readStream.schema(schema).csv("unused"),
+    lambda spark, schema: spark.readStream.csv("unused", schema=schema),
+    lambda spark, schema: spark.readStream.load("unused", format="json", 
schema=schema),
+], ids=["read.schema", "read.csv", "read.load", "readStream.schema", 
"readStream.csv",
+        "readStream.load"])
+def test_reader_schema_is_sent_as_pyspark_sends_it(mock_connect, read):
+    from pyspark.errors import PySparkTypeError
+
+    spark, service = mock_connect
+    struct = T.StructType([T.StructField("a", T.LongType(), True, {"comment": 
"key"})])
+    # pyspark sends a StructType as its JSON and a string as is, for the 
server to parse.
+    for schema, sent in [
+        (struct, 
'{"fields":[{"metadata":{"comment":"key"},"name":"a","nullable":true,'
+                 '"type":"long"}],"type":"struct"}'),
+        ("a long COMMENT 'key'", "a long COMMENT 'key'"),
+    ]:
+        read(spark, schema).columns
+        assert _sent_read_schema(service.analyze_requests[-1]) == sent
+    with pytest.raises(PySparkTypeError) as error:
+        read(spark, T.LongType())
+    assert error.value.getCondition() == "NOT_EXPECTED_TYPE"
+    assert error.value.getMessageParameters() == {
+        "arg_name": "schema", "expected_type": "str or struct type", 
"arg_type": "LongType"}


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to