This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new a8d6c76 [SPARK-60907][CONNECT] Send reader schemas as PySpark does,
and accept a StructType in schema()
a8d6c76 is described below
commit a8d6c76a4da1f9f90ae07130a4e8542781e192b3
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Thu Nov 5 19:05:57 2026 +0900
[SPARK-60907][CONNECT] Send reader schemas as PySpark does, and accept a
StructType in schema()
### What changes were proposed in this pull request?
[SPARK-60907](https://issues.apache.org/jira/browse/SPARK-60907): reader
schemas of the Rust-backed Python client are now sent as PySpark Connect 4.3.0
sends them.
- `DataFrameReader.schema` and `DataStreamReader.schema` accept a
StructType as well as a string.
- As in PySpark, a StructType is sent as its JSON and a string as is, for
the server to parse. Anything else raises `NOT_EXPECTED_TYPE` with PySpark's
parameters.
- The `schema` arguments of `load()`, `csv()`, `json()` and the other
format methods, batch and streaming, go through the same conversion, as they go
through `schema()` in PySpark. Before, they were parsed on the client and sent
as a simple string.
- The JSON of a spatial type now names its SRID's coordinate reference
system, as PySpark's does: `geometry(OGC:CRS84)` for 4326,
`geometry(EPSG:3857)`, `geometry(SRID:0)` and `geometry(SRID:ANY)`, and
`geography(OGC:CRS84, SPHERICAL)`. It wrote `OGC:CRS84` for any SRID, so that
`GeometryType(3857)` in a reader schema would reach the server as 4326. The
JSON parser reads these names back, so spatial types now round-trip through
JSON.
### Why are the changes needed?
`spark.read.schema(struct_type)` and `spark.readStream.schema(struct_type)`
raised `TypeError`. The format methods' `schema` arguments lost what the simple
string cannot express: a StructType with field metadata, such as a comment,
failed on the server with `PARSE_SYNTAX_ERROR`, and a DDL string with `COMMENT`
failed in the client parser with `ValueError`. Data types other than a
StructType were accepted, while PySpark rejects them.
### Does this PR introduce _any_ user-facing change?
Yes, these now match PySpark Connect 4.3.0:
- `spark.read.schema(struct_type)` and
`spark.readStream.schema(struct_type)` work.
- A schema with field metadata, given as a StructType or as DDL with
`COMMENT`, now works and keeps the metadata, through `schema()` and through the
`schema` arguments.
- A data type other than a StructType raises `NOT_EXPECTED_TYPE` (a
`PySparkTypeError`).
### How was this patch tested?
- Against a local Spark 4.3.0 server, a probe of seven reader calls, with a
StructType carrying field metadata, DDL with `COMMENT`, and a `LongType`,
through `read.schema()`, `read.csv(schema=...)` and `readStream.schema()`,
gives output identical to official PySpark 4.3.0: the resulting field metadata
and rows, or the error and its parameters.
- New offline tests check, over the mock server, the schema that both
readers send for `schema()` and the `schema` arguments of `load()` and `csv()`:
the JSON of a StructType, as official PySpark 4.3.0 writes it, or a string as
is. They also check `NOT_EXPECTED_TYPE` for anything else. Against master they
fail, where the StructType was sent as `struct<a:bigint>` or raised `TypeError`.
- New unit tests check the JSON of the spatial types, as official PySpark
4.3.0 writes it, and that it reads back. The JSON round-trip test now covers
the spatial types too. Against a local server, a reader schema with
`GeometryType(3857)`, `GeometryType(4326)`, `GeometryType(0)` or
`GeographyType(4326)` reaches the server with that SRID, as with official
PySpark.
- A new live test reads a CSV file through `read.schema()`,
`read.csv(schema=...)` and `readStream.schema()`, with both a StructType and
DDL, and checks the field metadata and rows; it also passes with official
PySpark 4.3.0.
- The lib tests and every integration test, e2e included, against the local
server (629 passed), the build of all tests with `wasm-udf`, the offline Python
suites (426 passed, 8 live cases skipped), and the live drop-in and Arrow
suites (23 passed) and live cases (9 passed). Clippy reports no diagnostics in
the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #151 from HyukjinKwon/SPARK-60907-reader-schema.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/readwriter.rs | 44 ++++++++++---
crates/pyspark-rs/src/streaming.rs | 16 +++--
crates/spark-connect/src/types.rs | 90 ++++++++++++++++-----------
crates/spark-connect/tests/types_roundtrip.rs | 41 +++++++++---
python/tests/_mock_connect_server.py | 4 ++
python/tests/test_dropin_e2e.py | 18 ++++++
python/tests/test_dropin_offline.py | 40 ++++++++++++
7 files changed, 193 insertions(+), 60 deletions(-)
diff --git a/crates/pyspark-rs/src/readwriter.rs
b/crates/pyspark-rs/src/readwriter.rs
index 3c0db1b..217034a 100644
--- a/crates/pyspark-rs/src/readwriter.rs
+++ b/crates/pyspark-rs/src/readwriter.rs
@@ -62,11 +62,35 @@ fn set_opt(
}
}
-/// Resolve a reader `schema` arg (a StructType/DataType or a DDL string) to
the
-/// reader's DDL schema.
+/// A reader `schema` as pyspark's `DataFrameReader.schema` and
`DataStreamReader.schema`
+/// send it: a StructType as its JSON, which keeps field metadata, and a
string as is, for
+/// the server to parse. Anything else raises `NOT_EXPECTED_TYPE`.
+pub(crate) fn reader_schema(schema: &Bound<'_, PyAny>) -> PyResult<String> {
+ if let Ok(schema) = schema.extract::<String>() {
+ return Ok(schema);
+ }
+ if let Ok(st) = schema.extract::<PyRef<crate::types::PyStructType>>() {
+ return Ok(spark_connect::types::DataType::Struct {
+ fields: st.fields.clone(),
+ }
+ .json());
+ }
+ let arg_type = schema.get_type().name()?.to_string();
+ Err(crate::column::raise_pyspark(
+ schema.py(),
+ "PySparkTypeError",
+ "NOT_EXPECTED_TYPE",
+ &[
+ ("arg_name", "schema"),
+ ("expected_type", "str or struct type"),
+ ("arg_type", &arg_type),
+ ],
+ ))
+}
+
+/// Apply the `schema` argument of a format method, as pyspark does through
`schema()`.
fn apply_reader_schema(r: DataFrameReader, v: &Bound<'_, PyAny>) ->
PyResult<DataFrameReader> {
- let dt = crate::types::py_to_data_type(v)?;
- Ok(r.schema(dt.simple_string()))
+ Ok(r.schema(reader_schema(v)?))
}
#[pymethods]
@@ -77,10 +101,14 @@ impl PyDataFrameReader {
slf
}
- /// Set the schema (a DDL string).
- fn schema<'py>(mut slf: PyRefMut<'py, Self>, schema: &str) ->
PyRefMut<'py, Self> {
- slf.inner = slf.inner.clone().schema(schema.to_string());
- slf
+ /// Set the schema: a StructType or a DDL string, as in pyspark.
+ fn schema<'py>(
+ mut slf: PyRefMut<'py, Self>,
+ schema: &Bound<'_, PyAny>,
+ ) -> PyResult<PyRefMut<'py, Self>> {
+ let schema = reader_schema(schema)?;
+ slf.inner = slf.inner.clone().schema(schema);
+ Ok(slf)
}
/// Set a single read option. `None` leaves the option unset (not the
string
diff --git a/crates/pyspark-rs/src/streaming.rs
b/crates/pyspark-rs/src/streaming.rs
index 88d3baf..ef080b1 100644
--- a/crates/pyspark-rs/src/streaming.rs
+++ b/crates/pyspark-rs/src/streaming.rs
@@ -57,11 +57,10 @@ fn set_ropt(
}
}
-/// Apply a `schema` argument (a DDL string or a StructType) to a streaming
reader,
-/// mirroring the batch `apply_reader_schema`.
+/// Apply a `schema` argument (a StructType or a DDL string) to a streaming
reader, as the
+/// batch reader does.
fn apply_stream_schema(r: DataStreamReader, v: &Bound<'_, PyAny>) ->
PyResult<DataStreamReader> {
- let dt = crate::types::py_to_data_type(v)?;
- Ok(r.schema(dt.simple_string()))
+ Ok(r.schema(crate::readwriter::reader_schema(v)?))
}
/// Coerce a `partitionBy` argument (a single column name or a list of names)
to a
@@ -96,8 +95,13 @@ impl PyDataStreamReader {
Ok(slf)
}
- fn schema<'py>(mut slf: PyRefMut<'py, Self>, schema: &str) ->
PyResult<PyRefMut<'py, Self>> {
- slf.inner = slf.inner.clone().schema(schema);
+ /// Set the schema: a StructType or a DDL string, as in pyspark.
+ fn schema<'py>(
+ mut slf: PyRefMut<'py, Self>,
+ schema: &Bound<'_, PyAny>,
+ ) -> PyResult<PyRefMut<'py, Self>> {
+ let schema = crate::readwriter::reader_schema(schema)?;
+ slf.inner = slf.inner.clone().schema(&schema);
Ok(slf)
}
diff --git a/crates/spark-connect/src/types.rs
b/crates/spark-connect/src/types.rs
index 976520d..fb026fd 100644
--- a/crates/spark-connect/src/types.rs
+++ b/crates/spark-connect/src/types.rs
@@ -434,19 +434,13 @@ impl DataType {
}
DataType::Variant => serde_json::json!("variant"),
DataType::Geometry { srid } => {
- if *srid == -1 {
- serde_json::json!("geometry(SRID:ANY)")
- } else {
- serde_json::json!(format!("geometry(OGC:CRS84)"))
- }
- }
- DataType::Geography { srid } => {
- if *srid == -1 {
- serde_json::json!("geography(SRID:ANY, SPHERICAL)")
- } else {
- serde_json::json!(format!("geography(OGC:CRS84,
SPHERICAL)"))
- }
+ serde_json::json!(format!("geometry({})", spatial_crs(*srid)))
}
+ DataType::Geography { srid } => serde_json::json!(format!(
+ "geography({}, {})",
+ spatial_crs(*srid),
+ GeographyType::DEFAULT_ALG
+ )),
DataType::Udt {
type_str: _,
jvm_class,
@@ -1456,16 +1450,12 @@ fn parse_geometry(s: &str) -> Result<DataType> {
)
})?;
- let srid = if inner.to_lowercase() == "any" {
- -1
- } else {
- inner.parse::<i32>().map_err(|_| {
- SparkError::value(
- "INVALID_DATATYPE_FORMAT",
- &[("detail", "Invalid geometry srid")],
- )
- })?
- };
+ let srid = spatial_srid(inner).ok_or_else(|| {
+ SparkError::value(
+ "INVALID_DATATYPE_FORMAT",
+ &[("detail", "Invalid geometry srid")],
+ )
+ })?;
Ok(DataType::Geometry { srid })
}
@@ -1481,25 +1471,51 @@ fn parse_geography(s: &str) -> Result<DataType> {
)
})?;
- let srid = if inner.to_lowercase() == "any" {
- -1
- } else {
- inner
- .split(',')
- .next()
- .unwrap_or("4326")
- .parse::<i32>()
- .map_err(|_| {
- SparkError::value(
- "INVALID_DATATYPE_FORMAT",
- &[("detail", "Invalid geography srid")],
- )
- })?
- };
+ // The JSON form also names the algorithm, as in `OGC:CRS84, SPHERICAL`.
+ let crs = inner.split(',').next().unwrap_or_default();
+ let srid = spatial_srid(crs).ok_or_else(|| {
+ SparkError::value(
+ "INVALID_DATATYPE_FORMAT",
+ &[("detail", "Invalid geography srid")],
+ )
+ })?;
Ok(DataType::Geography { srid })
}
+/// The coordinate reference system that a spatial type's SRID is named by in
its JSON, as
+/// PySpark's `geo_utils` names the SRIDs that it supports: `SRID:0`,
`EPSG:3857` and
+/// `OGC:CRS84` for 4326, and `SRID:ANY` for a mixed SRID. Another SRID, which
PySpark does not
+/// accept, is named `SRID:<srid>`.
+fn spatial_crs(srid: i32) -> String {
+ match srid {
+ SpatialType::MIXED_SRID => SpatialType::MIXED_CRS.to_string(),
+ 3857 => "EPSG:3857".to_string(),
+ GeometryType::DEFAULT_SRID => GeometryType::DEFAULT_CRS.to_string(),
+ other => format!("SRID:{other}"),
+ }
+}
+
+/// The SRID of a spatial type's CRS, as [`spatial_crs`] names it, or of the
SRID itself, as
+/// in a DDL string such as `geometry(4326)`, or `any`.
+fn spatial_srid(crs: &str) -> Option<i32> {
+ let crs = crs.trim();
+ if crs.eq_ignore_ascii_case("any") ||
crs.eq_ignore_ascii_case(SpatialType::MIXED_CRS) {
+ return Some(SpatialType::MIXED_SRID);
+ }
+ if crs.eq_ignore_ascii_case(GeometryType::DEFAULT_CRS) {
+ return Some(GeometryType::DEFAULT_SRID);
+ }
+ if crs.eq_ignore_ascii_case("EPSG:3857") {
+ return Some(3857);
+ }
+ let number = match crs.get(..5) {
+ Some(prefix) if prefix.eq_ignore_ascii_case("SRID:") => &crs[5..],
+ _ => crs,
+ };
+ number.parse().ok()
+}
+
/// Main DDL parser: parses DDL strings into DataType.
/// Handles primitive types, complex types (array, map, struct), and top-level
schemas.
fn parse_datatype_string(input: &str) -> Result<DataType> {
diff --git a/crates/spark-connect/tests/types_roundtrip.rs
b/crates/spark-connect/tests/types_roundtrip.rs
index a0516fc..bffae04 100644
--- a/crates/spark-connect/tests/types_roundtrip.rs
+++ b/crates/spark-connect/tests/types_roundtrip.rs
@@ -108,15 +108,7 @@ fn proto_roundtrip_all_variants() {
#[test]
fn json_roundtrip_all_variants() {
- // Geometry/Geography JSON is intentionally excluded: their json encoding
is
- // lossy today (emits a fixed CRS token rather than the numeric srid), so
it does
- // not round-trip. Their proto encoding round-trips and is covered above.
- let is_geo = |d: &DataType| matches!(d, DataType::Geometry { .. } |
DataType::Geography { .. });
- for dt in all_scalar_variants()
- .into_iter()
- .chain(nested_variants())
- .filter(|d| !is_geo(d))
- {
+ for dt in all_scalar_variants().into_iter().chain(nested_variants()) {
let value = dt.json_value();
let s = dt.json();
assert!(!s.is_empty());
@@ -347,3 +339,34 @@ fn timestamp_nanos_types() {
DataType::TimestampLtzNanos { precision: 7 }
);
}
+
+/// Spatial types name their SRID by its CRS in JSON, as PySpark does, and
read it back.
+#[test]
+fn spatial_json_names_the_crs_as_pyspark() {
+ for (dt, json) in [
+ (DataType::Geometry { srid: 4326 }, "\"geometry(OGC:CRS84)\""),
+ (DataType::Geometry { srid: 3857 }, "\"geometry(EPSG:3857)\""),
+ (DataType::Geometry { srid: 0 }, "\"geometry(SRID:0)\""),
+ (DataType::Geometry { srid: -1 }, "\"geometry(SRID:ANY)\""),
+ (
+ DataType::Geography { srid: 4326 },
+ "\"geography(OGC:CRS84, SPHERICAL)\"",
+ ),
+ (
+ DataType::Geography { srid: -1 },
+ "\"geography(SRID:ANY, SPHERICAL)\"",
+ ),
+ ] {
+ assert_eq!(dt.json(), json);
+ assert_eq!(DataType::from_json(&dt.json_value()).unwrap(), dt);
+ }
+ // The DDL forms name the SRID itself.
+ assert_eq!(
+ DataType::from_ddl("geometry(3857)").unwrap(),
+ DataType::Geometry { srid: 3857 }
+ );
+ assert_eq!(
+ DataType::from_ddl("geography(any)").unwrap(),
+ DataType::Geography { srid: -1 }
+ );
+}
diff --git a/python/tests/_mock_connect_server.py
b/python/tests/_mock_connect_server.py
index 5e88ee3..ae573fa 100644
--- a/python/tests/_mock_connect_server.py
+++ b/python/tests/_mock_connect_server.py
@@ -101,8 +101,12 @@ def serve(connection):
block_response = False
response_gate = Event()
+ # Each AnalyzePlanRequest, recorded.
+ analyze_requests = []
+
def analyze(self, request, context):
self.calls += 1
+ self.analyze_requests.append(request)
if self.block_response and not self.response_gate.wait(5):
context.abort(
grpc.StatusCode.DEADLINE_EXCEEDED,
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 682afbd..6137e18 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -258,6 +258,24 @@ def test_create_dataframe_infers_types_past_nulls(session,
data, simple_string,
assert [tuple(r) for r in df.collect()] == rows
[email protected]("ddl", [False, True], ids=["StructType", "DDL"])
+def test_reader_schema_keeps_field_metadata(session, tmp_path, ddl):
+ (tmp_path / "t.csv").write_text("1,x\n2,y\n")
+ path = str(tmp_path / "t.csv")
+ if ddl:
+ schema = "a long COMMENT 'the key', b string"
+ else:
+ schema = StructType(
+ [StructField("a", LongType(), True, {"comment": "the key"}),
StructField("b", StringType())]
+ )
+ for df in (session.read.schema(schema).csv(path), session.read.csv(path,
schema=schema)):
+ assert df.schema.fields[0].metadata == {"comment": "the key"}
+ assert [tuple(r) for r in df.collect()] == [(1, "x"), (2, "y")]
+ stream = session.readStream.schema(schema).csv(str(tmp_path))
+ assert stream.isStreaming
+ assert stream.schema.fields[0].metadata == {"comment": "the key"}
+
+
def test_collect_reads_a_result_across_stream_ends(session):
# Takes several response streams when the server's senderMaxStreamSize is
small.
n = 300_000
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index 01c2c44..68513c4 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -4609,3 +4609,43 @@ def
test_merge_into_writer_mutates_and_returns_self(offline_spark):
m.whenNotMatched().insertAll()
m.withSchemaEvolution()
assert m.whenMatched().updateAll() is m
+
+
+# ---------------------------------------------------------------------------
reader schema
+
+def _sent_read_schema(request):
+ """The DataSource.schema of a captured AnalyzePlanRequest: schema=4 ->
plan=1 -> root=1 ->
+ read=2 -> data_source=2 -> schema=2."""
+ message = parse_message(request)
+ for field in (4, 1, 1, 2, 2):
+ message = parse_message(message[field][0])
+ return message[2][0].decode()
+
+
[email protected]("read", [
+ lambda spark, schema: spark.read.schema(schema).csv("unused.csv"),
+ lambda spark, schema: spark.read.csv("unused.csv", schema=schema),
+ lambda spark, schema: spark.read.load("unused", format="json",
schema=schema),
+ lambda spark, schema: spark.readStream.schema(schema).csv("unused"),
+ lambda spark, schema: spark.readStream.csv("unused", schema=schema),
+ lambda spark, schema: spark.readStream.load("unused", format="json",
schema=schema),
+], ids=["read.schema", "read.csv", "read.load", "readStream.schema",
"readStream.csv",
+ "readStream.load"])
+def test_reader_schema_is_sent_as_pyspark_sends_it(mock_connect, read):
+ from pyspark.errors import PySparkTypeError
+
+ spark, service = mock_connect
+ struct = T.StructType([T.StructField("a", T.LongType(), True, {"comment":
"key"})])
+ # pyspark sends a StructType as its JSON and a string as is, for the
server to parse.
+ for schema, sent in [
+ (struct,
'{"fields":[{"metadata":{"comment":"key"},"name":"a","nullable":true,'
+ '"type":"long"}],"type":"struct"}'),
+ ("a long COMMENT 'key'", "a long COMMENT 'key'"),
+ ]:
+ read(spark, schema).columns
+ assert _sent_read_schema(service.analyze_requests[-1]) == sent
+ with pytest.raises(PySparkTypeError) as error:
+ read(spark, T.LongType())
+ assert error.value.getCondition() == "NOT_EXPECTED_TYPE"
+ assert error.value.getMessageParameters() == {
+ "arg_name": "schema", "expected_type": "str or struct type",
"arg_type": "LongType"}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]