HyukjinKwon opened a new pull request, #151: URL: https://github.com/apache/spark-connect-rust/pull/151
### What changes were proposed in this pull request? [SPARK-60907](https://issues.apache.org/jira/browse/SPARK-60907): reader schemas of the Rust-backed Python client are now sent as PySpark Connect 4.3.0 sends them. - `DataFrameReader.schema` and `DataStreamReader.schema` accept a StructType as well as a string. - As in PySpark, a StructType is sent as its JSON and a string as is, for the server to parse. Anything else raises `NOT_EXPECTED_TYPE` with PySpark's parameters. - The `schema` arguments of `load()`, `csv()`, `json()` and the other format methods, batch and streaming, go through the same conversion, as they go through `schema()` in PySpark. Before, they were parsed on the client and sent as a simple string. ### Why are the changes needed? `spark.read.schema(struct_type)` and `spark.readStream.schema(struct_type)` raised `TypeError`. The format methods' `schema` arguments lost what the simple string cannot express: a StructType with field metadata, such as a comment, failed on the server with `PARSE_SYNTAX_ERROR`, and a DDL string with `COMMENT` failed in the client parser with `ValueError`. Data types other than a StructType were accepted, while PySpark rejects them. ### Does this PR introduce _any_ user-facing change? Yes, these now match PySpark Connect 4.3.0: - `spark.read.schema(struct_type)` and `spark.readStream.schema(struct_type)` work. - A schema with field metadata, given as a StructType or as DDL with `COMMENT`, now works and keeps the metadata, through `schema()` and through the `schema` arguments. - A data type other than a StructType raises `NOT_EXPECTED_TYPE` (a `PySparkTypeError`). ### How was this patch tested? - Against a local Spark 4.3.0 server, a probe of seven reader calls, with a StructType carrying field metadata, DDL with `COMMENT`, and a `LongType`, through `read.schema()`, `read.csv(schema=...)` and `readStream.schema()`, gives output identical to official PySpark 4.3.0: the resulting field metadata and rows, or the error and its parameters. - New offline tests check that both readers' `schema()` and the `schema` arguments of `load()` and `csv()` take a StructType or a string, and raise `NOT_EXPECTED_TYPE` otherwise. - A new live test reads a CSV file through `read.schema()`, `read.csv(schema=...)` and `readStream.schema()`, with both a StructType and DDL, and checks the field metadata and rows; it also passes with official PySpark 4.3.0. - The offline Python suites (426 passed, 8 live cases skipped), and the live drop-in and Arrow suites (23 passed) and live cases (9 passed). Clippy reports no diagnostics in the changed files, as on master. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
