HyukjinKwon opened a new pull request, #275: URL: https://github.com/apache/spark-connect-rust/pull/275
### What changes were proposed in this pull request? [SPARK-61072](https://issues.apache.org/jira/browse/SPARK-61072): `StructType.toDDL()` now asks the server to convert the schema to DDL, as PySpark Connect does. PySpark Connect sends the JSON of the schema to the server of the active session, as the `JsonToDDL` of an `AnalyzePlanRequest`. So `toDDL()` returns the DDL of Spark, which quotes the names that need it and parses back: ``` a INT,`b b` STRING NOT NULL,m MAP<STRING, STRUCT<x: DECIMAL(10,2)>>,c STRING COMMENT 'the c' ``` Without an active session, it raises an `AssertionError`, as PySpark's does. The core adds `SparkSession::json_to_ddl`, and the session of the Python client adds `_to_ddl`, as PySpark Connect's. ### Why are the changes needed? The client wrote the DDL itself, as `a int,b b string NOT NULL,...`. It did not quote names, so the DDL of a name with a space did not parse back. It also wrote the types in lowercase, unlike Spark. ### Does this PR introduce _any_ user-facing change? Yes. `StructType.toDDL()` returns the DDL of PySpark 4.3.0. ### How was this patch tested? - A new live test checks the DDL of a schema with a quoted name, `NOT NULL`, a nested map and struct, and a `COMMENT`, and parses it back with `createDataFrame`. It passes with official PySpark 4.3.0 and fails without this change. - A new offline test checks the JSON that `toDDL()` sends to a mock Connect server, and the error without an active session. The mock server now answers `JsonToDDL` requests. Two older offline tests checked the DDL that the client wrote, without a session; their `toDDL` checks are removed. - A new e2e test checks `SparkSession::json_to_ddl` against the server. - On a local server, 11 schemas give the same DDL as official PySpark 4.3.0, including collations, char and varchar, intervals, variant, time, and an empty schema. - The lib tests of both crates (589 passed), the build of all tests, also with `wasm-udf`, the offline Python suites (1978 passed, 8 skipped), and the live drop-in and Arrow suites (134 passed) and live cases (10 passed) against a local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
