HyukjinKwon opened a new pull request, #235: URL: https://github.com/apache/spark-connect-rust/pull/235
### What changes were proposed in this pull request? [SPARK-61029](https://issues.apache.org/jira/browse/SPARK-61029): the Python client now parses the DDL string schemas of `createDataFrame` and of the map functions on the server, as PySpark Connect does. PySpark Connect parses a DDL string with `_parse_ddl`, the `DDLParse` of an `AnalyzePlanRequest`. `spark.createDataFrame(data, schema)`, `mapInPandas`, `mapInArrow`, and the `applyInPandas` and `applyInArrow` of grouped and cogrouped data parse their schemas that way. So they take any DDL that Spark does: - nested types, such as `"st struct<a:int,b:double>"` and `"m map<string,struct<x:int>>"` - backtick-quoted names, such as ``"`a b` int"`` - collations, such as `"s STRING COLLATE UTF8_LCASE"` - `NOT NULL` and `COMMENT` clauses They also raise the errors of Spark's parser, such as `PARSE_SYNTAX_ERROR` for `"a INT,"`. `transformWithStateInPandas` sends a DDL output schema as an `UnparsedDataType`, for the server to parse. The core adds `SparkSession::parse_ddl`, which asks the server to parse a DDL string, and `spark_session()` accessors for `GroupedData` and `CoGroupedData`. ### Why are the changes needed? The client parsed these schemas itself, and its parser differs from Spark's: - It raised `ValueError: invalid type string 'st struct<a:int,b:double>': SparkError { kind: ValueError, error_class: "INVALID_DATATYPE_FORMAT", ... }` for the nested, quoted, `NOT NULL` and `COMMENT` forms above. - It took `"a INT,"` as `a INT`. - For `"s STRING COLLATE UTF8_LCASE"`, it sent a type that the server failed to parse (`PARSE_SYNTAX_ERROR`). ### Does this PR introduce _any_ user-facing change? Yes. These functions take the DDL schemas of PySpark 4.3.0 and raise its errors. ### How was this patch tested? - A new live test creates DataFrames of these schemas and checks their types and rows, the collation, the errors, and the `NOT NULL` and `COMMENT` of a `mapInPandas` schema. It passes with official PySpark 4.3.0 and fails without this change. - A new e2e test checks `SparkSession::parse_ddl` against the server, including its errors. - A new offline test checks the DDL strings that `createDataFrame` and the map functions send to a mock Connect server. The mock servers of the offline and the Arrow tests now answer `DDLParse` requests. - On a local server, 59 DDL schemas give the same types and errors as official PySpark 4.3.0 for `createDataFrame`. So do 36 cases of the map and apply functions. - The lib tests of both crates (571 passed), the build of all tests, also with `wasm-udf`, the offline Python suites (1810 passed, 8 skipped), and the live drop-in and Arrow suites (109 passed) and live cases (10 passed) against a local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
