HyukjinKwon opened a new pull request, #237: URL: https://github.com/apache/spark-connect-rust/pull/237
### What changes were proposed in this pull request? [SPARK-61040](https://issues.apache.org/jira/browse/SPARK-61040): `DataType.fromDDL` and the DDL return types of UDFs are now parsed on the server, as PySpark Connect does. `DataType.fromDDL(ddl)` and `StructType.fromDDL` now parse with the new `pyspark.sql.types._parse_datatype_string(s)`. That asks the server of the active session to parse the string (the new `SparkSession._parse_ddl`, the `DDLParse` of an `AnalyzePlanRequest`), as PySpark Connect's do. So does a UDF of a DDL return type, as `udf(f, "a INT NOT NULL")`, whose `returnType` resolves through `DataType.fromDDL`. They take any DDL that Spark does: - `NOT NULL` and `COMMENT` clauses - nested types, such as `"st struct<a:int,b:double>"` - collations, such as `"s STRING COLLATE UTF8_LCASE"` They raise the errors of Spark's parser, such as `PARSE_SYNTAX_ERROR` for `"a INT,"` and `UNSUPPORTED_DATATYPE` for an unknown type, and `NO_ACTIVE_OR_DEFAULT_SESSION` without a session. This PR is on top of #235 ([SPARK-61029](https://issues.apache.org/jira/browse/SPARK-61029)), whose `DDLParse` requests and mock servers this change uses. It includes that commit until #235 merges, and its own change is the last commit. ### Why are the changes needed? The client parsed them on the client: - It raised `ValueError: [INVALID_DATATYPE_FORMAT]` for `NOT NULL` and nested types. - It took `"s STRING COLLATE UTF8_LCASE"` as a string of the collation `collate utf8_lcase`, and `"a INT,"` as `a INT`. - It had no `pyspark.sql.types._parse_datatype_string`, which PySpark code imports. ### Does this PR introduce _any_ user-facing change? Yes. These parse the DDL of PySpark 4.3.0, and raise its errors. ### How was this patch tested? - A new live test parses DDL strings with `DataType.fromDDL`, `StructType.fromDDL` and `_parse_datatype_string`, runs a UDF of a nested DDL return type, and checks the errors. It passes with official PySpark 4.3.0, and fails without this change. - On a local server, 12 cases of these behave as in official PySpark 4.3.0. - The lib tests of both crates (571 passed), the build of all tests, also with `wasm-udf`, the offline Python suites (1810 passed, 8 skipped), and the live drop-in and Arrow suites (110 passed) and live cases (10 passed) against a local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
