HyukjinKwon opened a new pull request, #298: URL: https://github.com/apache/spark-connect-rust/pull/298
### What changes were proposed in this pull request? [SPARK-61094](https://issues.apache.org/jira/browse/SPARK-61094): `pyspark.sql.connect.types.UnparsedDataType` is added, and the `returnType` of a UDTF given as a DDL string is one, as in PySpark Connect. As PySpark Connect's: - `UnparsedDataType` is a DDL string of a type that the server parses: - its `simpleString` is `unparsed('<ddl>')` and its repr `UnparsedDataType('<ddl>')` - its `jsonValue`, `json`, `needConversion`, `toInternal` and `fromInternal` raise `INVALID_CALL_ON_UNRESOLVED_OBJECT` - the `returnType` of a UDTF given as a DDL string is an `UnparsedDataType` - `spark.dataSource.register` of a class without a `name` raises the `AttributeError` of `name()` `pyspark.sql.connect.types` is a shim module with PySpark's class. Because the drop-in's `DataType` pickles through its JSON, which an unparsed type refuses, this class pickles from its DDL string instead. The client converts it to and from the core's unparsed type. The `json()` of a pure-Python subclass of `DataType` is now the `json.dumps` of its `jsonValue`, as PySpark's `DataType.json` is. ### Why are the changes needed? The client: - lacked `pyspark.sql.connect.types`, so `from pyspark.sql.connect.types import UnparsedDataType` raised an `ImportError` - kept a DDL string as the `returnType` of a UDTF - turned an unparsed type into a bare `DataType` - raised a `TypeError` ("DataSource class must have a classmethod name()") from `register` ### Does this PR introduce _any_ user-facing change? Yes. `UnparsedDataType` and the `returnType` of a UDTF are those of PySpark 4.3.0. ### How was this patch tested? - A new live test checks the `UnparsedDataType` `returnType` of a UDTF of a DDL string, and the rows that the server returns for it. It passes with official PySpark 4.3.0 and fails without this change. - A new offline test checks the repr, the `simpleString`, the errors and the pickling of `UnparsedDataType`, the `returnType` of a UDTF, and the error of `register`. Two older offline tests now expect an `UnparsedDataType` where they expected the DDL string or a bare `DataType`. - 17 cases of `UnparsedDataType`, a UDTF and a data source give the same results as official PySpark 4.3.0, except `typeName`, which SPARK-61080 (#281) makes the class's (`unparseddata`). - The lib tests of both crates (595 passed), the build of all tests, also with `wasm-udf`, the offline Python suites (2047 passed, 12 skipped), and the live drop-in and Arrow suites (151 passed) and live cases (10 passed) against a local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
