HyukjinKwon opened a new pull request, #150: URL: https://github.com/apache/spark-connect-rust/pull/150
### What changes were proposed in this pull request? [SPARK-60906](https://issues.apache.org/jira/browse/SPARK-60906): `str()`, `repr()`, and `fromDDL` of the Rust-backed Python client's data types now behave as in PySpark Connect 4.3.0. - The base `DataType` no longer defines `__str__`, which returned the simple string, so `str()` of a data type is its `repr()`, as in PySpark, where data types define only `__repr__`. - The base `DataType.__repr__` is the class name, as in PySpark. It applies to a bare `DataType()` and to pure-Python subclasses, such as a `UserDefinedType`, that do not define their own. The concrete types keep their `__repr__`, which already matched PySpark. - `DataType.fromDDL`, and the same classmethod on every type, returns the concrete type (for example a `StructType` or a `DecimalType`) instead of a bare `DataType`. ### Why are the changes needed? `str(LongType())` was `bigint` rather than `LongType()`, so `str()` of every data type differed from PySpark, as did error messages that format a type with `str()`, such as those of `UNSUPPORTED_DATA_TYPE_FOR_ARROW` and the UDF return-type checks. `repr()` of a bare `DataType()` or of a `UserDefinedType` was `void`, and `fromDDL` returned an object whose type was the base class. ### Does this PR introduce _any_ user-facing change? Yes, these now match PySpark Connect 4.3.0: - `str(LongType())` is `LongType()` (was `bigint`), and `str()` of a `StructType` is its `repr()`, such as `StructType([StructField('a', LongType(), True)])` (was `struct<a:bigint>`). `simpleString()` is unchanged. - `repr(DataType())` is `DataType()` and `repr(ExamplePointUDT())` is `ExamplePointUDT()` (both were `void`). - `DataType.fromDDL("b string, a int")` returns a `StructType`, and `DataType.fromDDL("decimal(10,10)")` a `DecimalType(10,10)`, as in PySpark's examples. ### How was this patch tested? - `repr()` and `str()` of 30 data types, covering every atomic, interval, decimal, char, varchar, array, map and struct type, a `StructField`, a bare `DataType()` and a `UserDefinedType`, are identical to official PySpark 4.3.0. - New offline tests check `str()` and `repr()` against PySpark's strings, the `UserDefinedType` repr, and `fromDDL` against the examples in PySpark's docstring. The `str()` and `repr()` tests also pass with official PySpark 4.3.0. The existing `StructType.add` regression now reads the base type through `DataType.simpleString`, which is what it checked through `str()` before. - The offline Python suites (437 passed, 8 live cases skipped), and the live drop-in and Arrow suites (21 passed) and live cases (9 passed) against a local Spark 4.3.0 server. The Rust layer never formats a data type with `str()`; clippy reports no diagnostics in `types.rs`, before or after. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
