HyukjinKwon opened a new pull request, #276: URL: https://github.com/apache/spark-connect-rust/pull/276
### What changes were proposed in this pull request? [SPARK-61074](https://issues.apache.org/jira/browse/SPARK-61074): the JSON of a struct field now holds the collations of its strings in its metadata, and the `fromJson` methods read them back, as PySpark's do. As Spark and PySpark do: - `StructType.json()` and `jsonValue()` write the collations of the strings of a field, also those in its arrays and maps, in the `__COLLATIONS` of its metadata. Each is `<provider>.<collation>`, keyed by its path: `s`, `a.element`, `m.key`, `m.value.element`. The type of the field is written without them, for example `"string"`. - `StructField.fromJson`, `StructType.fromJson` and `_parse_datatype_json_value` read them back. So do `ArrayType.fromJson` and `MapType.fromJson`, which use their `fieldPath` and `collationsMap`. They raise `COLLATION_INVALID_PROVIDER` for a provider that is neither `spark` nor `icu`. They raise `INVALID_JSON_DATA_TYPE_FOR_COLLATIONS` for a collation of a field that is not a string. - The dicts of `jsonValue()` have their keys in PySpark's order: `name`, `type`, `nullable`, `metadata` for a field, and `type` first for a nested type. The `fromJson` methods now call the helpers of the shim `pyspark.sql.types`, which follow PySpark's code. So they also read the struct of a user-defined type. The core reads the collations of the JSON of a struct field, and the JSON of a UDT, which it can write but could not read back. This PR is on top of #271 ([SPARK-61073](https://issues.apache.org/jira/browse/SPARK-61073)), as both change `_parse_datatype_json_value`. It includes that commit until it merges, and its own change is the last commit. ### Why are the changes needed? The client wrote `"string collate UTF8_LCASE"` as the type of a field, so the JSON of a schema differed from PySpark's: ``` {"fields":[{"metadata":{},"name":"s","nullable":false,"type":"string collate UTF8_LCASE"}],"type":"struct"} ``` In addition: - `ArrayType.fromJson` and `MapType.fromJson` ignored their `collationsMap`. - `StructType.fromJson` and `StructField.fromJson` raised `INVALID_DATATYPE_FORMAT` for a field of a user-defined type. - `jsonValue()` sorted its keys, so `json.dumps(schema.jsonValue())` differed from PySpark's. ### Does this PR introduce _any_ user-facing change? Yes. The JSON of the types, and its readers, are those of PySpark 4.3.0. ### How was this patch tested? - A new live test checks the JSON of the schema of collated columns from the server, and reads it back with `StructType.fromJson`. It passes with official PySpark 4.3.0 and fails without this change. - A new offline test checks the JSON and the `jsonValue` of a struct of collated strings, also in arrays, maps and nested structs, against PySpark's. It reads the struct back with each reader, round-trips a struct of a user-defined type, and checks the errors. - A new lib test checks the JSON of collated strings of the core, its errors, and the JSON of a UDT. - On 28 cases of the readers and their errors, and on the writers, the client gives the same results as official PySpark 4.3.0. - The lib tests of both crates (589 passed), the build of all tests, also with `wasm-udf`, the offline Python suites (1979 passed, 8 skipped), and the live drop-in and Arrow suites (135 passed) and live cases (10 passed) against a local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
