HyukjinKwon opened a new pull request, #271: URL: https://github.com/apache/spark-connect-rust/pull/271
### What changes were proposed in this pull request? [SPARK-61073](https://issues.apache.org/jira/browse/SPARK-61073): `StructType[key]` and `StructType.add` now take their arguments and raise their errors as PySpark's do. - `StructType[name]` raises the `PySparkKeyError` `KEY_NOT_EXISTS` for a name that no field has. - `StructType[index]` raises the `PySparkIndexError` `INDEX_OUT_OF_RANGE` for an index outside the fields. It raises `NOT_EXPECTED_TYPE` for a key that is not an int, a slice or a str. - `StructType.add(name, data_type)` parses a str `data_type` as the JSON value of a data type, such as `"integer"` or `"decimal(10,2)"`. For a str that is not one, such as `"int"`, it raises `CANNOT_PARSE_DATATYPE`. For a name without a data type, it raises `ARGUMENT_REQUIRED`. - `_parse_datatype_json_value` raises PySpark's `CANNOT_PARSE_DATATYPE` and `UNSUPPORTED_DATA_TYPE`. ### Why are the changes needed? The client raised these errors instead: - a plain `KeyError: 'No StructField named zz'` for a missing field - a plain `IndexError` and `TypeError` for the other keys `add` also differed: - it parsed a str data type as DDL, so `add("a", "int")` was accepted - it took a `StringType` for a name without a data type - it raised a plain `ValueError` for a type that does not parse ### Does this PR introduce _any_ user-facing change? Yes. `StructType[key]` and `StructType.add` take the arguments, and raise the errors, of PySpark 4.3.0. ### How was this patch tested? - A new live test builds a schema with `add` and the JSON values of types, creates a DataFrame of it, and checks the error for a missing field of its schema. It passes with official PySpark 4.3.0 and fails without this change. - A new offline test checks the errors of `StructType[key]` for names, indexes and other keys. It also checks the types that `add` parses, and its errors. - The lib tests of both crates (583 passed), the build of all tests, also with `wasm-udf`, the offline Python suites (1930 passed, 8 skipped), and the live drop-in and Arrow suites (127 passed) and live cases (10 passed) against a local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
