HyukjinKwon opened a new pull request, #251: URL: https://github.com/apache/spark-connect-rust/pull/251
### What changes were proposed in this pull request? [SPARK-61053](https://issues.apache.org/jira/browse/SPARK-61053): `DataFrameWriterV2` now takes the arguments of `overwrite`, `partitionedBy` and `tableProperty` as PySpark Connect's does. - `overwrite(condition)` takes a Column or a column name, as `F._to_col` does. - `partitionedBy(col, *cols)` takes a Column or a column name for each argument. It raises `NOT_EXPECTED_TYPE` of `col` for any other argument, also for a list. - `tableProperty(property, None)` leaves the property unset, also after an earlier value. The core adds `DataFrameWriterV2::unset_table_property`. ### Why are the changes needed? The client: - raised a `TypeError` for a str condition of `overwrite` - took a list of columns in `partitionedBy`, and named the argument of its error `cols` - raised a `TypeError` for a property of None ### Does this PR introduce _any_ user-facing change? Yes. These methods take the arguments of PySpark 4.3.0 and raise its errors. ### How was this patch tested? - A new live test checks that a property of None is not set on the table, the `NOT_EXPECTED_TYPE` of a list in `partitionedBy`, and that a column name in `overwrite` reaches the server. It passes with official PySpark 4.3.0, and fails without this change. - A new offline test checks the `WriteOperationV2`s that these send to a mock Connect server, and their errors. - The lib tests of both crates (574 passed), the build of all tests, also with `wasm-udf`, the offline Python suites (1811 passed, 8 skipped), and the live drop-in and Arrow suites (110 passed) and live cases (10 passed) against a local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
