HyukjinKwon opened a new pull request, #302: URL: https://github.com/apache/spark-connect-rust/pull/302
### What changes were proposed in this pull request? [SPARK-61101](https://issues.apache.org/jira/browse/SPARK-61101): the batch reader and writer now check their arguments as PySpark Connect's do. As PySpark Connect's: - A write raises `UNSUPPORTED_OPERATION` when the last save mode set is not `append`, `overwrite`, `error` or `ignore`, in any case. This covers `save`, `saveAsTable`, `insertInto`, the format methods and `jdbc`. The core's `DataFrameWriter` fails the write; it used to ignore the mode. - `format(None)` of `DataFrameReader`, `DataFrameWriter` and `DataStreamReader` unsets the format, so the server uses its default source. The core's readers and writer gain `clear_format`. - `DataFrameReader._jreader` raises `JVM_ATTRIBUTE_NOT_SUPPORTED`. - `DataFrameReader.jdbc`: - sets the format `jdbc` on the reader first - with a `column`, asserts `lowerBound`, `upperBound` and `numPartitions` (for example "lowerBound can not be None when ``column`` is specified") and ignores `predicates` - without a `column`, ignores the bounds and `numPartitions` - takes the column, the bounds, `numPartitions` and the values of `properties` as options of any type, like `DataFrameWriter.jdbc`'s `properties` - `DataFrameReader.changes` of a reader with a schema raises the `AnalysisException` `_LEGACY_ERROR_TEMP_1189`, as `DataStreamReader.changes` already does. The two share the code now. - `csv`, `text` and `orc` of None read without a path. The client keeps `errorifexists` as the save mode `error`. PySpark documents it, and pandas-on-Spark's `validate_mode` passes it to the writer. PySpark Connect 4.3.0 raises `UNSUPPORTED_OPERATION` for it, against its own documentation. ### Why are the changes needed? The client: - accepted any save mode: `df.write.mode("bad").save(path)` saved the data with the default mode, and a typo like `mode("overwite")` failed later with `PATH_ALREADY_EXISTS` - raised a `TypeError` for `format(None)` - sent a JDBC read of a column without bounds to the server - dropped the schema of a reader in `changes` - raised an `AttributeError` for `_jreader` - raised an `AssertionError` for `csv(None)`, `text(None)` and `orc(None)` ### Does this PR introduce _any_ user-facing change? Yes. The reader and the writer take their arguments, and raise their errors, as in PySpark 4.3.0. ### How was this patch tested? - A new live test checks an unsupported save mode, which writes nothing; `format(None)`, which writes and reads Parquet; `changes` of a schema; `jdbc` of a column without bounds; `_jreader`; and `text(None)`. It passes with official PySpark 4.3.0 and fails without this change. - A new offline test checks these errors and the requests sent for each mode and format, and for the options and predicates of `jdbc`. - A new core test checks the save modes of a write. - On a local server, 160 cases of the reader and the writer with good and bad arguments give the same results as official PySpark 4.3.0. These are left: - the protobuf and PyO3 messages of a `TypeError` of an argument of a wrong type - `errorifexists`, kept as above - `bucketBy(0, ...)`: PySpark Connect drops the bucketing (it sends one only for a positive count), while the client sends it and the server raises `INVALID_BUCKET_COUNT`, as classic PySpark does - The lib tests of both crates (589 passed), the build of all tests, also with `wasm-udf`, the offline Python suites (2051 passed, 12 skipped), and the live drop-in and Arrow suites (155 passed) and live cases (10 passed) against a local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
