HyukjinKwon opened a new pull request, #236: URL: https://github.com/apache/spark-connect-rust/pull/236
### What changes were proposed in this pull request? [SPARK-61031](https://issues.apache.org/jira/browse/SPARK-61031): the Python client now takes the arguments of the readers, `df.write` and `spark.catalog` as PySpark does, and `createTable` creates the table on the call. This PR is on top of #235 ([SPARK-61029](https://issues.apache.org/jira/browse/SPARK-61029)), whose server-side DDL parsing the `Parse` of the readers uses. It includes that commit until #235 merges, and its own change is the last commit. Readers: - `spark.read.load(path)` and the format readers, `csv`, `json`, `orc`, `text` and `xml`, take a path or a list of paths, and `parquet(*paths)` takes any number of them. As PySpark's `DataSource` asserts them, any other path fails an assertion. - `csv`, `json` and `xml` also take a DataFrame of one string column, whose strings they parse with a `Parse` relation, of the schema and the options of the reader. The server parses a DDL schema, as PySpark's `StructType.fromDDL` does. `json` and `xml` raise `NOT_EXPECTED_TYPE` for any other `path`. A parse keeps the schema and the options on the reader, but not the format, which only `load` sets. Writer: - `bucketBy(numBuckets, col, *cols)` and `sortBy(col, *cols)` take column names, or a list or tuple of them alone. They raise `NOT_EXPECTED_TYPE` for a `numBuckets` that is not an int and for columns that are not strings, and `CANNOT_SET_TOGETHER` for a list together with more columns. - `clusterBy` takes a list or tuple of columns too, and asserts that there is at least one. - `mode(None)` keeps the default mode. Catalog: - `createTable` creates the table on the call, as PySpark's `df._to_table()` does, of the schema and the string options given. It raises `NOT_EXPECTED_TYPE` for a schema that is not a StructType. - `createExternalTable` warns that it is deprecated and creates the table as `createTable` does. - `tableExists`, `functionExists` and `listColumns` warn that `dbName` is deprecated. The core adds `DataFrameReader::load_paths` and `parse`, a `Parse` plan, and `Catalog::create_table_with` and `create_external_table_with`, of a schema and options. It now creates these tables on the call. ### Why are the changes needed? The client: - raised a TypeError for a list of paths, as in `spark.read.csv([path])`, and for a DataFrame of strings, and a ValueError for `parquet` of more than one path - raised TypeErrors for the lists of columns of `bucketBy`, `sortBy` and `clusterBy`, and for `mode(None)`, and TypeErrors rather than PySpark's errors for the other arguments of `bucketBy` - ignored the schema and the options of `createTable` and `createExternalTable`, and created the table only when the returned DataFrame ran - did not warn of `createExternalTable` and of `dbName` ### Does this PR introduce _any_ user-facing change? Yes. These methods take the arguments and raise the errors of PySpark 4.3.0, and `createTable` creates the table on the call. ### How was this patch tested? - A new live test parses DataFrames of strings with `csv`, `json` and `xml`, loads lists of paths, and creates tables with `createTable` and `createExternalTable`. It passes with official PySpark 4.3.0. - New offline tests check, against a mock Connect server, the `Parse` relations, the paths that the readers load, and the `CreateTable` that `createTable` and `createExternalTable` send on the call. They also check the errors and the warnings. - A new lib test checks the `Parse` plan, and new e2e tests check `Parse` and the tables created against the server. - On a local server, 52 cases of these methods behave as in official PySpark 4.3.0: their results, errors, warnings and reader state. So do 19 `Parse` schemas. - The lib tests of both crates (572 passed), the build of all tests, also with `wasm-udf`, the offline Python suites (1826 passed, 8 skipped), and the live drop-in and Arrow suites (110 passed) and live cases (10 passed) against a local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
