HyukjinKwon opened a new pull request, #266: URL: https://github.com/apache/spark-connect-rust/pull/266
### What changes were proposed in this pull request? [SPARK-61057](https://issues.apache.org/jira/browse/SPARK-61057): `spark.readStream` now checks its arguments as PySpark Connect's `DataStreamReader` does. - `name(source_name)` raises: - `NOT_EXPECTED_TYPE` of `source_name` for one that is not a str - the `PySparkValueError` `INVALID_STREAMING_SOURCE_NAME` for one of other characters than ASCII letters, digits and underscores - `load(path)` raises `VALUE_NOT_NON_EMPTY_STR` of `path` for a path that is not a non-empty str. - `csv`, `json`, `orc`, `parquet`, `text` and `xml` raise `NOT_EXPECTED_TYPE` of `path` for one that is not a str. As in PySpark, they set their options on the reader before. - `changes(tableName)` of a reader of a schema raises the `AnalysisException` "User specified schema not supported with `changes`.", of `_LEGACY_ERROR_TEMP_1189`. ### Why are the changes needed? The client: - raised `TypeError`s for these, and a plain `ValueError` for a source name - sent an empty path to the server, which raised an `IllegalArgumentException` - sent `changes` of a schema to the server, which raised another error ### Does this PR introduce _any_ user-facing change? Yes. These raise the errors of PySpark 4.3.0. ### How was this patch tested? - A new live test checks a streaming read of a path, and the errors of an empty path, a list of paths, a source name and `changes` of a schema. It passes with official PySpark 4.3.0, and fails without this change. - A new offline test checks these errors and their parameters, and that a format reader keeps its options. - On a local server, the 31 cases of `DataStreamReader` give the errors of official PySpark 4.3.0. The exception is the message of the `TypeError` of a table name that is not a str. - The lib tests of both crates (582 passed), the build of all tests, also with `wasm-udf`, the offline Python suites (1900 passed, 8 skipped), and the live drop-in and Arrow suites (119 passed) and live cases (10 passed) against a local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
