HyukjinKwon opened a new pull request, #282: URL: https://github.com/apache/spark-connect-rust/pull/282
### What changes were proposed in this pull request? [SPARK-61083](https://issues.apache.org/jira/browse/SPARK-61083): the methods of a session now raise PySpark Connect's errors for their arguments. - `addTag` and `removeTag` raise these errors, all with the `arg_name` "Spark Connect tag": - `CANNOT_BE_NONE` for `None` - `VALUE_ALLOWED` for a tag that contains a comma - `VALUE_NOT_NON_EMPTY_STR` for an empty tag - `table` raises `NOT_EXPECTED_TYPE` for a name that is not a str. - `copyFromLocalToFs` raises `NO_SCHEMA_AND_DRIVER_DEFAULT_SCHEME` for a destination with a scheme, such as `hdfs://`. For a local path that is not a file, it raises an `AssertionError` ("local path must be a file path."). - `_jsc`, `_jconf`, `_jvm`, `_jsparkSession` and `sparkContext` raise the `PySparkAttributeError` `JVM_ATTRIBUTE_NOT_SUPPORTED`. Other missing attributes still raise a plain `AttributeError`. The core adds `SparkSession::check_tag`, which `add_tag` uses. ### Why are the changes needed? The client: - raised `INVALID_TAG` for the invalid tags of `addTag` - accepted any tag in `removeTag`, including an empty one - raised a plain `TypeError` for `table(1)` - raised a `PySparkRuntimeError` for a missing local file, and accepted a destination with a scheme - raised a plain `AttributeError` for `sparkContext` ### Does this PR introduce _any_ user-facing change? Yes. These methods raise the errors of PySpark 4.3.0. ### How was this patch tested? - A new live test checks the errors of a tag with a comma, an empty tag, a table name that is not a str, and `sparkContext`. It passes with official PySpark 4.3.0 and fails without this change. - A new offline test checks every error above against a mock Connect server. - A new lib test checks `SparkSession::check_tag`. - On a local server, 24 cases of these arguments, including valid ones, give the same results as official PySpark 4.3.0. - The lib tests of both crates (593 passed), the build of all tests, also with `wasm-udf`, the offline Python suites (2026 passed, 12 skipped), and the live drop-in and Arrow suites (142 passed) and live cases (10 passed) against a local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
