HyukjinKwon opened a new pull request, #263: URL: https://github.com/apache/spark-connect-rust/pull/263
### What changes were proposed in this pull request? [SPARK-61055](https://issues.apache.org/jira/browse/SPARK-61055): `StreamingQuery.status` and `StreamingQuery.exception()` now return what PySpark Connect's do, and the waits of a query check their timeouts as PySpark's. - `query.status` is a dict of the `message`, `isDataAvailable` and `isTriggerActive` of the query. - `query.exception()` of a failed query returns the `StreamingQueryException` of `pyspark.errors`: - of the message of the server, without the type of its Java exception, and with its JVM stack trace - of the error class as its reason - `pyspark.sql.streaming.StreamingQueryException` and `pyspark.sql.streaming.query.StreamingQueryException` are that class of `pyspark.errors`, as in PySpark. - `awaitTermination(timeout)` and `spark.streams.awaitAnyTermination(timeout)` raise `VALUE_NOT_POSITIVE` of `timeout` for a timeout that is not a positive number. - `explain(extended)` takes an int as a bool, as the proto field of PySpark does. The `StreamingQueryStatus` and `StreamingQueryException` classes of the extension, which PySpark does not have, are removed. The core `StreamingQueryException` now keeps the stack trace of the server. ### Why are the changes needed? - `query.status` was an object of `status_message`, `is_data_available` and `is_trigger_active` attributes, neither subscriptable nor iterable: `query.status["message"]` raised a `TypeError`. - `query.exception()` returned an object that was not an exception, whose `str()` was its repr. - `pyspark.sql.streaming.StreamingQueryException` was that class, so `except StreamingQueryException:` raised a `TypeError`. - The timeouts raised `INVALID_TIMEOUT` or `TypeError`s, and `explain(1)` raised a `TypeError`. ### Does this PR introduce _any_ user-facing change? Yes. These return and raise what PySpark 4.3.0 does. ### How was this patch tested? - A new live test checks the dict of `status`, the `StreamingQueryException` of a failed query, its message and JVM stack trace, and that `awaitTermination` raises it. It passes with official PySpark 4.3.0, and fails without this change. - A new offline test checks the class of `StreamingQueryException`, the errors of the timeouts, and those of `explain`. - On a local server, `status` and `exception()` of a failed query give the dict, the class and the 125 lines of the message of official PySpark 4.3.0. - The lib tests of both crates (580 passed), the build of all tests, also with `wasm-udf`, the offline Python suites (1847 passed, 8 skipped), and the live drop-in and Arrow suites (116 passed, without the test of `StreamingQuery.explain`, which hangs on a slow machine and #260 fixes) and live cases (10 passed) against a local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
