HyukjinKwon commented on PR #51: URL: https://github.com/apache/spark-connect-rust/pull/51#issuecomment-5403248305
Addressed the assisted-review findings on this PR. Commit on `HyukjinKwon:split-python-wrapper`. **1. [High] All server errors collapsed to `RuntimeError`** — fixed. `crates/pyspark-rs/src/errors.rs` now maps `SparkError.kind` to the corresponding `pyspark.errors` class — `Analysis → AnalysisException`, `Parse → ParseException`, `IllegalArgument → IllegalArgumentException`, `QueryExecution → QueryExecutionException`, `Python → PythonException`, `StreamingQuery → StreamingQueryException`, and the builtin-style kinds to their `PySpark*` subclasses (which also subclass the Python builtins), etc. It imports `pyspark.errors` under the GIL and constructs the typed exception, falling back to `PyRuntimeError` only if the import fails. So `except AnalysisException:` / `ParseException` / `IllegalArgumentException` catch server errors again. All 29 mapped class names verified present in the vendored `pyspark.errors`. **2. [High] Streaming listeners didn't deliver live events** — fixed (root cause on #50). `python/pyspark/sql/streaming/query.py`'s `StreamingQueryListenerBus` daemon thread now loops the **incremental** `streamListenerEvents()` iterator and dispatches `onQueryStarted`/`onQueryProgress`/`onQueryIdle`/`onQueryTerminated` as each event arrives; it stops when the last listener is removed. Critically, `PyListenerEventStream.__next__` releases the GIL (`py.detach`) around the blocking wait — otherwise the daemon thread would freeze the interpreter between events (the same GIL concern from the earlier round, applied to the listener bus). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
