HyukjinKwon commented on PR #51: URL: https://github.com/apache/spark-connect-rust/pull/51#issuecomment-5403048203
**Assisted code review** (vendored Spark trees — `pandas`/`testing`/`cloudpickle` — skipped; focused on the PyO3 bindings and adapted wrappers). **1. [High] All server errors collapse to `RuntimeError`** — `crates/pyspark-rs/src/errors.rs:8` `spark_error_to_py_exception` maps **every** `SparkError` to `PyRuntimeError`, discarding the `error_class`/kind the Rust core carefully preserves, and nothing in the Python connect layer re-maps it. So `except AnalysisException:` / `ParseException` / `IllegalArgumentException` etc. won't catch server errors — a regression against the "complete drop-in for the reference pyspark client" claim. Map by `error_class`/`SparkErrorKind` to the corresponding `pyspark.errors` exception types. **2. [High] Streaming query listeners don't deliver live events** (root cause in #50) `python/pyspark/sql/streaming/query.py:_query_event_handler` calls `streamListenerEvents()` once; the underlying Rust `stream_listener_events()` (#50, `streaming.rs:869`) returns a batch only when the stream closes, so live progress/idle/terminated events never reach registered listeners. See #50 finding 1. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
