HyukjinKwon opened a new pull request, #273: URL: https://github.com/apache/spark-connect-rust/pull/273
### What changes were proposed in this pull request? [SPARK-61052](https://issues.apache.org/jira/browse/SPARK-61052): the requests of the writers, the catalog and the streaming queries now release the GIL while they wait for the server, as PySpark Connect's requests do through grpcio. So other Python threads run meanwhile. They are: - `df.write.save`, `saveAsTable`, `insertInto` and the format writers, such as `parquet` - the writes of `df.writeTo` and `mergeInto` - the methods of `spark.catalog` - `df.isEmpty` - the `start` and `toTable` of `df.writeStream` - the methods of a `StreamingQuery`, such as `awaitTermination`, and of `spark.streams`, such as `addListener` The client already released it for `collect`, `count`, `sql` and the other requests of a DataFrame. The new `detached` helpers of the catalog and of the module run a request without the GIL. ### Why are the changes needed? The client held the GIL across these requests, so every other Python thread waited for them. `query.awaitTermination(3)` stopped every other thread for 3 seconds. A listener of `spark.streams` could not run its callbacks meanwhile: none of the progress of a query reached it during `awaitTermination`. ### Does this PR introduce _any_ user-facing change? Yes. Other Python threads, and the listeners of streaming queries, run during these requests, as in PySpark 4.3.0. ### How was this patch tested? - A new live test checks that another thread runs during `awaitTermination`, a write and catalog requests, and that a listener receives the progress of the query during `awaitTermination`. It passes with official PySpark 4.3.0, and fails without this change. - A new offline test checks that this thread runs while 12 of these requests wait for a mock Connect server. Without this change, they hold the GIL until the server gives up. - The lib tests of both crates (589 passed), the build of all tests, also with `wasm-udf`, the offline Python suites (1989 passed, 8 skipped), and the live drop-in and Arrow suites (134 passed) and live cases (10 passed) against a local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
