HyukjinKwon opened a new pull request, #274: URL: https://github.com/apache/spark-connect-rust/pull/274
### What changes were proposed in this pull request? [SPARK-61061](https://issues.apache.org/jira/browse/SPARK-61061): `pyspark.InheritableThread` and `pyspark.inheritable_thread_target` are added, as PySpark's of Spark Connect. - `InheritableThread(target, session=session)` runs `target` in a thread that inherits the tags of `session` from the thread that starts it. - `inheritable_thread_target(session)(f)` runs `f` with the tags of `session` of the thread that wraps it, and with `session` as the active session of its thread. - So `spark.interruptTag` of a tag of the parent interrupts the operations of the thread too. - They assert a session, as PySpark's of Spark Connect do. - `SparkSession._set_default_and_active_session` makes a session the default one, and the active one of the thread, when they are not set, as PySpark's. This PR is on top of #269 ([SPARK-61067](https://issues.apache.org/jira/browse/SPARK-61067)), which keeps the tags per thread, as these need. It includes that commit until it merges, and its own change is the last commit. ### Why are the changes needed? The client had neither: `from pyspark import InheritableThread` raised an `ImportError`. ### Does this PR introduce _any_ user-facing change? Yes. These are added, as in PySpark 4.3.0. ### How was this patch tested? - A new live test runs a slow query in an `InheritableThread` of a parent of a tag, interrupts the tag, and checks that the query is interrupted. It passes with official PySpark 4.3.0, and fails without this change. - A new offline test checks the tags and the active session of a plain thread, an `InheritableThread` and a thread of `inheritable_thread_target`, as in official PySpark 4.3.0, and their assertions. - The lib tests of both crates (590 passed), the build of all tests, also with `wasm-udf`, the offline Python suites (1979 passed, 8 skipped), and the live drop-in and Arrow suites (135 passed) and live cases (10 passed) against a local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
