This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 763d661f [SPARK-61166][CONNECT][PYTHON] Make the threads of
inheritable_thread_target share the copy of the tags of the session, in the
Rust and Python clients, as PySpark does
763d661f is described below
commit 763d661fcf6e5b1c8949e84e84d7900f877918c4
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 16:00:48 2026 +0900
[SPARK-61166][CONNECT][PYTHON] Make the threads of
inheritable_thread_target share the copy of the tags of the session, in the
Rust and Python clients, as PySpark does
### What changes were proposed in this pull request?
[SPARK-61166](https://issues.apache.org/jira/browse/SPARK-61166): PySpark
Connect's `inheritable_thread_target(session)` copies the `thread_local` of the
client, of the tags of the thread, when it decorates a function. Every thread
that runs the function gets that one copy, so the threads share it: a tag that
one of them adds is a tag of the others.
The client now does the same.
**Rust core**: the tags of a thread are a set that threads can share:
- `SparkSession::copy_thread_tags` copies the tags of the current thread,
as of now, into a new `SharedTags`.
- `SparkSession::share_thread_tags` makes such a copy the tags of the
current thread.
- The tags that a thread adds or removes are then those of all the threads
given it.
- `clear_tags` gives the thread a new empty set of its own, as PySpark's
`clearTags` sets the tags of the `thread_local` to a new set.
- A thread whose tags are empty is still forgotten, unless it shares them.
**Python client**: `inheritable_thread_target(session)` copies the tags
when it decorates the function, as before, and gives each of its threads the
copy (`_share_thread_tags`), rather than adding the tags to each thread.
`InheritableThread`, which copies the tags when the thread starts, is not
changed. Nor are the thread-local user context extensions, which PySpark's
decorator copies in the same `thread_local`: the client's decorator does not
propagate them, before or after this change.
### Why are the changes needed?
The client copied the tags into each thread instead. So a thread that the
function started, from another thread of the function, lacked the tags that
that other thread had added. Apache Spark's `test_inheritable_tags_with_deco`
of `test_parity_job_cancellation.py` failed on this client: the nested thread
lacked the tag "b" of the thread that started it.
### Does this PR introduce _any_ user-facing change?
Yes. The threads of `inheritable_thread_target` share their tags, as with
PySpark 4.3.0. For the Rust API, `SharedTags`, `SparkSession::copy_thread_tags`
and `SparkSession::share_thread_tags` are new.
### How was this patch tested?
- A new Rust unit test checks that threads given a copy share it, that the
copy is of the tags when it was made, and that a thread that clears its tags
gets a set of its own.
- A new offline test runs Apache Spark's case: a decorated function in a
thread, which starts a nested thread of it. It checks the tags of both threads
and of the decorating thread. It passes with official PySpark 4.3.0, and fails
without this change.
- The tags of the same case against a local server are those of official
PySpark 4.3.0.
- Apache Spark's `test_parity_job_cancellation.py` passes on this client
against a local server: 8 tests pass and 1 is skipped.
- The lib tests of both crates (598 passed), the build of all tests, also
with `wasm-udf`, the offline Python suites (2222 passed, 8 skipped), and the
live drop-in and Arrow suites (204 passed) and live cases (10 passed) against a
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #357 from HyukjinKwon/SPARK-61166-shared-tags.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/session.rs | 19 ++++++
crates/spark-connect/src/session.rs | 116 ++++++++++++++++++++++++++++++++----
python/pyspark/util.py | 10 ++--
python/tests/test_dropin_offline.py | 39 ++++++++++++
4 files changed, 168 insertions(+), 16 deletions(-)
diff --git a/crates/pyspark-rs/src/session.rs b/crates/pyspark-rs/src/session.rs
index 1bfe99b1..f9fd5f36 100644
--- a/crates/pyspark-rs/src/session.rs
+++ b/crates/pyspark-rs/src/session.rs
@@ -548,6 +548,11 @@ impl PyHook {
}
/// Python wrapper for a Spark session.
+/// A copy of the tags of a thread, which the threads that
`_share_thread_tags` gives it share,
+/// for `pyspark.util.inheritable_thread_target`.
+#[pyclass(name = "_SharedTags", module = "pyspark.sql.session", frozen)]
+pub struct PySharedTags(spark_connect::session::SharedTags);
+
/// A tag of `addTag` or `removeTag`: `CANNOT_BE_NONE` for None, as PySpark
Connect's.
fn tag_of(py: Python<'_>, tag: &Bound<'_, PyAny>) -> PyResult<String> {
if tag.is_none() {
@@ -2354,6 +2359,20 @@ impl PySparkSession {
self.session.clear_tags()
}
+ /// A copy of the tags of the current thread, which
`pyspark.util.inheritable_thread_target`
+ /// makes when it decorates a function, as PySpark Connect's copies the
`thread_local` of the
+ /// client.
+ fn _copy_thread_tags(&self) -> PySharedTags {
+ PySharedTags(self.session.copy_thread_tags())
+ }
+
+ /// Make `tags` those of the current thread, which shares them with the
other threads given
+ /// them, as `pyspark.util.inheritable_thread_target` gives every thread
that runs its
+ /// function the copy of the `thread_local` of the client of PySpark
Connect.
+ fn _share_thread_tags(&self, tags: &PySharedTags) {
+ self.session.share_thread_tags(&tags.0)
+ }
+
/// Add local files, or `file:` URIs, as artifacts of the session, as
PySpark's
/// `addArtifacts(*path, pyfile=False, archive=False, file=False)`: each
path is named by
/// the flags and its extension, and the errors for a path that is not
supported are
diff --git a/crates/spark-connect/src/session.rs
b/crates/spark-connect/src/session.rs
index 95b3eab4..0ec881f3 100644
--- a/crates/spark-connect/src/session.rs
+++ b/crates/spark-connect/src/session.rs
@@ -41,32 +41,72 @@ pub struct ExecutionInfo {
/// The operation tags of a session, per thread, as PySpark Connect keeps them
in the
/// `thread_local` of its client: those that a thread added, which the
operations that it
-/// starts carry. A new thread has none.
+/// starts carry. A new thread has none. The tags of a thread are a set that
it may share with
+/// other threads ([`SharedTags`]), as the threads of PySpark's
`inheritable_thread_target` share
+/// the set of tags that it copied.
#[derive(Default)]
pub(crate) struct ThreadTags {
- by_thread: Mutex<std::collections::HashMap<std::thread::ThreadId,
Vec<String>>>,
+ by_thread: Mutex<std::collections::HashMap<std::thread::ThreadId,
SharedTags>>,
}
+/// A set of operation tags of threads: a copy of the tags of a thread, of
+/// [`SparkSession::copy_thread_tags`], which the threads given it by
+/// [`SparkSession::share_thread_tags`] share, as the threads of PySpark's
+/// `inheritable_thread_target` share the copy of the `thread_local` of the
client of the thread
+/// that decorated their function. The tags that one of them adds or removes
are those of all.
+#[derive(Clone, Default)]
+pub struct SharedTags(Arc<Mutex<Vec<String>>>);
+
impl ThreadTags {
/// The tags of the current thread.
fn current(&self) -> Vec<String> {
- let by_thread = self.by_thread.lock().unwrap_or_else(|e|
e.into_inner());
- by_thread
+ let shared = self
+ .by_thread
+ .lock()
+ .unwrap_or_else(|e| e.into_inner())
.get(&std::thread::current().id())
- .cloned()
+ .cloned();
+ shared
+ .map(|tags| tags.0.lock().unwrap_or_else(|e|
e.into_inner()).clone())
.unwrap_or_default()
}
- /// Change the tags of the current thread, and forget a thread without
tags.
+ /// Change the tags of the current thread, and forget a thread without
tags that it shares
+ /// with no other.
fn update(&self, change: impl FnOnce(&mut Vec<String>)) {
let mut by_thread = self.by_thread.lock().unwrap_or_else(|e|
e.into_inner());
let thread = std::thread::current().id();
- let tags = by_thread.entry(thread).or_default();
- change(tags);
- if tags.is_empty() {
+ let shared = by_thread.entry(thread).or_default().clone();
+ let mut tags = shared.0.lock().unwrap_or_else(|e| e.into_inner());
+ change(&mut tags);
+ // The set is held by the map and here only: no other thread or copy
shares it.
+ if tags.is_empty() && Arc::strong_count(&shared.0) == 2 {
+ drop(tags);
by_thread.remove(&thread);
}
}
+
+ /// A new empty set of tags of the current thread, which shares no tags
with another, as
+ /// PySpark Connect's `clearTags` sets the tags of the `thread_local` to a
new set.
+ fn clear(&self) {
+ self.by_thread
+ .lock()
+ .unwrap_or_else(|e| e.into_inner())
+ .remove(&std::thread::current().id());
+ }
+
+ /// A copy of the tags of the current thread.
+ fn copy(&self) -> SharedTags {
+ SharedTags(Arc::new(Mutex::new(self.current())))
+ }
+
+ /// Make `tags` those of the current thread.
+ fn share(&self, tags: &SharedTags) {
+ self.by_thread
+ .lock()
+ .unwrap_or_else(|e| e.into_inner())
+ .insert(std::thread::current().id(), tags.clone());
+ }
}
/// A Spark Connect session for interacting with a remote Spark cluster.
@@ -163,9 +203,25 @@ impl SparkSession {
self.tags.current()
}
- /// Clear the tags that this thread set on this session. Mirrors
`SparkSession.clearTags`.
+ /// Clear the tags that this thread set on this session. Mirrors
`SparkSession.clearTags`:
+ /// the thread gets a new empty set of tags, which it shares with no other
thread.
pub fn clear_tags(&self) {
- self.tags.update(Vec::clear);
+ self.tags.clear();
+ }
+
+ /// A copy of the tags of the current thread, as of now, which the threads
that are given it
+ /// with [`SparkSession::share_thread_tags`] share. Mirrors how PySpark's
+ /// `inheritable_thread_target(session)` copies the tags of the thread
that decorates a
+ /// function, which every thread that runs the function then has.
+ pub fn copy_thread_tags(&self) -> SharedTags {
+ self.tags.copy()
+ }
+
+ /// Make `tags` the tags of the current thread, which it shares with the
other threads given
+ /// them: a tag that one of them adds or removes is added or removed for
all, until a thread
+ /// clears its tags.
+ pub fn share_thread_tags(&self, tags: &SharedTags) {
+ self.tags.share(tags);
}
/// Register a handler of the progress of the executions of this session:
it is called with
@@ -2611,6 +2667,44 @@ mod tests {
assert!(spark.tags.by_thread.lock().unwrap().is_empty());
}
+ #[test]
+ fn threads_given_a_copy_of_tags_share_it() {
+ // As the threads of PySpark's inheritable_thread_target: they share
the copy of the tags
+ // of the thread that decorated their function, of then, until one
clears its tags.
+ let spark = session();
+ spark.add_tag("a").unwrap();
+ let copy = spark.copy_thread_tags();
+ spark.add_tag("later").unwrap();
+ let run = |copy: SharedTags, tag: &'static str| {
+ let spark = spark.clone();
+ std::thread::spawn(move || {
+ spark.share_thread_tags(©);
+ spark.add_tag(tag).unwrap();
+ spark.get_tags()
+ })
+ .join()
+ .unwrap()
+ };
+ assert_eq!(run(copy.clone(), "b"), vec!["a", "b"]);
+ assert_eq!(run(copy.clone(), "c"), vec!["a", "b", "c"]);
+ assert_eq!(spark.get_tags(), vec!["a", "later"]);
+ let cleared = {
+ let spark = spark.clone();
+ let copy = copy.clone();
+ std::thread::spawn(move || {
+ spark.share_thread_tags(©);
+ spark.remove_tag("c");
+ spark.clear_tags();
+ spark.add_tag("d").unwrap();
+ spark.get_tags()
+ })
+ .join()
+ .unwrap()
+ };
+ assert_eq!(cleared, vec!["d"]);
+ assert_eq!(run(copy, "e"), vec!["a", "b", "e"]);
+ }
+
#[test]
fn arrow_local_relation_preserves_batches_and_rejects_schema_mismatch() {
use arrow::array::Array;
diff --git a/python/pyspark/util.py b/python/pyspark/util.py
index 5708a518..bf8877e8 100644
--- a/python/pyspark/util.py
+++ b/python/pyspark/util.py
@@ -141,7 +141,9 @@ def inheritable_thread_target(f: Union[Callable,
"SparkSession"]) -> Callable:
>>> Thread(target=inheritable_thread_target(session)(target_func)).start()
# doctest: +SKIP
"""
# As PySpark's of Spark Connect, the only mode of this client: `f` is the
session, whose
- # tags, those of this thread, the thread inherits, with the session as its
active one.
+ # tags, those of this thread when the function is decorated, the threads
of the function
+ # inherit, with the session as their active one. As PySpark's, which gives
each thread the
+ # one copy of the `thread_local` of the client, the threads share that
copy of the tags.
from pyspark.sql import SparkSession
session = f
@@ -150,7 +152,7 @@ def inheritable_thread_target(f: Union[Callable,
"SparkSession"]) -> Callable:
assert isinstance(session, SparkSession), (
"f is expected to be SparkSession for spark connect"
)
- tags = set(session.getTags())
+ tags = session._copy_thread_tags()
@functools.wraps(ff)
def inner(*args: Any, **kwargs: Any) -> Any:
@@ -161,9 +163,7 @@ def inheritable_thread_target(f: Union[Callable,
"SparkSession"]) -> Callable:
SparkSession._set_default_and_active_session(session)
# Set the tags of the session in the child thread.
- session.clearTags()
- for tag in tags:
- session.addTag(tag)
+ session._share_thread_tags(tags)
return ff(*args, **kwargs)
return inner
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index 0a4cef52..e4df13f8 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -5723,6 +5723,45 @@ def
test_inheritable_threads_inherit_the_tags_of_the_session_as_pyspark(mock_con
inheritable_thread_target(child)(child)
+def
test_threads_of_inheritable_thread_targets_share_their_tags_as_pyspark(mock_connect):
+ # As PySpark Connect's inheritable_thread_target, as Apache Spark's
+ # test_inheritable_tags_with_deco: the threads of the function share the
copy of the tags of
+ # the thread that decorated it, of then, so a thread that the function
starts sees the tags
+ # that the thread that started it added, while the decorating thread keeps
its own.
+ import threading
+
+ from pyspark import inheritable_thread_target
+
+ spark, _ = mock_connect
+ spark.addTag("a")
+ try:
+ @inheritable_thread_target(spark)
+ def func(target):
+ return target()
+
+ spark.addTag("later")
+ first, second = set(), set()
+
+ def inner():
+ spark.addTag("c")
+ second.update(spark.getTags())
+
+ def outer():
+ spark.addTag("b")
+ first.update(spark.getTags())
+ nested = threading.Thread(target=func, args=(inner,))
+ nested.start()
+ nested.join()
+
+ thread = threading.Thread(target=func, args=(outer,))
+ thread.start()
+ thread.join()
+ assert spark.getTags() == {"a", "later"}
+ assert first == {"a", "b"} and second == {"a", "b", "c"}
+ finally:
+ spark.clearTags()
+
+
def test_row_asdict():
"""Test Row.asDict() method."""
from pyspark.sql import Row
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]