This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new ec210e9f [SPARK-61186][CONNECT] Forget the tags and user context
extensions of a thread when the thread ends, in the Rust and Python clients, as
PySpark's thread locals do
ec210e9f is described below
commit ec210e9fb41b3e6c004158aa387c4059d6947b72
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 18:52:13 2026 +0900
[SPARK-61186][CONNECT] Forget the tags and user context extensions of a
thread when the thread ends, in the Rust and Python clients, as PySpark's
thread locals do
### What changes were proposed in this pull request?
[SPARK-61186](https://issues.apache.org/jira/browse/SPARK-61186): Forget
the tags and user context extensions of a thread when the thread ends, in the
Rust and Python clients, as PySpark's thread locals do.
Two maps in the Rust core are keyed by thread id, and neither removed the
entry of a thread that ended:
- the operation tags of a session (`ThreadTags`, added by SPARK-61166 for
`inheritable_thread_target`). A thread that shares a copy of the tags leaves an
entry, even an empty one, when it ends. The copy is shared, so the entry is not
removed by the thread itself;
- the user context extensions of a client (`by_thread`, of
`add_threadlocal_user_context_extension`), which were never removed except by
the thread that added them.
The new module `spark_connect_core::thread_entries` holds the fix. When a
thread inserts its entry into a map, `forget_when_thread_ends` registers the
map with the thread's thread-local state, through a `Weak` reference, so the
registration does not keep the map alive. When the thread ends, the
thread-local state drops, and the entry of that thread is removed from every
registered map that is still alive. Registering again is a no-op, and a
registration of a dropped map is pruned at t [...]
Behavior while a thread lives is unchanged: its tags and extensions are the
same, a copy shared with other threads is still shared, and a pool thread keeps
its entry while it lives, as `threading.local()` does. The Python client needs
no change: it already keeps these in the core.
### Why are the changes needed?
A program that starts a thread per task, with
`inheritable_thread_target(session)` or with
`add_threadlocal_user_context_extension`, grew the maps of the session and the
client by one entry, about 80 bytes, per finished thread, without bound.
PySpark's `thread_local` is freed when its thread ends, so official PySpark
keeps none of these.
The Rust test `finished_threads_leave_no_tags` fails on the code before
this change: 1000 finished threads that shared an empty copy of the tags left
1000 entries.
### Does this PR introduce _any_ user-facing change?
No. The tags and the user context extensions of a thread are the same while
it lives, and an ended thread's entries are no longer kept in memory. The new
module `spark_connect_core::thread_entries` is public, and its
`forget_when_thread_ends` is for the crate's own maps.
### How was this patch tested?
- `finished_threads_leave_no_tags` (Rust, `spark-connect`): 1000 finished
threads that share a copy of the tags and add a tag leave no entry, and the
main thread has no tags.
- `a_living_thread_keeps_its_tags_until_it_ends` (Rust, `spark-connect`): a
thread that shares a copy and adds a tag keeps its entry and its tags while it
lives, and its entry is gone once it ends; the sharing of
`threads_given_a_copy_of_tags_share_it` is unchanged.
- `threadlocal_user_context_extensions_of_finished_threads_are_forgotten`
and `threadlocal_user_context_extensions_of_a_living_thread_stay_until_it_ends`
(Rust, `spark-connect-core`): the same for the user context extensions of a
client.
- `entries_of_finished_threads_are_removed`,
`entry_of_a_living_thread_stays_until_it_ends`,
`registration_does_not_keep_a_map_alive`,
`the_registration_of_a_dropped_map_is_pruned` (Rust, `spark-connect-core`,
`thread_entries`).
- No Python test: the Python client's state is the core's, and the entry of
an ended thread is not observable from Python. The existing
`inheritable_thread_target` tests in the offline suite are unchanged and pass.
- The tests of both crates, e2e included, against a local Spark 4.3.0
server (902 passed), the build of all tests, also with `wasm-udf`, the offline
Python suites (2227 passed, 8 skipped), and the live drop-in, Arrow and ML
suites (209 passed) and live cases (10 passed). rustfmt passes, and clippy
reports no diagnostics in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #364 from HyukjinKwon/SPARK-61186-thread-entries-end-with-thread.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/spark-connect-core/src/client.rs | 76 ++++++++++-
crates/spark-connect-core/src/lib.rs | 3 +
crates/spark-connect-core/src/thread_entries.rs | 170 ++++++++++++++++++++++++
crates/spark-connect/src/session.rs | 67 +++++++++-
4 files changed, 308 insertions(+), 8 deletions(-)
diff --git a/crates/spark-connect-core/src/client.rs
b/crates/spark-connect-core/src/client.rs
index 84b4d2eb..0545b225 100644
--- a/crates/spark-connect-core/src/client.rs
+++ b/crates/spark-connect-core/src/client.rs
@@ -27,6 +27,7 @@ use crate::channel::{ChannelBuilder,
GRPC_MAX_MESSAGE_LENGTH_DEFAULT};
use crate::error::{Result, SparkError, SparkErrorKind};
use crate::reattach::ExecutePlanResponseReattachableIterator;
use crate::retries::{RetryPolicy, RetryPolicyState};
+use crate::thread_entries::forget_when_thread_ends;
/// How long to wait for the lazy gRPC channel to become ready before
surfacing a
/// retriable `UNAVAILABLE`. tonic's lazy channel waits indefinitely for an
unreachable
@@ -266,6 +267,12 @@ struct UserContextExtensions {
by_thread: std::collections::HashMap<std::thread::ThreadId, Vec<(String,
prost_types::Any)>>,
}
+/// Forget the extensions of a thread that ends, as PySpark's `thread_local`
is freed with its
+/// thread.
+fn forget_thread_extensions(extensions: &mut UserContextExtensions, thread:
std::thread::ThreadId) {
+ extensions.by_thread.remove(&thread);
+}
+
impl SparkConnectClient {
/// Create a new SparkConnectClient from a ChannelBuilder.
///
@@ -445,11 +452,17 @@ impl SparkConnectClient {
/// and return its ID, as PySpark's
`add_threadlocal_user_context_extension`.
pub fn add_threadlocal_user_context_extension(&self, extension:
prost_types::Any) -> String {
let id = format!("threadlocal_{}", Uuid::new_v4());
- self.user_context_extensions
+ let mut extensions = self
+ .user_context_extensions
.lock()
- .unwrap_or_else(|e| e.into_inner())
+ .unwrap_or_else(|e| e.into_inner());
+ let thread = std::thread::current().id();
+ if !extensions.by_thread.contains_key(&thread) {
+ forget_when_thread_ends(&self.user_context_extensions,
forget_thread_extensions);
+ }
+ extensions
.by_thread
- .entry(std::thread::current().id())
+ .entry(thread)
.or_default()
.push((id.clone(), extension));
id
@@ -2176,4 +2189,61 @@ mod tests {
];
let _result = client.get_config_with_defaults(&pairs);
}
+
+ #[tokio::test]
+ async fn
threadlocal_user_context_extensions_of_finished_threads_are_forgotten() {
+ let builder = ChannelBuilder::parse("sc://localhost").unwrap();
+ let client =
std::sync::Arc::new(SparkConnectClient::connect(&builder).await.unwrap());
+ for _ in 0..1000 {
+ let client = client.clone();
+ std::thread::spawn(move || {
+
client.add_threadlocal_user_context_extension(prost_types::Any::default());
+ })
+ .join()
+ .unwrap();
+ }
+ let entries = client
+ .user_context_extensions
+ .lock()
+ .unwrap()
+ .by_thread
+ .len();
+ assert_eq!(entries, 0);
+ }
+
+ #[tokio::test]
+ async fn
threadlocal_user_context_extensions_of_a_living_thread_stay_until_it_ends() {
+ let builder = ChannelBuilder::parse("sc://localhost").unwrap();
+ let client =
std::sync::Arc::new(SparkConnectClient::connect(&builder).await.unwrap());
+ let (added_tx, added_rx) = std::sync::mpsc::channel();
+ let (end_tx, end_rx) = std::sync::mpsc::channel::<()>();
+ let child = {
+ let client = client.clone();
+ std::thread::spawn(move || {
+
client.add_threadlocal_user_context_extension(prost_types::Any::default());
+ added_tx
+ .send(client.threadlocal_user_context_extensions().len())
+ .unwrap();
+ end_rx.recv().unwrap();
+ })
+ };
+ assert_eq!(added_rx.recv().unwrap(), 1);
+ assert_eq!(
+ client
+ .user_context_extensions
+ .lock()
+ .unwrap()
+ .by_thread
+ .len(),
+ 1
+ );
+ end_tx.send(()).unwrap();
+ child.join().unwrap();
+ assert!(client
+ .user_context_extensions
+ .lock()
+ .unwrap()
+ .by_thread
+ .is_empty());
+ }
}
diff --git a/crates/spark-connect-core/src/lib.rs
b/crates/spark-connect-core/src/lib.rs
index 6702ccb2..cf5b33c8 100644
--- a/crates/spark-connect-core/src/lib.rs
+++ b/crates/spark-connect-core/src/lib.rs
@@ -12,6 +12,9 @@ pub mod error;
pub mod reattach;
pub mod retries;
pub mod runtime;
+// Shared with `apache-spark-connect`, whose session keeps per-thread tags the
same way.
+#[doc(hidden)]
+pub mod thread_entries;
pub use artifact::{
build_artifact_request_stream, compute_crc32, ArtifactData, FileArtifact,
InMemoryArtifact,
diff --git a/crates/spark-connect-core/src/thread_entries.rs
b/crates/spark-connect-core/src/thread_entries.rs
new file mode 100644
index 00000000..55038a17
--- /dev/null
+++ b/crates/spark-connect-core/src/thread_entries.rs
@@ -0,0 +1,170 @@
+//! Entries of a map per thread, which end with their thread.
+//!
+//! PySpark keeps the per-thread state of its client in a `threading.local()`,
which is freed
+//! when its thread ends. A map from thread to entry must do the same, or
every thread that ends
+//! leaves its entry behind. [`forget_when_thread_ends`] registers the current
thread with the
+//! map when it inserts its entry, and the entry is removed from the map when
the thread ends.
+//! The registration holds the map only weakly, so a map that is dropped is
not kept alive.
+
+use std::cell::RefCell;
+use std::sync::{Arc, Mutex, Weak};
+use std::thread::ThreadId;
+
+/// A map that a thread has an entry of, which is removed when the thread ends.
+trait Registered {
+ fn is_alive(&self) -> bool;
+ fn address(&self) -> usize;
+ fn forget(&self, thread: ThreadId);
+}
+
+struct Owned<T: Send + 'static> {
+ owner: Weak<Mutex<T>>,
+ forget: fn(&mut T, ThreadId),
+}
+
+impl<T: Send + 'static> Registered for Owned<T> {
+ fn is_alive(&self) -> bool {
+ self.owner.strong_count() > 0
+ }
+
+ fn address(&self) -> usize {
+ self.owner.as_ptr() as usize
+ }
+
+ fn forget(&self, thread: ThreadId) {
+ if let Some(owner) = self.owner.upgrade() {
+ let mut value = owner.lock().unwrap_or_else(|e| e.into_inner());
+ (self.forget)(&mut value, thread);
+ }
+ }
+}
+
+/// The maps that the current thread has entries of. Its drop, when the thread
ends, removes
+/// those entries.
+#[derive(Default)]
+struct ThreadOwners {
+ thread: Option<ThreadId>,
+ owners: Vec<Box<dyn Registered>>,
+}
+
+impl Drop for ThreadOwners {
+ fn drop(&mut self) {
+ if let Some(thread) = self.thread {
+ for owner in &self.owners {
+ owner.forget(thread);
+ }
+ }
+ }
+}
+
+thread_local! {
+ static OWNERS: RefCell<ThreadOwners> =
RefCell::new(ThreadOwners::default());
+}
+
+/// Register the current thread with `owner`, a map that the thread has just
inserted its entry
+/// of: when the thread ends, `forget` removes that entry from the map, which
is then no longer
+/// kept alive by the registration. Registering again, as after the entry was
removed and
+/// inserted once more, is a no-op.
+pub fn forget_when_thread_ends<T: Send + 'static>(
+ owner: &Arc<Mutex<T>>,
+ forget: fn(&mut T, ThreadId),
+) {
+ let thread = std::thread::current().id();
+ let address = Arc::as_ptr(owner) as usize;
+ // Thread-local storage that is being destroyed has no registrations left
to make.
+ let _ = OWNERS.try_with(|owners| {
+ let mut owners = owners.borrow_mut();
+ owners.owners.retain(|owner| owner.is_alive());
+ owners.thread.get_or_insert(thread);
+ if !owners.owners.iter().any(|owner| owner.address() == address) {
+ owners.owners.push(Box::new(Owned {
+ owner: Arc::downgrade(owner),
+ forget,
+ }));
+ }
+ });
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+ use std::collections::HashMap;
+ use std::sync::mpsc;
+
+ type Map = Arc<Mutex<HashMap<ThreadId, u32>>>;
+
+ fn forget(map: &mut HashMap<ThreadId, u32>, thread: ThreadId) {
+ map.remove(&thread);
+ }
+
+ fn insert_and_register(map: &Map, value: u32) {
+ let thread = std::thread::current().id();
+ map.lock().unwrap().insert(thread, value);
+ forget_when_thread_ends(map, forget);
+ }
+
+ #[test]
+ fn entries_of_finished_threads_are_removed() {
+ let map: Map = Arc::default();
+ for _ in 0..1000 {
+ let map = map.clone();
+ std::thread::spawn(move || insert_and_register(&map, 1))
+ .join()
+ .unwrap();
+ }
+ assert!(map.lock().unwrap().is_empty());
+ let owners = OWNERS.with(|owners| owners.borrow().owners.len());
+ assert_eq!(owners, 0);
+ }
+
+ #[test]
+ fn entry_of_a_living_thread_stays_until_it_ends() {
+ let map: Map = Arc::default();
+ let (inserted_tx, inserted_rx) = mpsc::channel();
+ let (end_tx, end_rx) = mpsc::channel::<()>();
+ let child = {
+ let map = map.clone();
+ std::thread::spawn(move || {
+ insert_and_register(&map, 7);
+ // Registering again is a no-op: the thread is registered once
with the map.
+ forget_when_thread_ends(&map, forget);
+ let owners = OWNERS.with(|owners|
owners.borrow().owners.len());
+ inserted_tx.send(owners).unwrap();
+ end_rx.recv().unwrap();
+ })
+ };
+ assert_eq!(inserted_rx.recv().unwrap(), 1);
+ assert_eq!(
+ map.lock().unwrap().values().copied().collect::<Vec<_>>(),
+ [7]
+ );
+ end_tx.send(()).unwrap();
+ child.join().unwrap();
+ assert!(map.lock().unwrap().is_empty());
+ }
+
+ #[test]
+ fn registration_does_not_keep_a_map_alive() {
+ let map: Map = Arc::default();
+ let weak = Arc::downgrade(&map);
+ std::thread::spawn(move || insert_and_register(&map, 1))
+ .join()
+ .unwrap();
+ assert!(weak.upgrade().is_none());
+ }
+
+ #[test]
+ fn the_registration_of_a_dropped_map_is_pruned() {
+ std::thread::spawn(|| {
+ let first: Map = Arc::default();
+ insert_and_register(&first, 1);
+ drop(first);
+ let second: Map = Arc::default();
+ insert_and_register(&second, 2);
+ let owners = OWNERS.with(|owners| owners.borrow().owners.len());
+ assert_eq!(owners, 1);
+ })
+ .join()
+ .unwrap();
+ }
+}
diff --git a/crates/spark-connect/src/session.rs
b/crates/spark-connect/src/session.rs
index 0ec881f3..caf94421 100644
--- a/crates/spark-connect/src/session.rs
+++ b/crates/spark-connect/src/session.rs
@@ -9,6 +9,7 @@ use spark_connect_core::channel::ChannelBuilder;
use spark_connect_core::client::{ExecutePlanHook, SparkConnectClient};
use spark_connect_core::error::{Result, SparkError};
use spark_connect_core::runtime::block_on;
+use spark_connect_core::thread_entries::forget_when_thread_ends;
use spark_connect_proto as proto;
use crate::dataframe::DataFrame;
@@ -46,7 +47,15 @@ pub struct ExecutionInfo {
/// the set of tags that it copied.
#[derive(Default)]
pub(crate) struct ThreadTags {
- by_thread: Mutex<std::collections::HashMap<std::thread::ThreadId,
SharedTags>>,
+ by_thread: Arc<Mutex<std::collections::HashMap<std::thread::ThreadId,
SharedTags>>>,
+}
+
+/// Forget the tags of a thread that ends, as PySpark's `thread_local` is
freed with its thread.
+fn forget_thread_tags(
+ by_thread: &mut std::collections::HashMap<std::thread::ThreadId,
SharedTags>,
+ thread: std::thread::ThreadId,
+) {
+ by_thread.remove(&thread);
}
/// A set of operation tags of threads: a copy of the tags of a thread, of
@@ -76,6 +85,9 @@ impl ThreadTags {
fn update(&self, change: impl FnOnce(&mut Vec<String>)) {
let mut by_thread = self.by_thread.lock().unwrap_or_else(|e|
e.into_inner());
let thread = std::thread::current().id();
+ if !by_thread.contains_key(&thread) {
+ forget_when_thread_ends(&self.by_thread, forget_thread_tags);
+ }
let shared = by_thread.entry(thread).or_default().clone();
let mut tags = shared.0.lock().unwrap_or_else(|e| e.into_inner());
change(&mut tags);
@@ -102,10 +114,12 @@ impl ThreadTags {
/// Make `tags` those of the current thread.
fn share(&self, tags: &SharedTags) {
- self.by_thread
- .lock()
- .unwrap_or_else(|e| e.into_inner())
- .insert(std::thread::current().id(), tags.clone());
+ let mut by_thread = self.by_thread.lock().unwrap_or_else(|e|
e.into_inner());
+ let thread = std::thread::current().id();
+ if !by_thread.contains_key(&thread) {
+ forget_when_thread_ends(&self.by_thread, forget_thread_tags);
+ }
+ by_thread.insert(thread, tags.clone());
}
}
@@ -2705,6 +2719,49 @@ mod tests {
assert_eq!(run(copy, "e"), vec!["a", "b", "e"]);
}
+ #[test]
+ fn finished_threads_leave_no_tags() {
+ // The threads that share a copy of the tags are forgotten when they
end, as PySpark's
+ // thread_local is freed with its thread: 1000 short threads leave no
entry.
+ let spark = session();
+ let copy = spark.copy_thread_tags();
+ for _ in 0..1000 {
+ let spark = spark.clone();
+ let copy = copy.clone();
+ std::thread::spawn(move || {
+ spark.share_thread_tags(©);
+ spark.add_tag("task").unwrap();
+ })
+ .join()
+ .unwrap();
+ }
+ assert!(spark.tags.by_thread.lock().unwrap().is_empty());
+ assert!(spark.get_tags().is_empty());
+ }
+
+ #[test]
+ fn a_living_thread_keeps_its_tags_until_it_ends() {
+ let spark = session();
+ let copy = spark.copy_thread_tags();
+ let (tags_tx, tags_rx) = std::sync::mpsc::channel();
+ let (end_tx, end_rx) = std::sync::mpsc::channel::<()>();
+ let child = {
+ let spark = spark.clone();
+ std::thread::spawn(move || {
+ spark.share_thread_tags(©);
+ spark.add_tag("a").unwrap();
+ tags_tx.send(spark.get_tags()).unwrap();
+ end_rx.recv().unwrap();
+ spark.get_tags()
+ })
+ };
+ assert_eq!(tags_rx.recv().unwrap(), vec!["a"]);
+ assert_eq!(spark.tags.by_thread.lock().unwrap().len(), 1);
+ end_tx.send(()).unwrap();
+ assert_eq!(child.join().unwrap(), vec!["a"]);
+ assert!(spark.tags.by_thread.lock().unwrap().is_empty());
+ }
+
#[test]
fn arrow_local_relation_preserves_batches_and_rejects_schema_mismatch() {
use arrow::array::Array;
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]