james-willis opened a new issue, #1071: URL: https://github.com/apache/sedona-db/issues/1071
### Context `InternalContext` builds a fresh multi-threaded Tokio runtime on every `connect()` (`python/sedonadb/src/context.rs:69`), held as `Arc<Runtime>` and cloned into every `InternalDataFrame`, streaming reader, and exported table provider. PR #1067 fixed a free-threaded-Python deadlock in this area: the runtime's default `Drop` blocking-joins its worker threads, which stalls CPython's stop-the-world under free-threading (PEP 703). The fix makes teardown non-blocking (`shutdown_background`). That closes the deadlock for any usage, but the per-connection runtime remains, and this issue tracks consolidating it. ### Proposal Replace the per-`connect()` `Builder::new_multi_thread()` with a single process-shared runtime (a lazily-initialized `OnceLock<Runtime>` handle), threaded through `InternalContext` / `InternalDataFrame` and the shared `sedona-extension` types (streaming, table provider, execution plan), plus the ADBC path. ### Motivation - **Teardown happens once, at process exit**, instead of on every context drop — shrinking the window in which teardown can coincide with a free-threaded stop-the-world, and simplifying the lifecycle to reason about. - **No per-`connect()` thread-pool spawn.** A fresh multi-thread runtime spawns one worker per core each time. This is wasteful under the connect-per-request / connect-per-work-item anti-pattern (which users do reach for even though `SedonaContext` is session-scoped like a DB connection). - **Protects the invariant the teardown fix relies on.** `shutdown_background` is safe only because there is no in-flight work at teardown; a single long-lived runtime plus a documented + debug-asserted "no in-flight work" invariant keeps that durable as the code grows. ### Why this is low-risk - Per-context **cancellation is already independent of the runtime** — it flows through a per-operation `CancelChecker` (`c/sedona-extension/src/streaming.rs`), not runtime shutdown, so sharing the runtime does not change cancellation semantics. - The **shared-runtime pattern is already used in this repo**: `sedonadb-zarr` (`shared_runtime()` via `OnceLock`) and the R bindings (`OnceLock` global runtime). This aligns `sedonadb` with that. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
