JingsongLi opened a new pull request, #832: URL: https://github.com/apache/paimon-rust/pull/832
### Purpose Fix native planning hanging in forked Python workers after the parent has used a Paimon catalog or scan. A Torch DataLoader with `num_workers=2` hits this in apache/paimon#9825: the child inherits the global Tokio runtime but none of its worker threads, so catalog I/O and planning never finish. ### Brief change log - Associate the lazily created fallback runtime with its process ID and publish its state through an atomic pointer. - After fork, initialize a runtime for the child without acquiring a potentially inherited initialization lock or dropping the parent's runtime state. Published states remain allocated because dropping the inherited runtime could wait for threads that no longer exist. - Preserve the existing behavior of using a Tokio runtime already entered on the calling thread. ### Tests - Two real Python fork regressions: create a new catalog or reuse the parent's catalog, plan and serialize splits twice in the child, and confirm the parent can still plan. Both timed out before the fix and pass with it; each has bounded waiting and child cleanup. - Concurrent fallback-runtime reuse and preservation of an entered runtime: 2 Rust unit tests passed. - Python binding fork/read/table/write/catalog tests: 105 passed. - PyPaimon validation with the rebuilt wheel: the explicit-fork DataLoader reproduction completes; 25 contiguous-window dataset tests and 357 planner/reader regressions pass with native planning. - `cargo clippy --offline -p paimon-datafusion --all-targets -- -D warnings`, `cargo fmt --all -- --check`, Python test Flake8, and `git diff --check` passed. ### API and Format No public API or storage-format changes. This fixes the fallback runtime used by catalog and planning calls; explicitly entered Tokio runtimes retain their existing behavior. ### Documentation Updated runtime comments to explain process ownership and why inherited state is retained. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
