aetherxeg-source opened a new pull request, #2775: URL: https://github.com/apache/datafusion-ballista/pull/2775
## Summary Fixes #2767. The executor drops its automatically created `TempDir` before processing queries. Subsequent shuffle writes can recreate that directory with process-default permissions rather than the original private `0700` permissions, potentially exposing shuffle data and leaving files behind after shutdown. This change preserves ownership of the default work directory and addresses the related race where queued, detached, or blocking shuffle writers may still be active during cleanup. ## Changes - Retain the default `TempDir` for the executor lifetime while preserving caller ownership of explicitly configured work directories. - Introduce `WorkDirActivity` and `WorkDirLease` to account for active and queued shuffle writers. - Drain active work and atomically seal admission before private workdir cleanup. - Reject late writers after shutdown sealing. - Preserve private directories when safe quiescence cannot be established. - Add focused regression tests for writer lifetimes, cancellation, timeout, queued work, and cleanup races. ## Validation Qualified against Apache Ballista `main` at commit `1aad53ce9b1a25c18d4f143a07be2995740191fb`. - R11/M1/M2 focused Rust tests: **8/8 PASS** - Workdir regression tests: **2/2 PASS** - R9 shutdown regressions: **3/3 PASS** - `cargo check`: PASS - `cargo clippy -- -D warnings`: PASS - `cargo fmt --all -- --check`: PASS - `git diff --check`: PASS **Total: 13/13 targeted Rust tests passed.** Evidence: [[GitHub Actions qualification run](https://github.com/AETHERXGLOBAL/aether-institutional-memory/actions/runs/41874069380)](https://github.com/AETHERXGLOBAL/aether-institutional-memory/actions/runs/41874069380). Qualified source patch SHA-256: `838a6f3b447f62a0a7cc523e021f76be69d1dcfe17c130078c47e58361a80508`. These results cover the qualified source patch. Upstream PR CI and external integration tests have not yet been completed. ## Limitations The validated scope is Linux/Unix executor workdir and shuffle lifetime behavior. Windows ACL equivalence, hard power-loss recovery, external Scheduler/Flight end-to-end testing, and automated residual-directory cleanup are not yet certified. The proposed `WorkDirLeaseError` API is available for maintainer review. ## Contribution **AETHER X GLOBAL** (under formation). Responsible individual contributor: **Ahmed Younis** (`@aetherxeg-source`). The individual ASF Contributor License Agreement has been submitted to the ASF Secretary. Formal acknowledgement remains pending. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
