andygrove opened a new pull request, #2530: URL: https://github.com/apache/datafusion-ballista/pull/2530
# Which issue does this PR close? No linked issue. This brings Comet's executor allocation observability to Ballista, following apache/datafusion-comet#6162 and apache/datafusion-comet#6166. # Rationale for this change DataFusion memory pool reservations do not account for every allocation in an executor. Logging outstanding Rust allocator bytes makes that otherwise-untracked memory visible when investigating memory usage or sizing executors. # What changes are included in this PR? - Add an `AccountingAllocator` that wraps the existing allocator without changing its allocation behavior or enforcing a memory limit. - Preserve Comet's low-contention accounting: signed process-wide balance, 64 KiB thread-local batching, settlement on thread exit, and a non-panicking fallback during TLS teardown. - Install the wrapper around mimalloc in the standalone executor, with a system-allocator fallback. Embedded executors retain their application's global allocator. - Log outstanding allocations in MiB at INFO level on startup and every 10 seconds using an independent Tokio task. Stop the task when executor startup/execution returns, including error paths. - Add nine deterministic tests covering allocation/free, zeroed and failed allocations, realloc growth/shrink/failure, batching, thread-exit settlement, TLS teardown, and negative-balance reporting. - Document what the measurement includes and excludes. # Are there any user-facing changes? The standalone executor emits a new INFO log line on startup and every 10 seconds: ```text Ballista executor memory usage: allocated 5412.3 MiB (Rust allocator) ``` The value counts requested Rust allocation sizes, not RSS or memory pool reservations. It excludes allocator fragmentation, retained pages, memory mappings, and direct native-library allocations. Per-thread batching makes the value approximate, with less than 64 KiB of unsettled drift per live thread. There are no changes to memory limits or allocator selection. ## Validation - `cargo test --locked --profile ci -p ballista-executor --lib` — all 68 tests pass. - `cargo test --locked --profile ci -p ballista-executor --no-default-features --lib alloc_accounting` — all nine allocator tests pass. - `cargo build --locked --profile ci -p ballista-executor --bin ballista-executor` — passes. - `cargo clippy --locked --profile ci -p ballista-executor --all-targets -- -D warnings` — passes. - `cargo fmt --all -- --check` and Prettier 2.7.1 check — pass. - Global allocator smoke test: a held 32 MiB allocation raises the counter, and dropping it returns the counter to the baseline within the documented batching tolerance. - Executor binary smoke test: two INFO samples 10,000 ms apart while connecting to an unavailable scheduler; a separate immediate connection failure exits without hanging the logging or health tasks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
