andygrove opened a new pull request, #2163: URL: https://github.com/apache/datafusion-ballista/pull/2163
## Summary Add a small `TrackedMemoryPool` wrapper (`ballista/executor/src/memory_pool.rs`) around the per-task `FairSpillPool` that `memory_pool_policy()` builds. It records peak `reserved()` bytes and emits `tracing::info!` events on construction and on `Drop`, so a task's peak memory pool usage lands in executor logs when the task's `RuntimeEnv` drops. Log format (target `ballista_executor::memory_pool`): ``` memory pool created pool_id=<n> limit_bytes=<bytes|infinite|unknown> memory pool dropped pool_id=<n> peak_bytes=<n> limit_bytes=<bytes|infinite|unknown> ``` Correlate the two events by `pool_id` (monotonic within the executor process). ## Why Ballista builds one `FairSpillPool` per task, but there is currently no way to see how close a task got to its per-task ceiling without adding operator-level tracing or a custom pool. When benchmarking SF1000 TPC-H, individual query wall times varied significantly between standalone and in-suite runs, and memory-pool state drift is one hypothesis — but it's hard to test without per-task peak numbers. This makes the numbers observable in-place from executor logs alone. ## What it does not do - No API changes. - No new dependencies. - No behaviour change on the memory-pool hot path: `grow` / `shrink` / `try_grow` / `reserved` / `memory_limit` all delegate; only `grow` and successful `try_grow` additionally push the peak forward via a CAS on an `AtomicUsize`. - No metrics endpoint (Prometheus etc.) — logs only. A metrics-first version would be a natural follow-up. ## Test plan - [x] `cargo test -p ballista-executor` (42 tests pass, including 3 new tests for peak tracking + limit delegation). - [x] `cargo clippy -p ballista-executor -- -D warnings` clean. - [x] `cargo fmt`. - [ ] Reviewer sanity-check: run a query, look for `memory pool dropped` lines with plausible `peak_bytes`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
