andygrove opened a new pull request, #25651: URL: https://github.com/apache/datafusion/pull/25651
## Which issue does this PR close? - Part of #25650. ## Rationale for this change `MemoryPool` accounting only covers memory that operators explicitly reserve, so a process can be OOM killed while the pool reports plenty of headroom. As a first step, this makes the gap visible: it logs the difference between what pools have reserved and what's actually allocated, and it's on by default in SLT. It only logs and never fails a test, unlike #22626 which was reverted in #22860. ## What changes are included in this PR? - `datafusion-execution`: `MemoryDriftTracker` and `DriftLoggingPool`. The pool wraps any `MemoryPool` and reports reservation changes to a tracker, which can be shared by many pools. The tracker compares the reserved total against allocated bytes from a caller-supplied `Fn() -> usize`, since DataFusion doesn't choose the global allocator. It logs at `info` each time positive drift rises by 64 MB, naming the pool and consumer, and records the peak. - `sqllogictest`: a `CountingAllocator` global allocator, with every test file's pool wrapped against one process-wide tracker. The peak drift is printed at the end of the run. Disable it with `--memory-drift false`. Test files run concurrently, so the comparison is process-wide rather than per file. Files that `SET datafusion.runtime.memory_limit` replace their pool and drop out of the reserved total (6 of 504 files). Example output from a full local run: ``` Peak memory drift: drift=84.3 MB allocated=85.4 MB reserved=1176.3 KB pool=order.slt consumer=ExternalSorterMerge[0] ``` The allocator batches its counts per thread, so SLT runtime is unchanged locally (7s both with and without it). ## Are these changes tested? Yes. There are unit tests for the tracker and pool, plus a doc test. I also ran the full SLT suite locally with drift logging on and off. ## Are there any user-facing changes? New public types in `datafusion_execution::memory_pool`. No changes to existing APIs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
