Internal Jenkins has submitted this change and it was merged. Change subject: IMPALA-3611: track unused Disk IO buffer memory ......................................................................
IMPALA-3611: track unused Disk IO buffer memory Track I/O buffers against separate MemTrackers. This gives us better visibility into memory consumption from the debug webpage and from MemTracker consumption dumps. The immediate motivation was in trying to determine whether idle memory consumption of an impalad was caused by a memory leak. We add two trackers: for buffers cached in DiskIoMgr's free list, and another for clients that don't provide a MemTracker (the only one is BufferedBlockMgr, which will be removed at some point). The previous code "tracked" the buffers against the process-wide tracker, but it was a no-op outside of ASAN builds since the process-wide tracker took its value from TCMalloc. The test code required fixing because it assumed that buffers were always credited against the DiskIoMgr's tracker. This only made sense when the DiskIoMgr's tracker is the root process-wide tracker. Fix backend test logging for disk-io-mgr-test. Testing: Ran exhaustive tests. Change-Id: I8777cf76f04d34a46f53d53005412e0f1d63b5b7 Reviewed-on: http://gerrit.cloudera.org:8080/3799 Reviewed-by: Dan Hecht <[email protected]> Tested-by: Internal Jenkins --- M be/src/runtime/disk-io-mgr-scan-range.cc M be/src/runtime/disk-io-mgr-test.cc M be/src/runtime/disk-io-mgr.cc M be/src/runtime/disk-io-mgr.h M be/src/runtime/row-batch.cc 5 files changed, 242 insertions(+), 168 deletions(-) Approvals: Internal Jenkins: Verified Dan Hecht: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/3799 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: merged Gerrit-Change-Id: I8777cf76f04d34a46f53d53005412e0f1d63b5b7 Gerrit-PatchSet: 8 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Tim Armstrong <[email protected]> Gerrit-Reviewer: Dan Hecht <[email protected]> Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Tim Armstrong <[email protected]>
