Impala Public Jenkins has submitted this change and it was merged.
Change subject: IMPALA-5158,IMPALA-5236: account for unused buffer pool
reservations
......................................................................
IMPALA-5158,IMPALA-5236: account for unused buffer pool reservations
We were missing accounting for this, since it is part of the expected
difference between query and process memory consumption. The identity
that applies is is:
buffers allocated from system =
reservation + cached buffers - unused reservation
Where "cached buffers" includes free buffers and buffers attached to
clean pages. The reservation is accounted against queries and "buffers
allocated from system" is accounted against the process MemTracker.
Reporting this in a direct way required adding a MemTracker with a
negative consumption consumption, which fortunately did not require
any major changes to the MemTracker code.
Example output when applied to buffer pool branch:
Process: Limit=8.35 GB Total=579.18 MB Peak=590.41 MB
Buffer Pool: Free Buffers: Total=268.25 MB
Buffer Pool: Clean Pages: Total=172.25 MB
Buffer Pool: Unused Reservation: Total=-8.25 MB
Free Disk IO Buffers: Total=21.98 MB Peak=21.98 MB
RequestPool=default-pool: Total=12.07 MB Peak=71.58 MB
... <snip> ...
RequestPool=fe-eval-exprs: Total=0 Peak=4.00 KB
Untracked Memory: Total=112.88 MB
Testing:
Added a basic test for MemTrackers with negative metrics.
Change-Id: Idb1fa3110dc893321f9f4e8ced6b7ede12194dad
Reviewed-on: http://gerrit.cloudera.org:8080/7380
Reviewed-by: Tim Armstrong <[email protected]>
Tested-by: Impala Public Jenkins
---
M be/src/runtime/exec-env.cc
M be/src/runtime/mem-tracker-test.cc
M be/src/runtime/mem-tracker.cc
M be/src/runtime/mem-tracker.h
M be/src/util/memory-metrics.cc
M be/src/util/memory-metrics.h
M be/src/util/metrics-test.cc
M be/src/util/metrics.h
M be/src/util/pretty-printer-test.cc
M be/src/util/pretty-printer.h
M common/thrift/metrics.json
11 files changed, 136 insertions(+), 53 deletions(-)
Approvals:
Impala Public Jenkins: Verified
Tim Armstrong: Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/7380
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: Idb1fa3110dc893321f9f4e8ced6b7ede12194dad
Gerrit-PatchSet: 11
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Tim Armstrong <[email protected]>
Gerrit-Reviewer: Dan Hecht <[email protected]>
Gerrit-Reviewer: Henry Robinson <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell <[email protected]>
Gerrit-Reviewer: Matthew Jacobs <[email protected]>
Gerrit-Reviewer: Tim Armstrong <[email protected]>
