Repository: impala Updated Branches: refs/heads/master 0936e3296 -> 3cbbaf3b3
IMPALA-5848: Account for TCMalloc overhead in MemTracker This patch adds a new MemTracker under the Process MemTracker called "TCMalloc Overhead" which accounts for different cache freelists maintained by TCMalloc. This added accounting also helps bring down the amount of untracked memory. An example dump of the Process MemTracker now looks like: Process: Limit=8.34 GB Total=119.10 MB Peak=119.10 MB Buffer Pool: Free Buffers: Total=0 Buffer Pool: Clean Pages: Total=0 Buffer Pool: Unused Reservation: Total=0 TCMalloc Overhead: Total=11.42 MB Untracked Memory: Total=107.69 MB Testing: Tested manually by checking the memz webpage. Change-Id: I602e9d5e8e8d7470dcfe4addde3265057c16263a Reviewed-on: http://gerrit.cloudera.org:8080/8782 Reviewed-by: Bikramjeet Vig <bikramjeet....@cloudera.com> Tested-by: Impala Public Jenkins Project: http://git-wip-us.apache.org/repos/asf/impala/repo Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/9a6e5fa9 Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/9a6e5fa9 Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/9a6e5fa9 Branch: refs/heads/master Commit: 9a6e5fa99650c4d916aa577742a7758af53f14ee Parents: 0936e32 Author: Bikramjeet Vig <bikramjeet....@cloudera.com> Authored: Wed Dec 6 14:55:40 2017 -0800 Committer: Impala Public Jenkins <impala-public-jenk...@gerrit.cloudera.org> Committed: Wed Dec 13 02:05:21 2017 +0000 ---------------------------------------------------------------------- be/src/runtime/exec-env.cc | 15 +++++++++++++++ 1 file changed, 15 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/impala/blob/9a6e5fa9/be/src/runtime/exec-env.cc ---------------------------------------------------------------------- diff --git a/be/src/runtime/exec-env.cc b/be/src/runtime/exec-env.cc index 6fb572c..2df696e 100644 --- a/be/src/runtime/exec-env.cc +++ b/be/src/runtime/exec-env.cc @@ -336,6 +336,21 @@ Status ExecEnv::Init() { if (!aggressive_decommit_enabled) { return Status("TCMalloc aggressive decommit is required but is disabled."); } + // A MemTracker for TCMalloc overhead which is the difference between the physical bytes + // reserved (TcmallocMetric::PHYSICAL_BYTES_RESERVED) and the bytes in use + // (TcmallocMetrics::BYTES_IN_USE). This overhead accounts for all the cached freelists + // used by TCMalloc. + IntGauge* negated_bytes_in_use = obj_pool_->Add(new NegatedGauge<int64_t>( + MakeTMetricDef("negated_tcmalloc_bytes_in_use", TMetricKind::GAUGE, TUnit::BYTES), + TcmallocMetric::BYTES_IN_USE)); + vector<IntGauge*> overhead_metrics; + overhead_metrics.push_back(negated_bytes_in_use); + overhead_metrics.push_back(TcmallocMetric::PHYSICAL_BYTES_RESERVED); + SumGauge<int64_t>* tcmalloc_overhead = obj_pool_->Add(new SumGauge<int64_t>( + MakeTMetricDef("tcmalloc_overhead", TMetricKind::GAUGE, TUnit::BYTES), + overhead_metrics)); + obj_pool_->Add( + new MemTracker(tcmalloc_overhead, -1, "TCMalloc Overhead", mem_tracker_.get())); #endif mem_tracker_->RegisterMetrics(metrics_.get(), "mem-tracker.process");
