[
https://issues.apache.org/jira/browse/KUDU-1772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon resolved KUDU-1772.
-------------------------------
Resolution: Duplicate
Fix Version/s: 1.4.0
See note in KUDU-1953. The throttling and memtrackers are now decoupled.
> Inconsistency between /mem-trackers and /memz
> ---------------------------------------------
>
> Key: KUDU-1772
> URL: https://issues.apache.org/jira/browse/KUDU-1772
> Project: Kudu
> Issue Type: Bug
> Components: supportability
> Affects Versions: 1.1.0
> Reporter: Adar Dembo
> Fix For: 1.4.0
>
>
> The /mem-trackers page shows hierarchical memory consumption according to
> Kudu's memtrackers while the /memz page dumps tcmalloc-specific statistics.
> Unlike other memtrackers, the root tracker's "consumption" is determined by
> asking tcmalloc for the "generic.current_allocated_bytes" stat, so the value
> ought to be the same as what's reported by /memz. A cluster under heavy load
> had a kudu master whose root tracker consumption was ~500MB while /memz
> showed a consumption of ~6GB. After some digging, I think this discrepancy is
> due to the interplay between regular trackers and "observer" trackers.
> First, some background. Regular memory trackers have a consumption counter
> that is updated when memory is consumed or released. Observer trackers (of
> which there is only one: the root tracker) also have a "consumption
> function". At predetermined times, the function is invoked and the return
> value is stored in the consumption counter. These predetermined times are the
> MemTracker Consume(), Release(), and GcMemory() functions. As alluded to
> earlier, memory tracking is hierarchical: when MemTracker::Consume() is
> invoked, all of this tracker's ancestors are also updated, all the way up to
> the root. One would expect this hierarchical update to happen by the way of
> Consume(), but actually, each ancestor's consumption counter is updated
> directly. This is profound for the root tracker: unless Consume() or
> Release() are invoked directly on the root tracker, its consumption function
> is only invoked on GcMemory(). I don't know why it works this way; perhaps
> it's an optimization to prevent the consumption function from being invoked
> too often.
> So, why the discrepancy? This cluster is under heavy load, which means the
> master is seeing some action (most of the load is on the tservers). Normally
> the master's tcmalloc would release memory to the system after 128MB of
> MemTracker::Release(), but perhaps these allocations are all "incidental" and
> thus untracked by Kudu, so tcmalloc isn't releasing them. Meanwhile, there's
> no "real" load on the master (i.e. transactions, or something else that would
> check memory limits), so GcMemory() isn't invoked, and the root tracker's
> consumption is never reset from tcmalloc.
> The main question is whether this is a cosmetic issue or a real one. A server
> under real load (e.g. a tserver) wouldn't experience this. As for the master,
> well, it's not doing much memory limit checking of its own, so the
> discrepancy shouldn't affect ongoing operations.
> Nevertheless, this should be fixed.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
