Joe McDonnell created IMPALA-15022:
--------------------------------------
Summary: Consider modifying our tcmalloc metrics to include malloc
metadata
Key: IMPALA-15022
URL: https://issues.apache.org/jira/browse/IMPALA-15022
Project: IMPALA
Issue Type: Task
Components: Backend
Affects Versions: Impala 5.0.0
Reporter: Joe McDonnell
Our existing gperftools tcmalloc metrics historically have not included the
malloc metadata. Back when the tcmalloc metrics were added to Impala,
gperftools did not expose metrics that would include the malloc metadata. Since
then, gperftools has added new metrics that do include it, e.g.
generic.total_physical_bytes. Malloc metadata can be a couple hundred MBs on
busy systems, so this is not a major change, but it seems like a slight
improvement to our metrics.
One up side to making this change is that our metrics would line up exactly to
the text output from tcmalloc in the /memz page.
{noformat}
// ------------------------------------------------
// MALLOC: 145138800 ( 138.4 MiB) Bytes in use by application (#1)
// MALLOC: + 1056768 ( 1.0 MiB) Bytes in page heap freelist (#2)
// MALLOC: + 1929472 ( 1.8 MiB) Bytes in central cache freelist (#3)
// MALLOC: + 7655168 ( 7.3 MiB) Bytes in transfer cache freelist
(#4)
// MALLOC: + 6626192 ( 6.3 MiB) Bytes in thread cache freelists (#5)
// MALLOC: + 6160448 ( 5.9 MiB) Bytes in malloc metadata
// MALLOC: ------------
// MALLOC: = 168566848 ( 160.8 MiB) Actual memory used (physical +
swap) (#6**)
// MALLOC: + 51183616 ( 48.8 MiB) Bytes released to OS (aka unmapped)
(#7)
// MALLOC: ------------
// MALLOC: = 219750464 ( 209.6 MiB) Virtual address space used (#8**)
// MALLOC:
// MALLOC: 4354 Spans in use
// MALLOC: 790 Thread heaps in use
// MALLOC: 8192 Tcmalloc page size{noformat}
#1 = tcmalloc.bytes-in-use (generic.current_allocated_bytes)
#2 = tcmalloc.pageheap-free-bytes (tcmalloc.pageheap_free_bytes)
#3 = tcmalloc.central-cache-free-bytes (tcmalloc.central_cache_free_bytes)
#4 = tcmalloc.transfer-cache-free-bytes (tcmalloc.transfer_cache_free_bytes)
#5 = tcmalloc.thread-cache-free-bytes (tcmalloc.thread_cache_free_bytes)
#6 = tcmalloc.physical-bytes-reserved = #8 - #7 ** Except Impala's metric is
currently missing the metadata
#7 = tcmalloc.pageheap-unmapped-bytes (tcmalloc.pageheap_unmapped_bytes)
#8 = tcmalloc.total-bytes-reserved (generic.heap_size) ** Except Impala's
metric is currently missing the metadata
If we reworked it to include the metadata, then #6 and #8 would line up nicely
with the memz output.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
