[
https://issues.apache.org/jira/browse/IMPALA-15022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18087724#comment-18087724
]
ASF subversion and git services commented on IMPALA-15022:
----------------------------------------------------------
Commit 498d24d982affcc92ebaedf700be2c3156d8c1bc in impala's branch
refs/heads/master from Joe McDonnell
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=498d24d98 ]
IMPALA-15022: Rework tcmalloc metrics to include tcmalloc metadata
Currently, our gperftools tcmalloc metrics like
tcmalloc.total-bytes-reserved do not include the malloc metadata.
Back when the tcmalloc metrics were added to Impala, gperftools
did not expose metrics that would include the malloc metadata.
Since then, gperftools has added the generic.total_physical_bytes
metric, which includes the malloc metadata. Malloc metadata can be
a couple hundred MBs on busy systems, so this is not a major
change, but it seems like a slight improvement to our metrics.
This changes tcmalloc.physical-bytes-reserved to use the
generic.total_physical_bytes metric. It changes
tcmalloc.total-bytes-reserved into a derived metric of
tcmalloc.physical-bytes-reserved + tcmalloc.pageheap-unmapped-bytes.
This brings our metrics in line with the text description from
gperftools used for /memz.
Testing:
- Compared the metrics to the /memz page
Change-Id: I5b89193ed0b488dce1498ba630fec91611d7c4fd
Reviewed-on: http://gerrit.cloudera.org:8080/24400
Reviewed-by: Csaba Ringhofer <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Consider modifying our tcmalloc metrics to include malloc metadata
> ------------------------------------------------------------------
>
> Key: IMPALA-15022
> URL: https://issues.apache.org/jira/browse/IMPALA-15022
> Project: IMPALA
> Issue Type: Task
> Components: Backend
> Affects Versions: Impala 5.0.0
> Reporter: Joe McDonnell
> Priority: Major
>
> Our existing gperftools tcmalloc metrics historically have not included the
> malloc metadata. Back when the tcmalloc metrics were added to Impala,
> gperftools did not expose metrics that would include the malloc metadata.
> Since then, gperftools has added new metrics that do include it, e.g.
> generic.total_physical_bytes. Malloc metadata can be a couple hundred MBs on
> busy systems, so this is not a major change, but it seems like a slight
> improvement to our metrics.
> One up side to making this change is that our metrics would line up exactly
> to the text output from tcmalloc in the /memz page.
> {noformat}
> // ------------------------------------------------
> // MALLOC: 145138800 ( 138.4 MiB) Bytes in use by application (#1)
> // MALLOC: + 1056768 ( 1.0 MiB) Bytes in page heap freelist (#2)
> // MALLOC: + 1929472 ( 1.8 MiB) Bytes in central cache freelist
> (#3)
> // MALLOC: + 7655168 ( 7.3 MiB) Bytes in transfer cache freelist
> (#4)
> // MALLOC: + 6626192 ( 6.3 MiB) Bytes in thread cache freelists
> (#5)
> // MALLOC: + 6160448 ( 5.9 MiB) Bytes in malloc metadata
> // MALLOC: ------------
> // MALLOC: = 168566848 ( 160.8 MiB) Actual memory used (physical +
> swap) (#6**)
> // MALLOC: + 51183616 ( 48.8 MiB) Bytes released to OS (aka
> unmapped) (#7)
> // MALLOC: ------------
> // MALLOC: = 219750464 ( 209.6 MiB) Virtual address space used (#8**)
> // MALLOC:
> // MALLOC: 4354 Spans in use
> // MALLOC: 790 Thread heaps in use
> // MALLOC: 8192 Tcmalloc page size{noformat}
> #1 = tcmalloc.bytes-in-use (generic.current_allocated_bytes)
> #2 = tcmalloc.pageheap-free-bytes (tcmalloc.pageheap_free_bytes)
> #3 = tcmalloc.central-cache-free-bytes (tcmalloc.central_cache_free_bytes)
> #4 = tcmalloc.transfer-cache-free-bytes (tcmalloc.transfer_cache_free_bytes)
> #5 = tcmalloc.thread-cache-free-bytes (tcmalloc.thread_cache_free_bytes)
> #6 = tcmalloc.physical-bytes-reserved = #8 - #7 ** Except Impala's metric is
> currently missing the metadata
> #7 = tcmalloc.pageheap-unmapped-bytes (tcmalloc.pageheap_unmapped_bytes)
> #8 = tcmalloc.total-bytes-reserved (generic.heap_size) ** Except Impala's
> metric is currently missing the metadata
> If we reworked it to include the metadata, then #6 and #8 would line up
> nicely with the memz output.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
