[
https://issues.apache.org/jira/browse/IMPALA-15022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joe McDonnell resolved IMPALA-15022.
------------------------------------
Fix Version/s: Impala 5.0.0
Assignee: Joe McDonnell
Resolution: Fixed
> Consider modifying our tcmalloc metrics to include malloc metadata
> ------------------------------------------------------------------
>
> Key: IMPALA-15022
> URL: https://issues.apache.org/jira/browse/IMPALA-15022
> Project: IMPALA
> Issue Type: Task
> Components: Backend
> Affects Versions: Impala 5.0.0
> Reporter: Joe McDonnell
> Assignee: Joe McDonnell
> Priority: Major
> Fix For: Impala 5.0.0
>
>
> Our existing gperftools tcmalloc metrics historically have not included the
> malloc metadata. Back when the tcmalloc metrics were added to Impala,
> gperftools did not expose metrics that would include the malloc metadata.
> Since then, gperftools has added new metrics that do include it, e.g.
> generic.total_physical_bytes. Malloc metadata can be a couple hundred MBs on
> busy systems, so this is not a major change, but it seems like a slight
> improvement to our metrics.
> One up side to making this change is that our metrics would line up exactly
> to the text output from tcmalloc in the /memz page.
> {noformat}
> // ------------------------------------------------
> // MALLOC: 145138800 ( 138.4 MiB) Bytes in use by application (#1)
> // MALLOC: + 1056768 ( 1.0 MiB) Bytes in page heap freelist (#2)
> // MALLOC: + 1929472 ( 1.8 MiB) Bytes in central cache freelist
> (#3)
> // MALLOC: + 7655168 ( 7.3 MiB) Bytes in transfer cache freelist
> (#4)
> // MALLOC: + 6626192 ( 6.3 MiB) Bytes in thread cache freelists
> (#5)
> // MALLOC: + 6160448 ( 5.9 MiB) Bytes in malloc metadata
> // MALLOC: ------------
> // MALLOC: = 168566848 ( 160.8 MiB) Actual memory used (physical +
> swap) (#6**)
> // MALLOC: + 51183616 ( 48.8 MiB) Bytes released to OS (aka
> unmapped) (#7)
> // MALLOC: ------------
> // MALLOC: = 219750464 ( 209.6 MiB) Virtual address space used (#8**)
> // MALLOC:
> // MALLOC: 4354 Spans in use
> // MALLOC: 790 Thread heaps in use
> // MALLOC: 8192 Tcmalloc page size{noformat}
> #1 = tcmalloc.bytes-in-use (generic.current_allocated_bytes)
> #2 = tcmalloc.pageheap-free-bytes (tcmalloc.pageheap_free_bytes)
> #3 = tcmalloc.central-cache-free-bytes (tcmalloc.central_cache_free_bytes)
> #4 = tcmalloc.transfer-cache-free-bytes (tcmalloc.transfer_cache_free_bytes)
> #5 = tcmalloc.thread-cache-free-bytes (tcmalloc.thread_cache_free_bytes)
> #6 = tcmalloc.physical-bytes-reserved = #8 - #7 ** Except Impala's metric is
> currently missing the metadata
> #7 = tcmalloc.pageheap-unmapped-bytes (tcmalloc.pageheap_unmapped_bytes)
> #8 = tcmalloc.total-bytes-reserved (generic.heap_size) ** Except Impala's
> metric is currently missing the metadata
> If we reworked it to include the metadata, then #6 and #8 would line up
> nicely with the memz output.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
