Tim Armstrong created IMPALA-5848:
-------------------------------------
Summary: Account for TCMalloc overhead and client cache buffers in
MemTracker
Key: IMPALA-5848
URL: https://issues.apache.org/jira/browse/IMPALA-5848
Project: IMPALA
Issue Type: Improvement
Components: Backend
Affects Versions: Impala 2.10.0
Reporter: Tim Armstrong
Assignee: Tim Armstrong
On our stress test cluster I see a significant amount of Untracked memory. I
think we can account for a chunk of it, which will further help understanding
the state of a cluster.
I see:
{code}
Untracked Memory: Total=4.30 GB
{code}
I think some of this memory is TCMalloc caches and metadata (I'm not sure
exactly which is counted)
{code}
MALLOC: 61821970968 (58958.0 MiB) Bytes in use by application
MALLOC: + 0 ( 0.0 MiB) Bytes in page heap freelist
MALLOC: + 808683664 ( 771.2 MiB) Bytes in central cache freelist
MALLOC: + 3794688 ( 3.6 MiB) Bytes in transfer cache freelist
MALLOC: + 520586840 ( 496.5 MiB) Bytes in thread cache freelists
MALLOC: + 182366400 ( 173.9 MiB) Bytes in malloc metadata
MALLOC: ------------
MALLOC: = 63337402560 (60403.3 MiB) Actual memory used (physical + swap)
MALLOC: + 45341532160 (43241.1 MiB) Bytes released to OS (aka unmapped)
MALLOC: ------------
MALLOC: = 108678934720 (103644.3 MiB) Virtual address space used
MALLOC:
MALLOC: 368033 Spans in use
MALLOC: 3511 Thread heaps in use
MALLOC: 8192 Tcmalloc page size
------------------------------------------------
Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()).
Bytes released to the OS take up virtual address space but no physical memory.
{code}
The client cache also is likely taking up a significant amount of memory, since
I believe the clients all have buffers attached:
{code}
impala-server.backends.client-cache.total-clients 1006
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
