memory management, cut down memory waste ?
------------------------------------------
Key: TS-1006
URL: https://issues.apache.org/jira/browse/TS-1006
Project: Traffic Server
Issue Type: Improvement
Components: Core
Affects Versions: 3.1.1
Reporter: Zhao Yongming
Assignee: Zhao Yongming
Fix For: 3.1.3
when we review the memory usage in the production, there is something abnormal,
ie, looks like TS take much memory than index data + common system waste, and
here is some memory dump result by set "proxy.config.dump_mem_info_frequency"
1, the one on a not so busy forwarding system:
physics memory: 32G
RAM cache: 22G
DISK: 6140 GB
average_object_size 64000
{code}
allocated | in-use | type size | free list name
--------------------|--------------------|------------|----------------------------------
671088640 | 37748736 | 2097152 |
memory/ioBufAllocator[14]
2248146944 | 2135949312 | 1048576 |
memory/ioBufAllocator[13]
1711276032 | 1705508864 | 524288 |
memory/ioBufAllocator[12]
1669332992 | 1667760128 | 262144 |
memory/ioBufAllocator[11]
2214592512 | 2211840000 | 131072 |
memory/ioBufAllocator[10]
2325741568 | 2323775488 | 65536 | memory/ioBufAllocator[9]
2091909120 | 2089123840 | 32768 | memory/ioBufAllocator[8]
1956642816 | 1956478976 | 16384 | memory/ioBufAllocator[7]
2094530560 | 2094071808 | 8192 | memory/ioBufAllocator[6]
356515840 | 355540992 | 4096 | memory/ioBufAllocator[5]
1048576 | 14336 | 2048 | memory/ioBufAllocator[4]
131072 | 0 | 1024 | memory/ioBufAllocator[3]
65536 | 0 | 512 | memory/ioBufAllocator[2]
32768 | 0 | 256 | memory/ioBufAllocator[1]
16384 | 0 | 128 | memory/ioBufAllocator[0]
0 | 0 | 576 |
memory/ICPRequestCont_allocator
0 | 0 | 112 |
memory/ICPPeerReadContAllocator
0 | 0 | 432 |
memory/PeerReadDataAllocator
0 | 0 | 32 |
memory/MIMEFieldSDKHandle
0 | 0 | 240 | memory/INKVConnAllocator
0 | 0 | 96 | memory/INKContAllocator
4096 | 0 | 32 | memory/apiHookAllocator
0 | 0 | 288 | memory/FetchSMAllocator
0 | 0 | 80 |
memory/prefetchLockHandlerAllocator
0 | 0 | 176 |
memory/PrefetchBlasterAllocator
0 | 0 | 80 |
memory/prefetchUrlBlaster
0 | 0 | 96 | memory/blasterUrlList
0 | 0 | 96 |
memory/prefetchUrlEntryAllocator
0 | 0 | 128 |
memory/socksProxyAllocator
0 | 0 | 144 | memory/ObjectReloadCont
3258368 | 576016 | 592 |
memory/httpClientSessionAllocator
825344 | 139568 | 208 |
memory/httpServerSessionAllocator
22597632 | 1284848 | 9808 | memory/httpSMAllocator
0 | 0 | 32 |
memory/CacheLookupHttpConfigAllocator
0 | 0 | 9856 |
memory/httpUpdateSMAllocator
0 | 0 | 128 | memory/RemapPluginsAlloc
0 | 0 | 48 |
memory/CongestRequestParamAllocator
0 | 0 | 128 |
memory/CongestionDBContAllocator
5767168 | 704512 | 2048 | memory/hdrStrHeap
18350080 | 1153024 | 2048 | memory/hdrHeap
53248 | 2912 | 208 |
memory/httpCacheAltAllocator
0 | 0 | 112 |
memory/OneWayTunnelAllocator
157696 | 1232 | 1232 |
memory/hostDBContAllocator
102240 | 17040 | 17040 | memory/dnsBufAllocator
323584 | 0 | 1264 | memory/dnsEntryAllocator
0 | 0 | 16 |
memory/DNSRequestDataAllocator
0 | 0 | 1072 | memory/SRVAllocator
0 | 0 | 48 |
memory/ClusterVConnectionCache::Entry
0 | 0 | 560 |
memory/cacheContAllocator
0 | 0 | 112 |
memory/inControlAllocator
0 | 0 | 112 |
memory/outControlAllocator
0 | 0 | 32 | memory/byteBankAllocator
0 | 0 | 576 |
memory/clusterVCAllocator
0 | 0 | 48 | memory/evacuationKey
6144 | 0 | 48 | memory/cacheRemoveCont
270336 | 262560 | 96 | memory/evacuationBlock
4997120 | 3968416 | 976 | memory/cacheVConnection
798720 | 522080 | 160 | memory/openDirEntry
0 | 0 | 64 | memory/RamCacheLRUEntry
56426496 | 56426304 | 96 |
memory/RamCacheCLFUSEntry
9584640 | 6168000 | 960 | memory/netVCAllocator
0 | 0 | 128 |
memory/udpReadContAllocator
0 | 0 | 128 |
memory/udpWorkContinuationAllocator
0 | 0 | 160 |
memory/udpPacketAllocator
0 | 0 | 304 | memory/socksAllocator
139264 | 68544 | 1088 | memory/sslNetVCAllocator
0 | 0 | 128 |
memory/UDPIOEventAllocator
671744 | 115520 | 64 | memory/ioBlockAllocator
28305408 | 28301520 | 48 | memory/ioDataAllocator
2273280 | 406320 | 240 | memory/ioAllocator
1904640 | 1489920 | 80 | memory/mutexAllocator
1105920 | 188544 | 96 | memory/eventAllocator
2359296 | 129024 | 1024 | memory/ArenaBlock
{code}
this box will crash every 2days, so the memory waste may no that high
2, our production reverse system:
physics memory: 16G
RAM cache: 8G
DISK: 1516 GB
average_object_size 16384
and it run for a much long time:
{code}
allocated | in-use | type size | free list name
--------------------|--------------------|------------|----------------------------------
805306368 | 0 | 2097152 |
memory/ioBufAllocator[14]
738197504 | 8388608 | 1048576 |
memory/ioBufAllocator[13]
1258291200 | 46661632 | 524288 |
memory/ioBufAllocator[12]
1300234240 | 183762944 | 262144 |
memory/ioBufAllocator[11]
1170210816 | 466223104 | 131072 |
memory/ioBufAllocator[10]
1790967808 | 1223426048 | 65536 | memory/ioBufAllocator[9]
2970615808 | 2601418752 | 32768 | memory/ioBufAllocator[8]
2067791872 | 2044608512 | 16384 | memory/ioBufAllocator[7]
1169424384 | 1169121280 | 8192 | memory/ioBufAllocator[6]
711458816 | 710463488 | 4096 | memory/ioBufAllocator[5]
1572864 | 0 | 2048 | memory/ioBufAllocator[4]
131072 | 0 | 1024 | memory/ioBufAllocator[3]
65536 | 0 | 512 | memory/ioBufAllocator[2]
32768 | 0 | 256 | memory/ioBufAllocator[1]
16384 | 0 | 128 | memory/ioBufAllocator[0]
0 | 0 | 576 |
memory/ICPRequestCont_allocator
0 | 0 | 112 |
memory/ICPPeerReadContAllocator
0 | 0 | 432 |
memory/PeerReadDataAllocator
0 | 0 | 32 |
memory/MIMEFieldSDKHandle
0 | 0 | 240 | memory/INKVConnAllocator
0 | 0 | 96 | memory/INKContAllocator
4096 | 0 | 32 | memory/apiHookAllocator
0 | 0 | 288 | memory/FetchSMAllocator
0 | 0 | 80 |
memory/prefetchLockHandlerAllocator
0 | 0 | 176 |
memory/PrefetchBlasterAllocator
0 | 0 | 80 |
memory/prefetchUrlBlaster
0 | 0 | 96 | memory/blasterUrlList
0 | 0 | 96 |
memory/prefetchUrlEntryAllocator
0 | 0 | 128 |
memory/socksProxyAllocator
0 | 0 | 144 | memory/ObjectReloadCont
1136640 | 125504 | 592 |
memory/httpClientSessionAllocator
372736 | 27248 | 208 |
memory/httpServerSessionAllocator
11317248 | 39296 | 9824 | memory/httpSMAllocator
0 | 0 | 32 |
memory/CacheLookupHttpConfigAllocator
0 | 0 | 9888 |
memory/httpUpdateSMAllocator
0 | 0 | 128 | memory/RemapPluginsAlloc
0 | 0 | 512 | memory/HCSMAllocator
0 | 0 | 48 | memory/VCEntryAllocator
0 | 0 | 96 | memory/HCEntryAllocator
0 | 0 | 64 |
memory/HCHandlerAllocator
0 | 0 | 48 |
memory/CongestRequestParamAllocator
0 | 0 | 128 |
memory/CongestionDBContAllocator
6029312 | 643072 | 2048 | memory/hdrStrHeap
7077888 | 657408 | 2048 | memory/hdrHeap
26624 | 208 | 208 |
memory/httpCacheAltAllocator
0 | 0 | 112 |
memory/OneWayTunnelAllocator
630784 | 1232 | 1232 |
memory/hostDBContAllocator
238560 | 17040 | 17040 | memory/dnsBufAllocator
161792 | 0 | 1264 | memory/dnsEntryAllocator
0 | 0 | 16 |
memory/DNSRequestDataAllocator
0 | 0 | 1072 | memory/SRVAllocator
0 | 0 | 48 |
memory/ClusterVConnectionCache::Entry
0 | 0 | 560 |
memory/cacheContAllocator
0 | 0 | 112 |
memory/inControlAllocator
0 | 0 | 112 |
memory/outControlAllocator
0 | 0 | 32 | memory/byteBankAllocator
0 | 0 | 576 |
memory/clusterVCAllocator
0 | 0 | 48 | memory/evacuationKey
6144 | 0 | 48 | memory/cacheRemoveCont
17006592 | 14972928 | 96 | memory/evacuationBlock
1777664 | 759872 | 992 | memory/cacheVConnection
307200 | 111520 | 160 | memory/openDirEntry
0 | 0 | 64 | memory/RamCacheLRUEntry
104275968 | 104274048 | 96 |
memory/RamCacheCLFUSEntry
3440640 | 1819200 | 960 | memory/netVCAllocator
0 | 0 | 128 |
memory/udpReadContAllocator
0 | 0 | 128 |
memory/udpWorkContinuationAllocator
0 | 0 | 160 |
memory/udpPacketAllocator
0 | 0 | 304 | memory/socksAllocator
0 | 0 | 1088 | memory/sslNetVCAllocator
0 | 0 | 128 |
memory/UDPIOEventAllocator
237568 | 22528 | 64 | memory/ioBlockAllocator
26087424 | 26081904 | 48 | memory/ioDataAllocator
890880 | 84240 | 240 | memory/ioAllocator
1525760 | 1403440 | 80 | memory/mutexAllocator
565248 | 129696 | 96 | memory/eventAllocator
1179648 | 4096 | 1024 | memory/ArenaBlock
{code}
our team is working on the memory free issue, trying to improve the memory
management. and this a big project, the more input|comment the better.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
