[
https://issues.apache.org/jira/browse/TS-4246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15175618#comment-15175618
]
Alan M. Carroll commented on TS-4246:
-------------------------------------
It's unclear to me there is a leak. The number of CacheVC instance in use is
around 20,000 with an almost identical number for HttpSM instances. This is not
a large number for a busy instance of Traffic Server. By far the largest
allocation type in bytes is ioBufAllocator[13] which are 1M IO buffers. This is
the size used by the RAM cache and so it might well be normal RAM cache usage.
Note the 1M IOBuffer usage is roughly 99% of the memory in use. Also, with 22TB
of cache and an average object size of 8000 (the default) it would be around
10G for that.
> ATS possible memory leak in CacheVC
> -----------------------------------
>
> Key: TS-4246
> URL: https://issues.apache.org/jira/browse/TS-4246
> Project: Traffic Server
> Issue Type: Bug
> Components: Cache
> Affects Versions: 6.1.1
> Reporter: taoyunxing
> Labels: CacheVC
> Fix For: 6.1.2
>
>
> CPU:40Cores, Mem:132GB, Hard Disk:300GB sys + 11* 2TB data(naked), Network
> Card: 4*GbE bond, ATS: 5.3.1 or 6.1.1
> Recently I found possible memory leak in production host with ATS version
> from 5.3.1 to 6.1.1, the memory tracker log is as follows:
> {code}
> --------------------------------------------------------------------------------------------
> allocated | in-use | type size | free list name
> --------------------|--------------------|------------|----------------------------------
> 67108864 | 33554432 | 2097152 |
> memory/ioBufAllocator[14]
> 41271951360 | 40564162560 | 1048576 |
> memory/ioBufAllocator[13]
> 570425344 | 566755328 | 524288 |
> memory/ioBufAllocator[12]
> 276824064 | 270794752 | 262144 |
> memory/ioBufAllocator[11]
> 155189248 | 150994944 | 131072 |
> memory/ioBufAllocator[10]
> 113246208 | 111607808 | 65536 |
> memory/ioBufAllocator[9]
> 1319108608 | 1278607360 | 32768 |
> memory/ioBufAllocator[8]
> 39845888 | 39501824 | 16384 |
> memory/ioBufAllocator[7]
> 78381056 | 55869440 | 8192 |
> memory/ioBufAllocator[6]
> 228589568 | 212971520 | 4096 |
> memory/ioBufAllocator[5]
> 262144 | 0 | 2048 |
> memory/ioBufAllocator[4]
> 131072 | 0 | 1024 |
> memory/ioBufAllocator[3]
> 65536 | 0 | 512 |
> memory/ioBufAllocator[2]
> 163840 | 256 | 256 |
> memory/ioBufAllocator[1]
> 1785856 | 768 | 128 |
> memory/ioBufAllocator[0]
> 0 | 0 | 512 |
> memory/FetchSMAllocator
> 0 | 0 | 592 |
> memory/ICPRequestCont_allocator
> 0 | 0 | 128 |
> memory/ICPPeerReadContAllocator
> 0 | 0 | 432 |
> memory/PeerReadDataAllocator
> 0 | 0 | 32 |
> memory/MIMEFieldSDKHandle
> 0 | 0 | 240 |
> memory/INKVConnAllocator
> 0 | 0 | 96 |
> memory/INKContAllocator
> 503808 | 492864 | 32 |
> memory/apiHookAllocator
> 0 | 0 | 128 |
> memory/socksProxyAllocator
> 17211392 | 17169856 | 704 |
> memory/httpClientSessionAllocator
> 180633600 | 170085888 | 8064 | memory/httpSMAllocator
> 2236416 | 2229024 | 224 |
> memory/httpServerSessionAllocator
> 0 | 0 | 48 |
> memory/CacheLookupHttpConfigAllocator
> 0 | 0 | 848 |
> memory/http2ClientSessionAllocator
> 0 | 0 | 128 |
> memory/RemapPluginsAlloc
> 0 | 0 | 48 |
> memory/CongestRequestParamAllocator
> 0 | 0 | 160 |
> memory/CongestionDBContAllocator
> 4325376 | 4245248 | 256 |
> memory/httpCacheAltAllocator
> 155451392 | 146763776 | 2048 | memory/hdrStrHeap
> 269221888 | 253577216 | 2048 | memory/hdrHeap
> 0 | 0 | 128 |
> memory/OneWayTunnelAllocator
> 0 | 0 | 96 |
> memory/hostDBFileContAllocator
> 296960 | 4640 | 2320 |
> memory/hostDBContAllocator
> 135424 | 33856 | 33856 | memory/dnsBufAllocator
> 163840 | 1280 | 1280 |
> memory/dnsEntryAllocator
> 0 | 0 | 16 |
> memory/DNSRequestDataAllocator
> 0 | 0 | 112 |
> memory/inControlAllocator
> 0 | 0 | 128 |
> memory/outControlAllocator
> 0 | 0 | 32 |
> memory/byteBankAllocator
> 0 | 0 | 592 |
> memory/clusterVCAllocator
> 0 | 0 | 48 |
> memory/ClusterVConnectionCache::Entry
> 0 | 0 | 576 |
> memory/cacheContAllocator
> 0 | 0 | 48 | memory/evacuationKey
> 0 | 0 | 64 | memory/cacheRemoveCont
> 380928 | 380928 | 96 | memory/evacuationBlock
> 20003360 | 19959936 | 944 |
> memory/cacheVConnection
> 163840 | 156960 | 160 | memory/openDirEntry
> 2260992 | 2253408 | 96 |
> memory/RamCacheCLFUSEntry
> 0 | 0 | 64 |
> memory/RamCacheLRUEntry
> 0 | 0 | 880 |
> memory/sslNetVCAllocator
> 0 | 0 | 128 |
> memory/UDPIOEventAllocator
> 24399872 | 23407008 | 736 | memory/netVCAllocator
> 0 | 0 | 128 |
> memory/udpReadContAllocator
> 0 | 0 | 160 |
> memory/udpPacketAllocator
> 0 | 0 | 416 | memory/socksAllocator
> 7053312 | 7051072 | 64 |
> memory/ioBlockAllocator
> 7441920 | 7440432 | 48 | memory/ioDataAllocator
> 19355520 | 19350000 | 240 | memory/ioAllocator
> 3402720 | 3306880 | 80 | memory/mutexAllocator
> 147456 | 103680 | 96 | memory/eventAllocator
> 22937600 | 21599232 | 1024 | memory/ArenaBlock
> 6206100608 | 5329728512 | | TOTAL
> Allocs | Frees | Size In-use | Avg Size | Location
>
> -----------|------------|----------------------|------------|--------------------------------------------------------------------
> 39443091 | 39399441 | 178790400 | 4096 |
> memory/IOBuffer/ProtocolProbeSessionAccept.cc:66
> 61 | 61 | 0 | 0 |
> memory/IOBuffer/Transform.cc:791
> 4064 | 4064 | 0 | 0 |
> memory/IOBuffer/HttpClientSession.cc:211
> 10271910 | 10270634 | 23633920 | 18521 |
> memory/IOBuffer/HttpSM.cc:5831
> 261122 | 261122 | 0 | 0 |
> memory/IOBuffer/HttpSM.cc:5880
> 14785 | 14785 | 0 | 0 |
> memory/IOBuffer/HttpSM.cc:5401
> 524803 | 494019 | 1008730112 | 32768 |
> memory/IOBuffer/HttpSM.cc:6199
> 7365218 | 7365218 | 0 | 0 |
> memory/IOBuffer/HttpSM.cc:6021
> 11827313 | 11827056 | 1052672 | 4096 |
> memory/IOBuffer/HttpSM.cc:5750
> 51960076 | 51954854 | 170999808 | 32746 |
> memory/IOBuffer/HttpSM.cc:6302
> 1156 | 1156 | 0 | 0 |
> memory/IOBuffer/HttpSM.cc:6269
> 8806600 | 8803759 | 23273472 | 8192 |
> memory/IOBuffer/HttpServerSession.cc:86
> 708080 | 708074 | 768 | 128 |
> memory/IOBuffer/HttpTunnel.cc:106
> 34649 | 34648 | 256 | 256 |
> memory/IOBuffer/HttpTunnel.cc:111
> 8493 | 8101 | 108728320 | 277368 |
> memory/IOBuffer/Cache.cc:2476
> 31232454 | 31172959 | 41769619456 | 702069 |
> memory/IOBuffer/Cache.cc:2490
> 43284829184 | | TOTAL
> ---------------------------------------------------------------------------------------------
> {code}
> the possilbe memory leak position is in Cache.cc:2490, including new_CacheVC
> and free_CacheVC, and memory leak has 41769619456 bytes. I wonder some
> places forget to call free_CacheVC, or cont->buf.clear(), or anything like
> this, but not sure.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
