[
https://issues.apache.org/jira/browse/TS-2824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phil Sorber resolved TS-2824.
-----------------------------
Resolution: Fixed
Fix Version/s: 5.0.0
Assignee: Phil Sorber
Fixed this by reverting TS-2592 (2fbd4b6)
> ioBufferAllocator objects in ATS master keep growing endlessly
> --------------------------------------------------------------
>
> Key: TS-2824
> URL: https://issues.apache.org/jira/browse/TS-2824
> Project: Traffic Server
> Issue Type: Bug
> Components: Core
> Reporter: Sudheer Vinukonda
> Assignee: Phil Sorber
> Fix For: 5.0.0
>
>
> During our production testing of ATS master, I noticed that ATS memory keeps
> growing beyond ram cache size limits. Upon enabling the memory allocation
> dump, it looks like the ioBufAllocator objects keep growing endlessly.
> Talking to [~psudaemon] and [~amc] on irc, it looks like the issue might be
> caused due to NOT in-use proxy allocator (local thread) objects accumulating
> until they reach the per thread object count limit (250). This may be
> significant memory for large sized objects (e.g 1MB or 2MB). [~psudaemon]
> recommended to try reverting the changes in TS-2592 that force to use Proxy
> allocator instead of global pool objects for ioBuffers. (Note that my
> production host has ram cache set to 6G and that is completely used, but, the
> ioBuffers just keep growing)
> Here's the dump of memory allocations:
> {code}
> -bash-4.1$ sudo grep "ioBufAllocator\[13\]" traffic.out
> 0 | 0 | 1048576 |
> memory/ioBufAllocator[13]
> 1174405120 | 1151336448 | 1048576 |
> memory/ioBufAllocator[13]
> 4194304000 | 4186963968 | 1048576 |
> memory/ioBufAllocator[13]
> 1912602624 | 1908408320 | 1048576 |
> memory/ioBufAllocator[13]
> 771751936 | 759169024 | 1048576 |
> memory/ioBufAllocator[13]
> 1610612736 | 1608515584 | 1048576 |
> memory/ioBufAllocator[13]
> 1509949440 | 1484783616 | 1048576 |
> memory/ioBufAllocator[13]
> 3724541952 | 3697278976 | 1048576 |
> memory/ioBufAllocator[13]
> 2046820352 | 2023751680 | 1048576 |
> memory/ioBufAllocator[13]
> 2046820352 | 2036334592 | 1048576 |
> memory/ioBufAllocator[13]
> 2147483648 | 2128609280 | 1048576 |
> memory/ioBufAllocator[13]
> 2181038080 | 2172649472 | 1048576 |
> memory/ioBufAllocator[13]
> 2214592512 | 2204106752 | 1048576 |
> memory/ioBufAllocator[13]
> 2281701376 | 2253389824 | 1048576 |
> memory/ioBufAllocator[13]
> 2315255808 | 2287992832 | 1048576 |
> memory/ioBufAllocator[13]
> 2348810240 | 2316304384 | 1048576 |
> memory/ioBufAllocator[13]
> 2348810240 | 2336227328 | 1048576 |
> memory/ioBufAllocator[13]
> 2382364672 | 2380267520 | 1048576 |
> memory/ioBufAllocator[13]
> 2449473536 | 2424307712 | 1048576 |
> memory/ioBufAllocator[13]
> 2449473536 | 2447376384 | 1048576 |
> memory/ioBufAllocator[13]
> 2483027968 | 2474639360 | 1048576 |
> memory/ioBufAllocator[13]
> 2516582400 | 2507145216 | 1048576 |
> memory/ioBufAllocator[13]
> 2550136832 | 2543845376 | 1048576 |
> memory/ioBufAllocator[13]
> 2617245696 | 2599419904 | 1048576 |
> memory/ioBufAllocator[13]
> 2650800128 | 2626682880 | 1048576 |
> memory/ioBufAllocator[13]
> 2684354560 | 2659188736 | 1048576 |
> memory/ioBufAllocator[13]
> 2684354560 | 2675965952 | 1048576 |
> memory/ioBufAllocator[13]
> 2717908992 | 2690646016 | 1048576 |
> memory/ioBufAllocator[13]
> 2717908992 | 2713714688 | 1048576 |
> memory/ioBufAllocator[13]
> 2785017856 | 2758803456 | 1048576 |
> memory/ioBufAllocator[13]
> 2818572288 | 2792357888 | 1048576 |
> memory/ioBufAllocator[13]
> 2852126720 | 2823815168 | 1048576 |
> memory/ioBufAllocator[13]
> 2885681152 | 2882535424 | 1048576 |
> memory/ioBufAllocator[13]
> 2952790016 | 2921332736 | 1048576 |
> memory/ioBufAllocator[13]
> 2986344448 | 2959081472 | 1048576 |
> memory/ioBufAllocator[13]
> 2986344448 | 2985295872 | 1048576 |
> memory/ioBufAllocator[13]
> 3019898880 | 3014656000 | 1048576 |
> memory/ioBufAllocator[13]
> 3087007744 | 3067084800 | 1048576 |
> memory/ioBufAllocator[13]
> 3120562176 | 3104833536 | 1048576 |
> memory/ioBufAllocator[13]
> 3154116608 | 3146776576 | 1048576 |
> memory/ioBufAllocator[13]
> 3187671040 | 3186622464 | 1048576 |
> memory/ioBufAllocator[13]
> 3221225472 | 3220176896 | 1048576 |
> memory/ioBufAllocator[13]
> 3288334336 | 3268411392 | 1048576 |
> memory/ioBufAllocator[13]
> 3321888768 | 3292528640 | 1048576 |
> memory/ioBufAllocator[13]
> 3355443200 | 3344957440 | 1048576 |
> memory/ioBufAllocator[13]
> 3388997632 | 3364880384 | 1048576 |
> memory/ioBufAllocator[13]
> 3388997632 | 3387949056 | 1048576 |
> memory/ioBufAllocator[13]
> 3422552064 | 3404726272 | 1048576 |
> memory/ioBufAllocator[13]
> 3456106496 | 3423600640 | 1048576 |
> memory/ioBufAllocator[13]
> Complete memory dump at some point:
> allocated | in-use | type size | free list name
> --------------------|--------------------|------------|----------------------------------
> 201326592 | 174063616 | 2097152 |
> memory/ioBufAllocator[14]
> 3623878656 | 3597664256 | 1048576 |
> memory/ioBufAllocator[13]
> 1811939328 | 1811939328 | 524288 |
> memory/ioBufAllocator[12]
> 1400897536 | 1400635392 | 262144 |
> memory/ioBufAllocator[11]
> 1283457024 | 1281490944 | 131072 |
> memory/ioBufAllocator[10]
> 1663041536 | 1542127616 | 65536 |
> memory/ioBufAllocator[9]
> 1693450240 | 1058897920 | 32768 |
> memory/ioBufAllocator[8]
> 725614592 | 635879424 | 16384 |
> memory/ioBufAllocator[7]
> 398196736 | 357244928 | 8192 |
> memory/ioBufAllocator[6]
> 856686592 | 833167360 | 4096 |
> memory/ioBufAllocator[5]
> 0 | 0 | 2048 |
> memory/ioBufAllocator[4]
> 131072 | 61440 | 1024 |
> memory/ioBufAllocator[3]
> 65536 | 24576 | 512 |
> memory/ioBufAllocator[2]
> 1638400 | 1135616 | 256 |
> memory/ioBufAllocator[1]
> 1081344 | 643072 | 128 |
> memory/ioBufAllocator[0]
> 4325376 | 250368 | 512 |
> memory/FetchSMAllocator
> 0 | 0 | 592 |
> memory/ICPRequestCont_allocator
> 0 | 0 | 112 |
> memory/ICPPeerReadContAllocator
> 0 | 0 | 432 |
> memory/PeerReadDataAllocator
> 4096 | 0 | 32 |
> memory/MIMEFieldSDKHandle
> 0 | 0 | 240 |
> memory/INKVConnAllocator
> 12288 | 1248 | 96 |
> memory/INKContAllocator
> 4096 | 32 | 32 |
> memory/apiHookAllocator
> 0 | 0 | 96 |
> memory/prefetchLockHandlerAllocator
> 0 | 0 | 320 |
> memory/PrefetchBlasterAllocator
> 0 | 0 | 208 |
> memory/prefetchUrlBlaster
> 0 | 0 | 224 | memory/blasterUrlList
> 0 | 0 | 192 |
> memory/prefetchUrlEntryAllocator
> 0 | 0 | 128 |
> memory/socksProxyAllocator
> 0 | 0 | 144 |
> memory/ObjectReloadCont
> 16625664 | 13732048 | 656 |
> memory/httpClientSessionAllocator
> 56150016 | 1593072 | 7696 | memory/httpSMAllocator
> 200704 | 62720 | 224 |
> memory/httpServerSessionAllocator
> 0 | 0 | 48 |
> memory/CacheLookupHttpConfigAllocator
> 0 | 0 | 7744 |
> memory/httpUpdateSMAllocator
> 1892352 | 109312 | 224 |
> memory/spdyRequestAllocator
> 1794048 | 1698432 | 192 |
> memory/spdyClientSessionAllocator
> 0 | 0 | 128 |
> memory/RemapPluginsAlloc
> 0 | 0 | 48 |
> memory/CongestRequestParamAllocator
> 0 | 0 | 144 |
> memory/CongestionDBContAllocator
> 360448 | 1792 | 256 |
> memory/httpCacheAltAllocator
> 41943040 | 20996096 | 2048 | memory/hdrStrHeap
> 58982400 | 23791616 | 2048 | memory/hdrHeap
> 0 | 0 | 112 |
> memory/OneWayTunnelAllocator
> 589824 | 2304 | 2304 |
> memory/hostDBContAllocator
> 203136 | 33856 | 33856 | memory/dnsBufAllocator
> 327680 | 0 | 1280 |
> memory/dnsEntryAllocator
> 0 | 0 | 16 |
> memory/DNSRequestDataAllocator
> 0 | 0 | 48 |
> memory/ClusterVConnectionCache::Entry
> 0 | 0 | 576 |
> memory/cacheContAllocator
> 0 | 0 | 112 |
> memory/inControlAllocator
> 0 | 0 | 112 |
> memory/outControlAllocator
> 0 | 0 | 32 |
> memory/byteBankAllocator
> 0 | 0 | 592 |
> memory/clusterVCAllocator
> 0 | 0 | 48 | memory/evacuationKey
> 6144 | 0 | 48 | memory/cacheRemoveCont
> 208896 | 207552 | 96 | memory/evacuationBlock
> 9621504 | 6307616 | 928 |
> memory/cacheVConnection
> 1474560 | 941280 | 160 | memory/openDirEntry
> 0 | 0 | 96 |
> memory/RamCacheCLFUSEntry
> 8986624 | 8785728 | 64 |
> memory/RamCacheLRUEntry
> 69396480 | 68240160 | 720 |
> memory/sslNetVCAllocator
> 0 | 0 | 128 |
> memory/UDPIOEventAllocator
> 13473792 | 11788192 | 688 | memory/netVCAllocator
> 0 | 0 | 128 |
> memory/udpReadContAllocator
> 0 | 0 | 160 |
> memory/udpPacketAllocator
> 0 | 0 | 384 | memory/socksAllocator
> 14876672 | 12889408 | 64 |
> memory/ioBlockAllocator
> 17006592 | 15877632 | 48 | memory/ioDataAllocator
> 34836480 | 27895920 | 240 | memory/ioAllocator
> 11970560 | 11386880 | 80 | memory/mutexAllocator
> 3170304 | 1586976 | 96 | memory/eventAllocator
> 4456448 | 159744 | 1024 | memory/ArenaBlock
> top command showing traffic_server taking 15g RES mem:
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>
>
> 6274 nobody 20 0 19.9g 15g 8428 S 480.3 65.2 1798:44
> /home/y/bin/traffic_server -M --httpport
> 80:fd=7,81:fd=8,82:fd=9,83:fd=10,84:fd=11,85:fd=12,90:fd=13,80:fd=14:ipv6,81:fd=15
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
