[
https://issues.apache.org/jira/browse/TS-3032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14115442#comment-14115442
]
Nikolai Gorchilov commented on TS-3032:
---------------------------------------
I prefer to keep only objects smaller than 64k in RAM, thus
proxy.config.cache.ram_cache_cutoff = 65536. The question was about the
possibility this limit to trigger the un-handled out-of-memory exception I'm
facing.
Total storage is just bellow 8TB and proxy.config.cache.min_average_object_size
= 8000.
RAM cache is 1G (proxy.config.cache.ram_cache.size = 1073741824)
Here's the current ATS memory allocation on this box:
{noformat}
allocated | in-use | type size | free list name
--------------------|--------------------|------------|----------------------------------
67108864 | 10485760 | 2097152 |
memory/ioBufAllocator[14]
738197504 | 679477248 | 1048576 |
memory/ioBufAllocator[13]
100663296 | 58720256 | 524288 |
memory/ioBufAllocator[12]
50331648 | 34340864 | 262144 |
memory/ioBufAllocator[11]
50331648 | 35782656 | 131072 |
memory/ioBufAllocator[10]
312475648 | 270270464 | 65536 | memory/ioBufAllocator[9]
1249902592 | 1039499264 | 32768 | memory/ioBufAllocator[8]
293601280 | 261799936 | 16384 | memory/ioBufAllocator[7]
418381824 | 397361152 | 8192 | memory/ioBufAllocator[6]
1760559104 | 1741463552 | 4096 | memory/ioBufAllocator[5]
786432 | 550912 | 2048 | memory/ioBufAllocator[4]
393216 | 275456 | 1024 | memory/ioBufAllocator[3]
3997696 | 3930624 | 512 | memory/ioBufAllocator[2]
5537792 | 273152 | 256 | memory/ioBufAllocator[1]
13795328 | 1084544 | 128 | memory/ioBufAllocator[0]
2580480 | 320352 | 96 | memory/eventAllocator
29511680 | 29227040 | 80 | memory/mutexAllocator
38346752 | 31291840 | 64 | memory/ioBlockAllocator
27420672 | 26394288 | 48 | memory/ioDataAllocator
97720320 | 97719360 | 240 | memory/ioAllocator
0 | 0 | 384 | memory/socksAllocator
0 | 0 | 128 |
memory/udpReadContAllocator
0 | 0 | 160 |
memory/udpPacketAllocator
256929792 | 254691360 | 672 | memory/netVCAllocator
0 | 0 | 128 |
memory/UDPIOEventAllocator
92160 | 2880 | 720 | memory/sslNetVCAllocator
4546560 | 4426496 | 64 | memory/RamCacheLRUEntry
0 | 0 | 96 |
memory/RamCacheCLFUSEntry
983040 | 964960 | 160 | memory/openDirEntry
14336 | 112 | 112 |
memory/migrateToInterimCache
0 | 0 | 48 | memory/evacuationKey
6144 | 0 | 48 | memory/cacheRemoveCont
2691072 | 2683968 | 96 | memory/evacuationBlock
7004160 | 6917760 | 960 | memory/cacheVConnection
0 | 0 | 32 | memory/byteBankAllocator
0 | 0 | 592 |
memory/clusterVCAllocator
0 | 0 | 112 |
memory/inControlAllocator
0 | 0 | 112 |
memory/outControlAllocator
0 | 0 | 48 |
memory/ClusterVConnectionCache::Entry
0 | 0 | 576 |
memory/cacheContAllocator
0 | 0 | 16 |
memory/DNSRequestDataAllocator
6364928 | 33856 | 33856 | memory/dnsBufAllocator
983040 | 21760 | 1280 | memory/dnsEntryAllocator
7372800 | 46080 | 2304 |
memory/hostDBContAllocator
0 | 0 | 112 |
memory/OneWayTunnelAllocator
64225280 | 64210944 | 2048 | memory/hdrStrHeap
120848384 | 117352448 | 2048 | memory/hdrHeap
163840 | 144896 | 256 |
memory/httpCacheAltAllocator
0 | 0 | 48 |
memory/CongestRequestParamAllocator
0 | 0 | 144 |
memory/CongestionDBContAllocator
0 | 0 | 128 | memory/RemapPluginsAlloc
0 | 0 | 7760 |
memory/httpUpdateSMAllocator
0 | 0 | 48 |
memory/CacheLookupHttpConfigAllocator
10350592 | 10337600 | 224 |
memory/httpServerSessionAllocator
157614080 | 146824288 | 7696 | memory/httpSMAllocator
45846528 | 45783360 | 672 |
memory/httpClientSessionAllocator
0 | 0 | 144 | memory/ObjectReloadCont
0 | 0 | 128 |
memory/socksProxyAllocator
0 | 0 | 96 |
memory/prefetchLockHandlerAllocator
0 | 0 | 320 |
memory/PrefetchBlasterAllocator
0 | 0 | 208 |
memory/prefetchUrlBlaster
0 | 0 | 224 | memory/blasterUrlList
0 | 0 | 192 |
memory/prefetchUrlEntryAllocator
8192 | 0 | 32 |
memory/MIMEFieldSDKHandle
30720 | 0 | 240 | memory/INKVConnAllocator
1388544 | 20160 | 96 | memory/INKContAllocator
20480 | 12896 | 32 | memory/apiHookAllocator
0 | 0 | 592 |
memory/ICPRequestCont_allocator
0 | 0 | 112 |
memory/ICPPeerReadContAllocator
0 | 0 | 432 |
memory/PeerReadDataAllocator
0 | 0 | 496 | memory/FetchSMAllocator
21102592 | 19129344 | 1024 | memory/ArenaBlock
{noformat}
> FATAL: ats_malloc: couldn't allocate XXXXXX bytes
> -------------------------------------------------
>
> Key: TS-3032
> URL: https://issues.apache.org/jira/browse/TS-3032
> Project: Traffic Server
> Issue Type: Bug
> Components: Core
> Affects Versions: 5.0.1
> Reporter: Nikolai Gorchilov
> Assignee: Brian Geffon
> Labels: crash
> Fix For: 5.2.0
>
> Attachments: memory.d.png
>
>
> ATS 5.0.1 under Unbuntu 12.04.4 running happily for days suddenly crashes due
> to memory allocation issue. Happens once or twice a week.
> Server is having plenty of RAM - 128G - out of which 64G+ are free. Nothing
> suspicious in dmesg.
> {noformat}
> FATAL: ats_malloc: couldn't allocate 155648 bytes
> /z/bin/traffic_server - STACK TRACE:
> /z/lib/libtsutil.so.5(+0x1e837)[0x2b6251b3d837]
> /z/lib/libtsutil.so.5(ats_malloc+0x30)[0x2b6251b40c50]
> /z/bin/traffic_server(HdrHeap::coalesce_str_heaps(int)+0x34)[0x62e834]
> /z/bin/traffic_server(http_hdr_clone(HTTPHdrImpl*, HdrHeap*,
> HdrHeap*)+0x8f)[0x62a54f]
> /z/bin/traffic_server(HttpTransactHeaders::copy_header_fields(HTTPHdr*,
> HTTPHdr*, bool, long)+0x1ae)[0x5d08de]
> /z/bin/traffic_server(HttpTransact::build_request(HttpTransact::State*,
> HTTPHdr*, HTTPHdr*, HTTPVersion)+0x5c)[0x5b280c]
> /z/bin/traffic_server(HttpTransact::HandleCacheOpenReadMiss(HttpTransact::State*)+0x2c8)[0x5c2ce8]
> /z/bin/traffic_server(HttpSM::call_transact_and_set_next_state(void
> (*)(HttpTransact::State*))+0x66)[0x58e356]
> /z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
> /z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
> /z/bin/traffic_server(HttpSM::do_hostdb_lookup()+0x27a)[0x58e84a]
> /z/bin/traffic_server(HttpSM::set_next_state()+0xd48)[0x5a1038]
> /z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
> /z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
> /z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
> /z/lib/plugins/x3me_dscp.so(http_txn_hook(tsapi_cont*, TSEvent,
> void*)+0x236)[0x2b626342b508]
> /z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
> /z/bin/traffic_server(HttpSM::state_cache_open_read(int,
> void*)+0x180)[0x59b070]
> /z/bin/traffic_server(HttpSM::main_handler(int, void*)+0xd8)[0x59ad98]
> /z/bin/traffic_server(HttpCacheSM::state_cache_open_read(int,
> void*)+0x173)[0x57bbb3]
> /z/bin/traffic_server(Cache::open_read(Continuation*, INK_MD5*, HTTPHdr*,
> CacheLookupHttpConfig*, CacheFragType, char*, int)+0x616)[0x6d65a6]
> /z/bin/traffic_server(CacheProcessor::open_read(Continuation*, URL*, bool,
> HTTPHdr*, CacheLookupHttpConfig*, long, CacheFragType)+0xb0)[0x6b1af0]
> /z/bin/traffic_server(HttpCacheSM::open_read(URL*, HTTPHdr*,
> CacheLookupHttpConfig*, long)+0x83)[0x57c2d3]
> /z/bin/traffic_server(HttpSM::do_cache_lookup_and_read()+0xfb)[0x58baeb]
> /z/bin/traffic_server(HttpSM::set_next_state()+0x888)[0x5a0b78]
> /z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
> /z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
> /z/bin/traffic_server(HttpSM::set_next_state()+0x7e2)[0x5a0ad2]
> /z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
> /z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
> /z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
> /z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
> /z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
> /z/lib/plugins/cacheurl.so(+0x17dc)[0x2b6263a477dc]
> /z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
> /z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
> /z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
> /z/lib/plugins/tslua.so(+0x596f)[0x2b626363396f]
> /z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
> /z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
> /z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
> /z/lib/plugins/stats_over_http.so(+0x1235)[0x2b6263228235]
> /z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
> /z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
> /z/bin/traffic_server(HttpSM::state_read_client_request_header(int,
> void*)+0x22b)[0x59270b]
> /z/bin/traffic_server(HttpSM::main_handler(int, void*)+0xd8)[0x59ad98]
> /z/bin/traffic_server[0x714a60]
> /z/bin/traffic_server(NetHandler::mainNetEvent(int, Event*)+0x1ed)[0x7077cd]
> /z/bin/traffic_server(EThread::process_event(Event*, int)+0x91)[0x736111]
> /z/bin/traffic_server(EThread::execute()+0x4fc)[0x736bcc]
> /z/bin/traffic_server[0x7353aa]
> /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a)[0x2b6253148e9a]
> /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x2b6253e6873d]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
