[
https://issues.apache.org/jira/browse/TS-3032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14109294#comment-14109294
]
Nikolai Gorchilov commented on TS-3032:
---------------------------------------
Yes, I have two boxes that have this issue and one of them is available for
testing. Have in mind it has just 48G of RAM.
Here's the relevant info on this box just after restarting the new build of
traffic server with requested config changes:
* cat /proc/meminfo
{noformat}
MemTotal: 49361540 kB
MemFree: 36481844 kB
MemAvailable: 37318832 kB
Buffers: 139888 kB
Cached: 1174072 kB
SwapCached: 0 kB
Active: 11619048 kB
Inactive: 472492 kB
Active(anon): 10944848 kB
Inactive(anon): 43404 kB
Active(file): 674200 kB
Inactive(file): 429088 kB
Unevictable: 175688 kB
Mlocked: 12852 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 0 kB
Writeback: 0 kB
AnonPages: 10953252 kB
Mapped: 61956 kB
Shmem: 44052 kB
Slab: 513432 kB
SReclaimable: 143176 kB
SUnreclaim: 370256 kB
KernelStack: 7728 kB
PageTables: 29912 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 24680768 kB
Committed_AS: 11943052 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 285388 kB
VmallocChunk: 34359406432 kB
AnonHugePages: 10416128 kB
DirectMap4k: 13824 kB
DirectMap2M: 4171776 kB
DirectMap1G: 48234496 kB
{noformat}
* /proc/<pid>/status after traffic_server restart
{noformat}
Name: [ET_NET 0]
State: S (sleeping)
Tgid: 24406
Ngid: 0
Pid: 24406
PPid: 24397
TracerPid: 0
Uid: 13 13 13 13
Gid: 13 13 13 13
FDSize: 128
Groups: 0
VmPeak: 19229780 kB
VmSize: 19229272 kB
VmLck: 0 kB
VmPin: 0 kB
VmHWM: 10616928 kB
VmRSS: 10616928 kB
VmData: 19129068 kB
VmStk: 136 kB
VmExe: 4096 kB
VmLib: 10736 kB
VmPTE: 21840 kB
VmSwap: 0 kB
Threads: 162
SigQ: 0/192701
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000000381000
SigCgt: 0000000180004e4f
CapInh: 0000000000000000
CapPrm: 0000000000005402
CapEff: 0000000000005400
CapBnd: 0000001fffffffff
Cpus_allowed: ffff
Cpus_allowed_list: 0-15
voluntary_ctxt_switches: 26618
nonvoluntary_ctxt_switches: 135
{noformat}
* relevant records.config options:
{noformat}
CONFIG proxy.config.allocator.enable_reclaim INT 1
CONFIG proxy.config.cache.ram_cache_cutoff INT 65536
CONFIG proxy.config.cache.ram_cache.size INT 1073741824
CONFIG proxy.config.cache.ram_cache.algorithm INT 1
{noformat}
NB! Could it be related to ram_cache_cutoff? Seems all crashes are for
allocations above 64k.
> FATAL: ats_malloc: couldn't allocate XXXXXX bytes
> -------------------------------------------------
>
> Key: TS-3032
> URL: https://issues.apache.org/jira/browse/TS-3032
> Project: Traffic Server
> Issue Type: Bug
> Components: Core
> Affects Versions: 5.0.1
> Reporter: Nikolai Gorchilov
> Assignee: Brian Geffon
> Labels: crash
> Fix For: 5.2.0
>
>
> ATS 5.0.1 under Unbuntu 12.04.4 running happily for days suddenly crashes due
> to memory allocation issue. Happens once or twice a week.
> Server is having plenty of RAM - 128G - out of which 64G+ are free. Nothing
> suspicious in dmesg.
> {noformat}
> FATAL: ats_malloc: couldn't allocate 155648 bytes
> /z/bin/traffic_server - STACK TRACE:
> /z/lib/libtsutil.so.5(+0x1e837)[0x2b6251b3d837]
> /z/lib/libtsutil.so.5(ats_malloc+0x30)[0x2b6251b40c50]
> /z/bin/traffic_server(HdrHeap::coalesce_str_heaps(int)+0x34)[0x62e834]
> /z/bin/traffic_server(http_hdr_clone(HTTPHdrImpl*, HdrHeap*,
> HdrHeap*)+0x8f)[0x62a54f]
> /z/bin/traffic_server(HttpTransactHeaders::copy_header_fields(HTTPHdr*,
> HTTPHdr*, bool, long)+0x1ae)[0x5d08de]
> /z/bin/traffic_server(HttpTransact::build_request(HttpTransact::State*,
> HTTPHdr*, HTTPHdr*, HTTPVersion)+0x5c)[0x5b280c]
> /z/bin/traffic_server(HttpTransact::HandleCacheOpenReadMiss(HttpTransact::State*)+0x2c8)[0x5c2ce8]
> /z/bin/traffic_server(HttpSM::call_transact_and_set_next_state(void
> (*)(HttpTransact::State*))+0x66)[0x58e356]
> /z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
> /z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
> /z/bin/traffic_server(HttpSM::do_hostdb_lookup()+0x27a)[0x58e84a]
> /z/bin/traffic_server(HttpSM::set_next_state()+0xd48)[0x5a1038]
> /z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
> /z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
> /z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
> /z/lib/plugins/x3me_dscp.so(http_txn_hook(tsapi_cont*, TSEvent,
> void*)+0x236)[0x2b626342b508]
> /z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
> /z/bin/traffic_server(HttpSM::state_cache_open_read(int,
> void*)+0x180)[0x59b070]
> /z/bin/traffic_server(HttpSM::main_handler(int, void*)+0xd8)[0x59ad98]
> /z/bin/traffic_server(HttpCacheSM::state_cache_open_read(int,
> void*)+0x173)[0x57bbb3]
> /z/bin/traffic_server(Cache::open_read(Continuation*, INK_MD5*, HTTPHdr*,
> CacheLookupHttpConfig*, CacheFragType, char*, int)+0x616)[0x6d65a6]
> /z/bin/traffic_server(CacheProcessor::open_read(Continuation*, URL*, bool,
> HTTPHdr*, CacheLookupHttpConfig*, long, CacheFragType)+0xb0)[0x6b1af0]
> /z/bin/traffic_server(HttpCacheSM::open_read(URL*, HTTPHdr*,
> CacheLookupHttpConfig*, long)+0x83)[0x57c2d3]
> /z/bin/traffic_server(HttpSM::do_cache_lookup_and_read()+0xfb)[0x58baeb]
> /z/bin/traffic_server(HttpSM::set_next_state()+0x888)[0x5a0b78]
> /z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
> /z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
> /z/bin/traffic_server(HttpSM::set_next_state()+0x7e2)[0x5a0ad2]
> /z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
> /z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
> /z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
> /z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
> /z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
> /z/lib/plugins/cacheurl.so(+0x17dc)[0x2b6263a477dc]
> /z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
> /z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
> /z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
> /z/lib/plugins/tslua.so(+0x596f)[0x2b626363396f]
> /z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
> /z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
> /z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
> /z/lib/plugins/stats_over_http.so(+0x1235)[0x2b6263228235]
> /z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
> /z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
> /z/bin/traffic_server(HttpSM::state_read_client_request_header(int,
> void*)+0x22b)[0x59270b]
> /z/bin/traffic_server(HttpSM::main_handler(int, void*)+0xd8)[0x59ad98]
> /z/bin/traffic_server[0x714a60]
> /z/bin/traffic_server(NetHandler::mainNetEvent(int, Event*)+0x1ed)[0x7077cd]
> /z/bin/traffic_server(EThread::process_event(Event*, int)+0x91)[0x736111]
> /z/bin/traffic_server(EThread::execute()+0x4fc)[0x736bcc]
> /z/bin/traffic_server[0x7353aa]
> /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a)[0x2b6253148e9a]
> /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x2b6253e6873d]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
