[jira] [Commented] (TS-3032) FATAL: ats_malloc: couldn't allocate XXXXXX bytes
[ https://issues.apache.org/jira/browse/TS-3032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14108393#comment-14108393 ] Nikolai Gorchilov commented on TS-3032: --- Another manifestation: {noformat} FATAL: ats_malloc: couldn't allocate 65536 bytes /z/bin/traffic_server - STACK TRACE: /z/lib/libtsutil.so.5(+0x1e837)[0x2b0203d6a837] /z/lib/libtsutil.so.5(ats_malloc+0x30)[0x2b0203d6dc50] /z/bin/traffic_server(HdrHeap::allocate_obj(int, int)+0xa4)[0x62e414] /z/bin/traffic_server(mime_hdr_copy_onto(MIMEHdrImpl*, HdrHeap*, MIMEHdrImpl*, HdrHeap*, bool)+0x102)[0x634bc2] /z/bin/traffic_server(http_hdr_copy_onto(HTTPHdrImpl*, HdrHeap*, HTTPHdrImpl*, HdrHeap*, bool)+0x87)[0x62a427] /z/bin/traffic_server(HttpSM::set_next_state()+0x9c1)[0x5a0cc1] /z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c13] /z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c82a] /z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51] /z/lib/plugins/x3me_dscp.so(http_txn_hook(tsapi_cont*, TSEvent, void*)+0x236)[0x2b0211806508] /z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999d2] /z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0538] /z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c13] /z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0538] /z/bin/traffic_server(HttpSM::state_cache_open_read(int, void*)+0x103)[0x59b003] /z/bin/traffic_server(HttpSM::main_handler(int, void*)+0xd8)[0x59ada8] /z/bin/traffic_server(HttpCacheSM::state_cache_open_read(int, void*)+0x1d6)[0x57bc16] /z/bin/traffic_server[0x6d56b8] /z/bin/traffic_server(CacheVC::openReadStartEarliest(int, Event*)+0x7d0)[0x6d9120] /z/bin/traffic_server(CacheVC::handleReadDone(int, Event*)+0x27b)[0x6b79cb] /z/bin/traffic_server(AIOCallbackInternal::io_complete(int, void*)+0x3d)[0x662e5d] /z/bin/traffic_server(EThread::process_event(Event*, int)+0x91)[0x736121] /z/bin/traffic_server(EThread::execute()+0x6a3)[0x736d83] /z/bin/traffic_server[0x7353ba] /lib64/libpthread.so.0(+0x7034)[0x2b0205370034] /lib64/libc.so.6(clone+0x6d)[0x2b02060b475d] {noformat} FATAL: ats_malloc: couldn't allocate XX bytes - Key: TS-3032 URL: https://issues.apache.org/jira/browse/TS-3032 Project: Traffic Server Issue Type: Bug Components: Core Affects Versions: 5.0.1 Reporter: Nikolai Gorchilov Assignee: Brian Geffon Labels: crash Fix For: 5.2.0 ATS 5.0.1 under Unbuntu 12.04.4 running happily for days suddenly crashes due to memory allocation issue. Happens once or twice a week. Server is having plenty of RAM - 128G - out of which 64G+ are free. Nothing suspicious in dmesg. {noformat} FATAL: ats_malloc: couldn't allocate 155648 bytes /z/bin/traffic_server - STACK TRACE: /z/lib/libtsutil.so.5(+0x1e837)[0x2b6251b3d837] /z/lib/libtsutil.so.5(ats_malloc+0x30)[0x2b6251b40c50] /z/bin/traffic_server(HdrHeap::coalesce_str_heaps(int)+0x34)[0x62e834] /z/bin/traffic_server(http_hdr_clone(HTTPHdrImpl*, HdrHeap*, HdrHeap*)+0x8f)[0x62a54f] /z/bin/traffic_server(HttpTransactHeaders::copy_header_fields(HTTPHdr*, HTTPHdr*, bool, long)+0x1ae)[0x5d08de] /z/bin/traffic_server(HttpTransact::build_request(HttpTransact::State*, HTTPHdr*, HTTPHdr*, HTTPVersion)+0x5c)[0x5b280c] /z/bin/traffic_server(HttpTransact::HandleCacheOpenReadMiss(HttpTransact::State*)+0x2c8)[0x5c2ce8] /z/bin/traffic_server(HttpSM::call_transact_and_set_next_state(void (*)(HttpTransact::State*))+0x66)[0x58e356] /z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03] /z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528] /z/bin/traffic_server(HttpSM::do_hostdb_lookup()+0x27a)[0x58e84a] /z/bin/traffic_server(HttpSM::set_next_state()+0xd48)[0x5a1038] /z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03] /z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a] /z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51] /z/lib/plugins/x3me_dscp.so(http_txn_hook(tsapi_cont*, TSEvent, void*)+0x236)[0x2b626342b508] /z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2] /z/bin/traffic_server(HttpSM::state_cache_open_read(int, void*)+0x180)[0x59b070] /z/bin/traffic_server(HttpSM::main_handler(int, void*)+0xd8)[0x59ad98] /z/bin/traffic_server(HttpCacheSM::state_cache_open_read(int, void*)+0x173)[0x57bbb3] /z/bin/traffic_server(Cache::open_read(Continuation*, INK_MD5*, HTTPHdr*, CacheLookupHttpConfig*, CacheFragType, char*, int)+0x616)[0x6d65a6] /z/bin/traffic_server(CacheProcessor::open_read(Continuation*, URL*, bool, HTTPHdr*, CacheLookupHttpConfig*, long, CacheFragType)+0xb0)[0x6b1af0] /z/bin/traffic_server(HttpCacheSM::open_read(URL*, HTTPHdr*,
[jira] [Commented] (TS-3032) FATAL: ats_malloc: couldn't allocate XXXXXX bytes
[ https://issues.apache.org/jira/browse/TS-3032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14108410#comment-14108410 ] Zhao Yongming commented on TS-3032: --- I don't know who have any sucess story on BIG memory system, I'd like to hear if any. for the problem you have, please attach some more data such as: 1. /proc/meminfo 2. the traffic_server process status: /proc//status 3. more system log related to alloc and memory, such as dmesg syslog and, please tell us your configure options when building the binary too. hopes that will help us inspect the problem. FATAL: ats_malloc: couldn't allocate XX bytes - Key: TS-3032 URL: https://issues.apache.org/jira/browse/TS-3032 Project: Traffic Server Issue Type: Bug Components: Core Affects Versions: 5.0.1 Reporter: Nikolai Gorchilov Assignee: Brian Geffon Labels: crash Fix For: 5.2.0 ATS 5.0.1 under Unbuntu 12.04.4 running happily for days suddenly crashes due to memory allocation issue. Happens once or twice a week. Server is having plenty of RAM - 128G - out of which 64G+ are free. Nothing suspicious in dmesg. {noformat} FATAL: ats_malloc: couldn't allocate 155648 bytes /z/bin/traffic_server - STACK TRACE: /z/lib/libtsutil.so.5(+0x1e837)[0x2b6251b3d837] /z/lib/libtsutil.so.5(ats_malloc+0x30)[0x2b6251b40c50] /z/bin/traffic_server(HdrHeap::coalesce_str_heaps(int)+0x34)[0x62e834] /z/bin/traffic_server(http_hdr_clone(HTTPHdrImpl*, HdrHeap*, HdrHeap*)+0x8f)[0x62a54f] /z/bin/traffic_server(HttpTransactHeaders::copy_header_fields(HTTPHdr*, HTTPHdr*, bool, long)+0x1ae)[0x5d08de] /z/bin/traffic_server(HttpTransact::build_request(HttpTransact::State*, HTTPHdr*, HTTPHdr*, HTTPVersion)+0x5c)[0x5b280c] /z/bin/traffic_server(HttpTransact::HandleCacheOpenReadMiss(HttpTransact::State*)+0x2c8)[0x5c2ce8] /z/bin/traffic_server(HttpSM::call_transact_and_set_next_state(void (*)(HttpTransact::State*))+0x66)[0x58e356] /z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03] /z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528] /z/bin/traffic_server(HttpSM::do_hostdb_lookup()+0x27a)[0x58e84a] /z/bin/traffic_server(HttpSM::set_next_state()+0xd48)[0x5a1038] /z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03] /z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a] /z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51] /z/lib/plugins/x3me_dscp.so(http_txn_hook(tsapi_cont*, TSEvent, void*)+0x236)[0x2b626342b508] /z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2] /z/bin/traffic_server(HttpSM::state_cache_open_read(int, void*)+0x180)[0x59b070] /z/bin/traffic_server(HttpSM::main_handler(int, void*)+0xd8)[0x59ad98] /z/bin/traffic_server(HttpCacheSM::state_cache_open_read(int, void*)+0x173)[0x57bbb3] /z/bin/traffic_server(Cache::open_read(Continuation*, INK_MD5*, HTTPHdr*, CacheLookupHttpConfig*, CacheFragType, char*, int)+0x616)[0x6d65a6] /z/bin/traffic_server(CacheProcessor::open_read(Continuation*, URL*, bool, HTTPHdr*, CacheLookupHttpConfig*, long, CacheFragType)+0xb0)[0x6b1af0] /z/bin/traffic_server(HttpCacheSM::open_read(URL*, HTTPHdr*, CacheLookupHttpConfig*, long)+0x83)[0x57c2d3] /z/bin/traffic_server(HttpSM::do_cache_lookup_and_read()+0xfb)[0x58baeb] /z/bin/traffic_server(HttpSM::set_next_state()+0x888)[0x5a0b78] /z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03] /z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528] /z/bin/traffic_server(HttpSM::set_next_state()+0x7e2)[0x5a0ad2] /z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03] /z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528] /z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03] /z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a] /z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51] /z/lib/plugins/cacheurl.so(+0x17dc)[0x2b6263a477dc] /z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2] /z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a] /z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51] /z/lib/plugins/tslua.so(+0x596f)[0x2b626363396f] /z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2] /z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a] /z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51] /z/lib/plugins/stats_over_http.so(+0x1235)[0x2b6263228235] /z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2] /z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528] /z/bin/traffic_server(HttpSM::state_read_client_request_header(int,