[jira] [Commented] (TS-3032) FATAL: ats_malloc: couldn't allocate XXXXXX bytes

2014-08-24 Thread Nikolai Gorchilov (JIRA)

[ 
https://issues.apache.org/jira/browse/TS-3032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14108393#comment-14108393
 ] 

Nikolai Gorchilov commented on TS-3032:
---

Another manifestation:

{noformat}
FATAL: ats_malloc: couldn't allocate 65536 bytes
/z/bin/traffic_server - STACK TRACE: 
/z/lib/libtsutil.so.5(+0x1e837)[0x2b0203d6a837]
/z/lib/libtsutil.so.5(ats_malloc+0x30)[0x2b0203d6dc50]
/z/bin/traffic_server(HdrHeap::allocate_obj(int, int)+0xa4)[0x62e414]
/z/bin/traffic_server(mime_hdr_copy_onto(MIMEHdrImpl*, HdrHeap*, MIMEHdrImpl*, 
HdrHeap*, bool)+0x102)[0x634bc2]
/z/bin/traffic_server(http_hdr_copy_onto(HTTPHdrImpl*, HdrHeap*, HTTPHdrImpl*, 
HdrHeap*, bool)+0x87)[0x62a427]
/z/bin/traffic_server(HttpSM::set_next_state()+0x9c1)[0x5a0cc1]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c13]
/z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c82a]
/z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
/z/lib/plugins/x3me_dscp.so(http_txn_hook(tsapi_cont*, TSEvent, 
void*)+0x236)[0x2b0211806508]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999d2]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0538]
/z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c13]
/z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0538]
/z/bin/traffic_server(HttpSM::state_cache_open_read(int, void*)+0x103)[0x59b003]
/z/bin/traffic_server(HttpSM::main_handler(int, void*)+0xd8)[0x59ada8]
/z/bin/traffic_server(HttpCacheSM::state_cache_open_read(int, 
void*)+0x1d6)[0x57bc16]
/z/bin/traffic_server[0x6d56b8]
/z/bin/traffic_server(CacheVC::openReadStartEarliest(int, 
Event*)+0x7d0)[0x6d9120]
/z/bin/traffic_server(CacheVC::handleReadDone(int, Event*)+0x27b)[0x6b79cb]
/z/bin/traffic_server(AIOCallbackInternal::io_complete(int, 
void*)+0x3d)[0x662e5d]
/z/bin/traffic_server(EThread::process_event(Event*, int)+0x91)[0x736121]
/z/bin/traffic_server(EThread::execute()+0x6a3)[0x736d83]
/z/bin/traffic_server[0x7353ba]
/lib64/libpthread.so.0(+0x7034)[0x2b0205370034]
/lib64/libc.so.6(clone+0x6d)[0x2b02060b475d]
{noformat}

 FATAL: ats_malloc: couldn't allocate XX bytes
 -

 Key: TS-3032
 URL: https://issues.apache.org/jira/browse/TS-3032
 Project: Traffic Server
  Issue Type: Bug
  Components: Core
Affects Versions: 5.0.1
Reporter: Nikolai Gorchilov
Assignee: Brian Geffon
  Labels: crash
 Fix For: 5.2.0


 ATS 5.0.1 under Unbuntu 12.04.4 running happily for days suddenly crashes due 
 to memory allocation issue. Happens once or twice a week.
 Server is having plenty of RAM - 128G - out of which 64G+ are free. Nothing 
 suspicious in dmesg.
 {noformat}
 FATAL: ats_malloc: couldn't allocate 155648 bytes
 /z/bin/traffic_server - STACK TRACE: 
 /z/lib/libtsutil.so.5(+0x1e837)[0x2b6251b3d837]
 /z/lib/libtsutil.so.5(ats_malloc+0x30)[0x2b6251b40c50]
 /z/bin/traffic_server(HdrHeap::coalesce_str_heaps(int)+0x34)[0x62e834]
 /z/bin/traffic_server(http_hdr_clone(HTTPHdrImpl*, HdrHeap*, 
 HdrHeap*)+0x8f)[0x62a54f]
 /z/bin/traffic_server(HttpTransactHeaders::copy_header_fields(HTTPHdr*, 
 HTTPHdr*, bool, long)+0x1ae)[0x5d08de]
 /z/bin/traffic_server(HttpTransact::build_request(HttpTransact::State*, 
 HTTPHdr*, HTTPHdr*, HTTPVersion)+0x5c)[0x5b280c]
 /z/bin/traffic_server(HttpTransact::HandleCacheOpenReadMiss(HttpTransact::State*)+0x2c8)[0x5c2ce8]
 /z/bin/traffic_server(HttpSM::call_transact_and_set_next_state(void 
 (*)(HttpTransact::State*))+0x66)[0x58e356]
 /z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
 /z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
 /z/bin/traffic_server(HttpSM::do_hostdb_lookup()+0x27a)[0x58e84a]
 /z/bin/traffic_server(HttpSM::set_next_state()+0xd48)[0x5a1038]
 /z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
 /z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
 /z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
 /z/lib/plugins/x3me_dscp.so(http_txn_hook(tsapi_cont*, TSEvent, 
 void*)+0x236)[0x2b626342b508]
 /z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
 /z/bin/traffic_server(HttpSM::state_cache_open_read(int, 
 void*)+0x180)[0x59b070]
 /z/bin/traffic_server(HttpSM::main_handler(int, void*)+0xd8)[0x59ad98]
 /z/bin/traffic_server(HttpCacheSM::state_cache_open_read(int, 
 void*)+0x173)[0x57bbb3]
 /z/bin/traffic_server(Cache::open_read(Continuation*, INK_MD5*, HTTPHdr*, 
 CacheLookupHttpConfig*, CacheFragType, char*, int)+0x616)[0x6d65a6]
 /z/bin/traffic_server(CacheProcessor::open_read(Continuation*, URL*, bool, 
 HTTPHdr*, CacheLookupHttpConfig*, long, CacheFragType)+0xb0)[0x6b1af0]
 /z/bin/traffic_server(HttpCacheSM::open_read(URL*, HTTPHdr*, 
 

[jira] [Commented] (TS-3032) FATAL: ats_malloc: couldn't allocate XXXXXX bytes

2014-08-24 Thread Zhao Yongming (JIRA)

[ 
https://issues.apache.org/jira/browse/TS-3032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14108410#comment-14108410
 ] 

Zhao Yongming commented on TS-3032:
---

I don't know who have any sucess story on BIG memory system, I'd like to hear 
if any.

for the problem you have, please attach some more data such as:
1. /proc/meminfo
2. the traffic_server process status: /proc//status
3. more system log related to alloc and memory, such as dmesg  syslog

and, please tell us your configure options when building the binary too. hopes 
that will help us inspect the problem.

 FATAL: ats_malloc: couldn't allocate XX bytes
 -

 Key: TS-3032
 URL: https://issues.apache.org/jira/browse/TS-3032
 Project: Traffic Server
  Issue Type: Bug
  Components: Core
Affects Versions: 5.0.1
Reporter: Nikolai Gorchilov
Assignee: Brian Geffon
  Labels: crash
 Fix For: 5.2.0


 ATS 5.0.1 under Unbuntu 12.04.4 running happily for days suddenly crashes due 
 to memory allocation issue. Happens once or twice a week.
 Server is having plenty of RAM - 128G - out of which 64G+ are free. Nothing 
 suspicious in dmesg.
 {noformat}
 FATAL: ats_malloc: couldn't allocate 155648 bytes
 /z/bin/traffic_server - STACK TRACE: 
 /z/lib/libtsutil.so.5(+0x1e837)[0x2b6251b3d837]
 /z/lib/libtsutil.so.5(ats_malloc+0x30)[0x2b6251b40c50]
 /z/bin/traffic_server(HdrHeap::coalesce_str_heaps(int)+0x34)[0x62e834]
 /z/bin/traffic_server(http_hdr_clone(HTTPHdrImpl*, HdrHeap*, 
 HdrHeap*)+0x8f)[0x62a54f]
 /z/bin/traffic_server(HttpTransactHeaders::copy_header_fields(HTTPHdr*, 
 HTTPHdr*, bool, long)+0x1ae)[0x5d08de]
 /z/bin/traffic_server(HttpTransact::build_request(HttpTransact::State*, 
 HTTPHdr*, HTTPHdr*, HTTPVersion)+0x5c)[0x5b280c]
 /z/bin/traffic_server(HttpTransact::HandleCacheOpenReadMiss(HttpTransact::State*)+0x2c8)[0x5c2ce8]
 /z/bin/traffic_server(HttpSM::call_transact_and_set_next_state(void 
 (*)(HttpTransact::State*))+0x66)[0x58e356]
 /z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
 /z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
 /z/bin/traffic_server(HttpSM::do_hostdb_lookup()+0x27a)[0x58e84a]
 /z/bin/traffic_server(HttpSM::set_next_state()+0xd48)[0x5a1038]
 /z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
 /z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
 /z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
 /z/lib/plugins/x3me_dscp.so(http_txn_hook(tsapi_cont*, TSEvent, 
 void*)+0x236)[0x2b626342b508]
 /z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
 /z/bin/traffic_server(HttpSM::state_cache_open_read(int, 
 void*)+0x180)[0x59b070]
 /z/bin/traffic_server(HttpSM::main_handler(int, void*)+0xd8)[0x59ad98]
 /z/bin/traffic_server(HttpCacheSM::state_cache_open_read(int, 
 void*)+0x173)[0x57bbb3]
 /z/bin/traffic_server(Cache::open_read(Continuation*, INK_MD5*, HTTPHdr*, 
 CacheLookupHttpConfig*, CacheFragType, char*, int)+0x616)[0x6d65a6]
 /z/bin/traffic_server(CacheProcessor::open_read(Continuation*, URL*, bool, 
 HTTPHdr*, CacheLookupHttpConfig*, long, CacheFragType)+0xb0)[0x6b1af0]
 /z/bin/traffic_server(HttpCacheSM::open_read(URL*, HTTPHdr*, 
 CacheLookupHttpConfig*, long)+0x83)[0x57c2d3]
 /z/bin/traffic_server(HttpSM::do_cache_lookup_and_read()+0xfb)[0x58baeb]
 /z/bin/traffic_server(HttpSM::set_next_state()+0x888)[0x5a0b78]
 /z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
 /z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
 /z/bin/traffic_server(HttpSM::set_next_state()+0x7e2)[0x5a0ad2]
 /z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
 /z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
 /z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x343)[0x599c03]
 /z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
 /z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
 /z/lib/plugins/cacheurl.so(+0x17dc)[0x2b6263a477dc]
 /z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
 /z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
 /z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
 /z/lib/plugins/tslua.so(+0x596f)[0x2b626363396f]
 /z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
 /z/bin/traffic_server(HttpSM::state_api_callback(int, void*)+0x8a)[0x59c81a]
 /z/bin/traffic_server(TSHttpTxnReenable+0x141)[0x4caa51]
 /z/lib/plugins/stats_over_http.so(+0x1235)[0x2b6263228235]
 /z/bin/traffic_server(HttpSM::state_api_callout(int, void*)+0x102)[0x5999c2]
 /z/bin/traffic_server(HttpSM::set_next_state()+0x238)[0x5a0528]
 /z/bin/traffic_server(HttpSM::state_read_client_request_header(int,