Re: [SR-Users] Errors leading to kamailio proc crash
Hello, On 2/21/13 9:59 AM, Grant Bagdasarian wrote: Can I increase the shared memory when kamailio is still running or configure it somewhere in a configuration file to use x amount of shared memory? The kamailio process is managed by heartbeat/pacemaker as a resource and I was advised not to manually restart a resource which is managed by heartbeat/pacemaker. usually kamailio is started via init.d script (even by monitoring tools such as heartbeat), there you can put the the parameter you want. The virtual server Kamailio is running on is up for 133 days now. But the kamailio process was up for a couple of weeks, since it has been restarted a few times. Where can I find the coredump? You can set the path where to write core dump via -w command line parameter. If you don't set that, then it is either in / or in a folder configured at OS level (e.g., in macosx is /cores). Cheers, Daniel *From:*sr-users-boun...@lists.sip-router.org [mailto:sr-users-boun...@lists.sip-router.org] *On Behalf Of *Daniel-Constantin Mierla *Sent:* Thursday, February 21, 2013 9:34 AM *To:* SIP Router - Kamailio (OpenSER) and SIP Express Router (SER) - Users Mailing List *Subject:* Re: [SR-Users] Errors leading to kamailio proc crash Default shared memory is 32MB, that is quite low if you have a lot of traffic. You can increase it with '-m' command line parameter. Try to get a coredump and then send the backtrace. It will help to discover where is the issue, even when out of memory, there should be no crash. For that, use 'ulimit -c unlimited' before starting the sip server. How long it was running? Can it be eventually reproduced? Also, to be sure is not an issue already fixed, upgrade to latest 3.3.x, you don't have to change anything in database or config file. Cheers, Daniel On 2/19/13 2:38 PM, Grant Bagdasarian wrote: Forgot to mention the version: version: kamailio 3.3.1 (i386/linux) 2d0f6f flags: STATS: Off, USE_IPV6, USE_TCP, USE_TLS, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, F_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 4MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: 2d0f6f compiled on 10:35:51 Oct 22 2012 with gcc 4.4.3 *From:*sr-users-boun...@lists.sip-router.org mailto:sr-users-boun...@lists.sip-router.org [mailto:sr-users-boun...@lists.sip-router.org] *On Behalf Of *Grant Bagdasarian *Sent:* Tuesday, February 19, 2013 2:36 PM *To:* sr-users@lists.sip-router.org mailto:sr-users@lists.sip-router.org *Subject:* [SR-Users] Errors leading to kamailio proc crash Hello, Last Thursday our kamailio process crashed. There are a lot of memory related errors in there. See attachment for details. Did the machine run out of memory? How do I prevent this from happening again? The machine has 4GB of memory. Regards, Grant ___ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org mailto:sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users -- Daniel-Constantin Mierla -http://www.asipto.com http://twitter.com/#!/miconda http://twitter.com/#%21/miconda -http://www.linkedin.com/in/miconda Kamailio World Conference, April 16-17, 2013, Berlin -http://conference.kamailio.com - -- Daniel-Constantin Mierla - http://www.asipto.com http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Kamailio World Conference, April 16-17, 2013, Berlin - http://conference.kamailio.com - ___ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
Re: [SR-Users] Errors leading to kamailio proc crash
Can I increase the shared memory when kamailio is still running or configure it somewhere in a configuration file to use x amount of shared memory? The kamailio process is managed by heartbeat/pacemaker as a resource and I was advised not to manually restart a resource which is managed by heartbeat/pacemaker. The virtual server Kamailio is running on is up for 133 days now. But the kamailio process was up for a couple of weeks, since it has been restarted a few times. Where can I find the coredump? From: sr-users-boun...@lists.sip-router.org [mailto:sr-users-boun...@lists.sip-router.org] On Behalf Of Daniel-Constantin Mierla Sent: Thursday, February 21, 2013 9:34 AM To: SIP Router - Kamailio (OpenSER) and SIP Express Router (SER) - Users Mailing List Subject: Re: [SR-Users] Errors leading to kamailio proc crash Default shared memory is 32MB, that is quite low if you have a lot of traffic. You can increase it with '-m' command line parameter. Try to get a coredump and then send the backtrace. It will help to discover where is the issue, even when out of memory, there should be no crash. For that, use 'ulimit -c unlimited' before starting the sip server. How long it was running? Can it be eventually reproduced? Also, to be sure is not an issue already fixed, upgrade to latest 3.3.x, you don't have to change anything in database or config file. Cheers, Daniel On 2/19/13 2:38 PM, Grant Bagdasarian wrote: Forgot to mention the version: version: kamailio 3.3.1 (i386/linux) 2d0f6f flags: STATS: Off, USE_IPV6, USE_TCP, USE_TLS, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, F_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 4MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: 2d0f6f compiled on 10:35:51 Oct 22 2012 with gcc 4.4.3 From: sr-users-boun...@lists.sip-router.orgmailto:sr-users-boun...@lists.sip-router.org [mailto:sr-users-boun...@lists.sip-router.org] On Behalf Of Grant Bagdasarian Sent: Tuesday, February 19, 2013 2:36 PM To: sr-users@lists.sip-router.orgmailto:sr-users@lists.sip-router.org Subject: [SR-Users] Errors leading to kamailio proc crash Hello, Last Thursday our kamailio process crashed. There are a lot of memory related errors in there. See attachment for details. Did the machine run out of memory? How do I prevent this from happening again? The machine has 4GB of memory. Regards, Grant ___ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.orgmailto:sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users -- Daniel-Constantin Mierla - http://www.asipto.com http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Kamailio World Conference, April 16-17, 2013, Berlin - http://conference.kamailio.com - ___ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
[SR-Users] Errors leading to kamailio proc crash
Hello, Last Thursday our kamailio process crashed. There are a lot of memory related errors in there. See attachment for details. Did the machine run out of memory? How do I prevent this from happening again? The machine has 4GB of memory. Regards, Grant 2013-02-14T12:31:24+01:00 kamailio01 /usr/local/sbin/kamailio[18697]: ERROR: core [msg_translator.c:1835]: ERROR: build_req_buf_from_sip_req: out of memory 2013-02-14T12:31:24+01:00 kamailio01 /usr/local/sbin/kamailio[18697]: ERROR: tm [t_fwd.c:431]: ERROR: print_uac_request: no shm mem 2013-02-14T12:31:24+01:00 kamailio01 /usr/local/sbin/kamailio[18697]: ERROR: tm [t_fwd.c:1530]: ERROR: t_forward_nonack: failure to add branches 2013-02-14T12:31:24+01:00 kamailio01 /usr/local/sbin/kamailio[18697]: ERROR: sl [sl_funcs.c:371]: ERROR: sl_reply_error used: No error (2/SL) 2013-02-14T12:31:24+01:00 kamailio01 /usr/local/sbin/kamailio[18697]: ERROR: tm [t_reply.c:601]: ERROR: _reply_light: cannot allocate shmem buffer 2013-02-14T12:31:24+01:00 kamailio01 /usr/local/sbin/kamailio[18697]: ERROR: tm [t_lookup.c:1538]: ERROR: t_unref: generation of a delayed stateful reply failed 2013-02-14T12:31:25+01:00 kamailio01 /usr/local/sbin/kamailio[18698]: ERROR: tm [t_lookup.c:1338]: ERROR: new_t: out of mem: 2013-02-14T12:31:25+01:00 kamailio01 /usr/local/sbin/kamailio[18698]: ERROR: tm [t_lookup.c:1478]: ERROR: t_newtran: new_t failed 2013-02-14T12:31:25+01:00 kamailio01 /usr/local/sbin/kamailio[18698]: ERROR: sl [sl_funcs.c:371]: ERROR: sl_reply_error used: No error (2/SL) 2013-02-14T12:31:25+01:00 kamailio01 /usr/local/sbin/kamailio[18700]: ERROR: dialog [dlg_hash.c:430]: no more shm mem (329) 2013-02-14T12:31:25+01:00 kamailio01 /usr/local/sbin/kamailio[18700]: ERROR: dialog [dlg_handlers.c:830]: failed to create new dialog 2013-02-14T12:31:25+01:00 kamailio01 /usr/local/sbin/kamailio[18697]: ERROR: dialog [dlg_hash.c:430]: no more shm mem (329) 2013-02-14T12:31:25+01:00 kamailio01 /usr/local/sbin/kamailio[18697]: ERROR: dialog [dlg_handlers.c:830]: failed to create new dialog 2013-02-14T12:31:27+01:00 kamailio01 /usr/local/sbin/kamailio[18699]: ERROR: dialog [dlg_hash.c:430]: no more shm mem (329) 2013-02-14T12:31:27+01:00 kamailio01 /usr/local/sbin/kamailio[18699]: ERROR: dialog [dlg_handlers.c:830]: failed to create new dialog 2013-02-14T12:31:27+01:00 kamailio01 /usr/local/sbin/kamailio[18695]: ERROR: dialog [dlg_hash.c:430]: no more shm mem (329) 2013-02-14T12:31:27+01:00 kamailio01 /usr/local/sbin/kamailio[18695]: ERROR: dialog [dlg_handlers.c:830]: failed to create new dialog 2013-02-14T12:31:27+01:00 kamailio01 /usr/local/sbin/kamailio[18696]: ERROR: tm [t_lookup.c:1338]: ERROR: new_t: out of mem: 2013-02-14T12:31:27+01:00 kamailio01 /usr/local/sbin/kamailio[18696]: ERROR: tm [t_lookup.c:1478]: ERROR: t_newtran: new_t failed 2013-02-14T12:31:27+01:00 kamailio01 /usr/local/sbin/kamailio[18696]: ERROR: sl [sl_funcs.c:371]: ERROR: sl_reply_error used: No error (2/SL) 2013-02-14T12:31:38+01:00 kamailio01 /usr/local/sbin/kamailio[18700]: ERROR: core [msg_translator.c:1835]: ERROR: build_req_buf_from_sip_req: out of memory 2013-02-14T12:31:38+01:00 kamailio01 /usr/local/sbin/kamailio[18700]: ERROR: tm [t_fwd.c:431]: ERROR: print_uac_request: no shm mem 2013-02-14T12:31:38+01:00 kamailio01 /usr/local/sbin/kamailio[18700]: ERROR: tm [t_fwd.c:1530]: ERROR: t_forward_nonack: failure to add branches 2013-02-14T12:31:38+01:00 kamailio01 /usr/local/sbin/kamailio[18700]: ERROR: sl [sl_funcs.c:371]: ERROR: sl_reply_error used: No error (2/SL) 2013-02-14T12:31:38+01:00 kamailio01 /usr/local/sbin/kamailio[18696]: ERROR: tm [t_lookup.c:1338]: ERROR: new_t: out of mem: 2013-02-14T12:31:38+01:00 kamailio01 /usr/local/sbin/kamailio[18696]: ERROR: tm [t_lookup.c:1478]: ERROR: t_newtran: new_t failed 2013-02-14T12:31:38+01:00 kamailio01 /usr/local/sbin/kamailio[18696]: ERROR: sl [sl_funcs.c:371]: ERROR: sl_reply_error used: No error (2/SL) 2013-02-14T12:31:43+01:00 kamailio01 /usr/local/sbin/kamailio[18699]: ERROR: tm [t_reply.c:601]: ERROR: _reply_light: cannot allocate shmem buffer 2013-02-14T12:31:43+01:00 kamailio01 /usr/local/sbin/kamailio[18699]: ERROR: core [sip_msg_clone.c:970]: ERROR: msg_lump_cloner: cannot allocate memory 2013-02-14T12:31:43+01:00 kamailio01 /usr/local/sbin/kamailio[18699]: ERROR: tm [t_fwd.c:1469]: ERROR: t_forward_nonack: failed to save the message lumps 2013-02-14T12:31:43+01:00 kamailio01 /usr/local/sbin/kamailio[18699]: ERROR: sl [sl_funcs.c:371]: ERROR: sl_reply_error used: I'm terribly sorry, server error occurred (1/SL) 2013-02-14T12:31:43+01:00 kamailio01 /usr/local/sbin/kamailio[18699]: ERROR: tm [t_reply.c:601]: ERROR: _reply_light: cannot allocate shmem buffer 2013-02-14T12:31:43+01:00 kamailio01 /usr/local/sbin/kamailio[18699]: ERROR: tm [t_lookup.c:1538]: ERROR: t_unref: generation of a delayed stateful reply failed 2013-02-14T12:31:43+01:00 kamailio01 /usr/local/sbin/kamailio[18697]: ERROR: dialog
Re: [SR-Users] Errors leading to kamailio proc crash
Forgot to mention the version: version: kamailio 3.3.1 (i386/linux) 2d0f6f flags: STATS: Off, USE_IPV6, USE_TCP, USE_TLS, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, F_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 4MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: 2d0f6f compiled on 10:35:51 Oct 22 2012 with gcc 4.4.3 From: sr-users-boun...@lists.sip-router.org [mailto:sr-users-boun...@lists.sip-router.org] On Behalf Of Grant Bagdasarian Sent: Tuesday, February 19, 2013 2:36 PM To: sr-users@lists.sip-router.org Subject: [SR-Users] Errors leading to kamailio proc crash Hello, Last Thursday our kamailio process crashed. There are a lot of memory related errors in there. See attachment for details. Did the machine run out of memory? How do I prevent this from happening again? The machine has 4GB of memory. Regards, Grant ___ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
