Found coredump on one of the server, see some partial message below while taking the back trace;
Core was generated by `/usr/sbin/opensips -P /var/run/opensips.pid -m 64 -u opensips -g opensips'. Program terminated with signal 11, Segmentation fault. #0 0x00007f650687a069 in sip_msg_cloner () from /usr/lib64/opensips/modules/tm.so Missing separate debuginfos, use: debuginfo-install opensips-1.6.3-notls.x86_64 Please advise what might be the reason causing opensips to crash. On Thu, Jan 12, 2017 at 1:28 PM, Ahmed Munir <ahmedmunir...@gmail.com> wrote: > > The version currently running is 1.6.3. Today again we got the opensips > crashed issues i.e. 5 out of 8 were crashed due to below message (common on > all 5); > > Jan 12 10:07:36 QORCLPSIPROXY02 kernel: opensips[2820]: segfault at 0 ip > 00000000004a2936 sp 00007fff5cfaa430 error 6 in opensips[400000+13a000] > > Jan 12 10:07:36 QORCLPSIPROXY02 abrt[39302]: Can't open 'core.2820': > Permission denied > > Jan 12 10:07:36 QORCLPSIPROXY02 abrt[39302]: Saved core dump of pid 2820 > (/usr/sbin/opensips) to /var/spool/abrt/ccpp-2017-01-12-10:07:36-2820 > (70246400 bytes) > > Jan 12 10:07:36 QORCLPSIPROXY02 abrtd: Directory > 'ccpp-2017-01-12-10:07:36-2820' > creation detected > > Jan 12 10:07:38 QORCLPSIPROXY02 kernel: Bridge firewalling registered > > Jan 12 10:08:46 QORCLPSIPROXY02 abrtd: Sending an email... > > Jan 12 10:08:46 QORCLPSIPROXY02 abrtd: /usr/sbin/sendmail: No such file or > directory > > Jan 12 10:08:46 QORCLPSIPROXY02 abrtd: . . . message not sent. > > Jan 12 10:08:46 QORCLPSIPROXY02 abrtd: Error running '/bin/mailx' > > Jan 12 10:08:46 QORCLPSIPROXY02 abrtd: 'post-create' on > '/var/spool/abrt/ccpp-2017-01-12-10:07:36-2820' exited with 1 > > Jan 12 10:08:46 QORCLPSIPROXY02 abrtd: Deleting problem directory > '/var/spool/abrt/ccpp-2017-01-12-10:07:36-2820' > > > We would like to know why we are getting this segmentation fault? Is there > a way to backtrace the issue as don't have the core dump? Further added, > did someone faced this similar issue(s) in past and got this fixed without > upgrade? > > Note: we are running opensips for quite a while didn't faced this kind of > issue and there is no changes made on opensips routing script. > > >> From: Răzvan Crainea <raz...@opensips.org> >>> To: users@lists.opensips.org >>> Subject: Re: [OpenSIPS-Users] OpenSIPs crashed >>> Message-ID: <2b35e2dc-6e11-1787-b87e-33bd29a32...@opensips.org> >>> Content-Type: text/plain; charset="utf-8"; Format="flowed" >>> >>> Hi, Ahmed! >>> >>> Make sure OpenSIPS is run as root and it is allowed to write in the >>> /var/spool/abrt/ directory, otherwise it is unable to write the core >>> dump, therefore we can't inspect it to say what is happening. If this >>> does not work, make OpenSIPS write the core dump in a writeble directory >>> by changing the /proc/sys/kernel/core_pattern settings. >>> Also, please let us know the version of OpenSIPS you are running. >>> >>> Best regards, >>> >>> Răzvan Crainea >>> OpenSIPS Solutions >>> www.opensips-solutions.com >>> >>> >>> On 01/11/2017 11:10 PM, Ahmed Munir wrote: >>> > Hi, >>> > >>> > Our OpenSIPs service crashed with below error; >>> > >>> > Jan 11 12:16:19 QORBLPSIPROXY05 abrtd: Directory >>> > 'ccpp-2017-01-11-12:16:19-2807' creation detected >>> > Jan 11 12:16:19 QORBLPSIPROXY05 abrt[65402]: Saved core dump of pid >>> > 2807 (/usr/sbin/opensips) to >>> > /var/spool/abrt/ccpp-2017-01-11-12:16:19-2807 (70225920 bytes) >>> > Jan 11 12:16:19 QORBLPSIPROXY05 abrt[65402]: Can't open 'core.2807': >>> > Permission denied >>> > Jan 11 12:16:19 QORBLPSIPROXY05 kernel: opensips[2807]: segfault at 29 >>> > ip 00000000004bae7a sp 00007fffdb7734d0 error 4 in >>> opensips[400000+13a000] >>> > >>> > >>> > We would like to know, what might be the reason for the crash. >>> > >>> > Further added, there is another server we are running OpenSIPs, the >>> > opensips child processes utilizing 100% of CPU and the system load >>> > average reach around 'load average: 20.01, 18.03, 24.00' as normally >>> > it is below 1 (load average). >>> > >>> > After looking into logs, unable to find the info what might causing >>> > the CPU to spike. >>> > >>> > Please advise what useful steps to take for narrowing down this issue. >>> > >>> > >>> > -- >>> > Regards, >>> > >>> > Ahmed Munir Chohan >>> > >>> >>> >> >> >> -- >> Regards, >> >> Ahmed Munir Chohan >> >> > > > -- > Regards, > > Ahmed Munir Chohan > > -- Regards, Ahmed Munir Chohan
_______________________________________________ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users