Re: [asterisk-users] Advices when Asterisk segfaults and nothing useful in logs

[Richard Mudgett]

On Fri, Feb 17, 2017 at 8:30 AM, Olivier  wrote:

>
>
> 2017-02-17 14:39 GMT+01:00 George Joseph :
>
>>
>>
>>
>> If asterisk was compiled with DEBUG_THREADS,
>>
>
> Would you then  advise to run an Asterisk server in production with
> DEBUG_THREADS enabled ?
>

No, you should not leave DEBUG_THREADS on in a normal production
environment.  Only enable it when you
are actually hunting for a deadlock.  DEBUG_THREADS causes a noticeable
drop in performance.

Richard
-- 
_
-- Bandwidth and Colocation Provided by http://www.api-digital.com --

Check out the new Asterisk community forum at: https://community.asterisk.org/

New to Asterisk? Start here:
  https://wiki.asterisk.org/wiki/display/AST/Getting+Started

asterisk-users mailing list
To UNSUBSCRIBE or update options visit:
   http://lists.digium.com/mailman/listinfo/asterisk-users

Re: [asterisk-users] Advices when Asterisk segfaults and nothing useful in logs

[Olivier]

2017-02-17 14:39 GMT+01:00 George Joseph :

>
>
>
> If asterisk was compiled with DEBUG_THREADS,
>

Would you then  advise to run an Asterisk server in production with
DEBUG_THREADS enabled ?
Page [1] does not mention to do so nor to avoid it.

In the production environment I'm thinking about, Asterisk is compiled with
DONT_OPTIMIZE and BETTER_BACKTRACES
hoping that if Asterisk ever fails, sysadmins would have everything needed
to track the issue no matter if those would increase resources needed to
run Asterisk properly.


[1]
https://wiki.asterisk.org/wiki/display/AST/Getting+a+Backtrace#GettingaBacktrace-GettingInformationForADeadlock
-- 
_
-- Bandwidth and Colocation Provided by http://www.api-digital.com --

Check out the new Asterisk community forum at: https://community.asterisk.org/

New to Asterisk? Start here:
  https://wiki.asterisk.org/wiki/display/AST/Getting+Started

asterisk-users mailing list
To UNSUBSCRIBE or update options visit:
   http://lists.digium.com/mailman/listinfo/asterisk-users

Re: [asterisk-users] Advices when Asterisk segfaults and nothing useful in logs

[George Joseph]

On Fri, Feb 17, 2017 at 5:17 AM, Olivier  wrote:

> Hi George,
>
> How does ast_coredumper compare to ast_grab_core ) ?
> Is it worth learning to use both or shall favor one ?
>
> PS: As I don't know either program, yet, my question may seem silly.
> Please, forgive me for this
>

Not silly at all.

ast_grab_core actually kills asterisk to get the core file while
ast_coredumper dumps the core file and let's asterisk continue.  If
asterisk is truly deadlocked this may not matter but in some situations you
might not want to kill asterisk.

If asterisk was compiled with DEBUG_THREADS, ast_coredumper dumps the locks
table which is critical in debugging deadlock scenarios.  It does this from
the coredump, as opposed to running "core show locks", which makes the
locks table consistent with the rest of the backtraces.

ast_coiredumper can also find and process existing coredumps and it can
create a tarball containing the backtraces and lock tables.



>
> 2017-02-14 22:52 GMT+01:00 George Joseph :
>
>>
>>
>> On Tue, Feb 14, 2017 at 2:51 PM, George Joseph 
>> wrote:
>>
>>>
>>>
>>> On Tue, Feb 14, 2017 at 10:21 AM, Olivier  wrote:
>>>
 Hello,

 I've got a 13.13.1 system using PJSIP stack on debian Jessie.
 It runs from 50 to 100 simultaneous calls (so 100 to 200  PJSIP
 channels) all day long.
 From time to time, roughly meaning once a month, it segfaults  with
 lines (from dmesg -T output) like this:
 asterisk[1160]: segfault at 7efe ip 005881d6 sp
 7fec95c33910 error 4 in asterisk[40+2a2000]


 Debug level was unfortunately not set in asterisk.conf but verbose
 level was set to 5.
 Asterisk runs with:
 /usr/sbin/asterisk -U asterisk -G asterisk -g

 Asterisk is compiled with DONT_OPTIMIZE and BETTER_BACKTRACES options.

 "core show settings" outputs:
 * Directories
   -
   Configuration file:
   Configuration directory: /etc/asterisk
   Module directory:/usr/lib/asterisk/modules
   Spool directory: /var/spool/asterisk
   Log directory:   /var/log/asterisk
   Run/Sockets directory:   /var/run/asterisk
   PID file:/var/run/asterisk/asterisk.pid
   VarLib directory:/var/lib/asterisk
   Data directory:  /var/lib/asterisk
   ASTDB:   /var/lib/asterisk/astdb
   IAX2 Keys directory: /var/lib/asterisk/keys



 1. Am I correct to expect a coredump file to be produced anytime
 asterisk segfaults ?

>>>
>>> Yes if -g is set and the user that's running asterisk has permissions to
>>> set ulimit -c.
>>>
>>>

 2. Does Asterisk prints any WARNING or ERROR message whenever it
 detects, at startup preferably, that it has not required permissions to
 write a coredump file ?

>>>
>>> No because it's the system that determines where a coredump goes and
>>> actually writes it, not asterisk.
>>> It's the sysctl kernel.core_pattern setting.
>>>
>>>

 3. Among above directories, which one is choosen to save coredump files
 ? Is it something that can/should be configured in /etc/asterisk (I've seen
 related options in some debian  /etc/default/asterisk files but I would be
 curious to know if such things exist

>>>
>>> See above.
>>>
>>>

 4. Is there anything useful I can do with a line such as :
 asterisk[1160]: segfault at 7efe ip 005881d6 sp
 7fec95c33910 error 4 in asterisk[40+2a2000] ? Any pointer ?

>>>
>>> Nope.  Not a thing.  Sorry.
>>>
>>>
>>>

 5. Suggestions ?

>>>
>>> If you can at least get the system to write a coredump file, there are
>>> new utilities in /var/lib/asterisk/scripts, namely ast_coredumper which can
>>> help create the backtraces if it can at least find the core file.  Just run
>>> "./ast_coredumper --help" for more info.   You should also be able to use
>>> those utilities with earlier Asterisk 13 versions.
>>>
>>>
>>>
>>
>> Oh yeah, and it's on my list to publish instructions on how ot use those
>> utilities but they were just released yesterday.
>>
>>
>>
>>>
 Best regards


 --
 _
 -- Bandwidth and Colocation Provided by http://www.api-digital.com --

 Check out the new Asterisk community forum at:
 https://community.asterisk.org/

 New to Asterisk? Start here:
   https://wiki.asterisk.org/wiki/display/AST/Getting+Started

 asterisk-users mailing list
 To UNSUBSCRIBE or update options visit:
http://lists.digium.com/mailman/listinfo/asterisk-users

>>>
>>>
>>>
>>> --
>>> George Joseph
>>> Digium, Inc. | Software Developer
>>> 445 Jan Davis Drive NW - Huntsville, AL 35806 - US
>>> Check us out at: 

Re: [asterisk-users] Advices when Asterisk segfaults and nothing useful in logs

[Olivier]

Hi George,

How does ast_coredumper compare to ast_grab_core ) ?
Is it worth learning to use both or shall favor one ?

PS: As I don't know either program, yet, my question may seem silly.
Please, forgive me for this

2017-02-14 22:52 GMT+01:00 George Joseph :

>
>
> On Tue, Feb 14, 2017 at 2:51 PM, George Joseph  wrote:
>
>>
>>
>> On Tue, Feb 14, 2017 at 10:21 AM, Olivier  wrote:
>>
>>> Hello,
>>>
>>> I've got a 13.13.1 system using PJSIP stack on debian Jessie.
>>> It runs from 50 to 100 simultaneous calls (so 100 to 200  PJSIP
>>> channels) all day long.
>>> From time to time, roughly meaning once a month, it segfaults  with
>>> lines (from dmesg -T output) like this:
>>> asterisk[1160]: segfault at 7efe ip 005881d6 sp
>>> 7fec95c33910 error 4 in asterisk[40+2a2000]
>>>
>>>
>>> Debug level was unfortunately not set in asterisk.conf but verbose level
>>> was set to 5.
>>> Asterisk runs with:
>>> /usr/sbin/asterisk -U asterisk -G asterisk -g
>>>
>>> Asterisk is compiled with DONT_OPTIMIZE and BETTER_BACKTRACES options.
>>>
>>> "core show settings" outputs:
>>> * Directories
>>>   -
>>>   Configuration file:
>>>   Configuration directory: /etc/asterisk
>>>   Module directory:/usr/lib/asterisk/modules
>>>   Spool directory: /var/spool/asterisk
>>>   Log directory:   /var/log/asterisk
>>>   Run/Sockets directory:   /var/run/asterisk
>>>   PID file:/var/run/asterisk/asterisk.pid
>>>   VarLib directory:/var/lib/asterisk
>>>   Data directory:  /var/lib/asterisk
>>>   ASTDB:   /var/lib/asterisk/astdb
>>>   IAX2 Keys directory: /var/lib/asterisk/keys
>>>
>>>
>>>
>>> 1. Am I correct to expect a coredump file to be produced anytime
>>> asterisk segfaults ?
>>>
>>
>> Yes if -g is set and the user that's running asterisk has permissions to
>> set ulimit -c.
>>
>>
>>>
>>> 2. Does Asterisk prints any WARNING or ERROR message whenever it
>>> detects, at startup preferably, that it has not required permissions to
>>> write a coredump file ?
>>>
>>
>> No because it's the system that determines where a coredump goes and
>> actually writes it, not asterisk.
>> It's the sysctl kernel.core_pattern setting.
>>
>>
>>>
>>> 3. Among above directories, which one is choosen to save coredump files
>>> ? Is it something that can/should be configured in /etc/asterisk (I've seen
>>> related options in some debian  /etc/default/asterisk files but I would be
>>> curious to know if such things exist
>>>
>>
>> See above.
>>
>>
>>>
>>> 4. Is there anything useful I can do with a line such as :
>>> asterisk[1160]: segfault at 7efe ip 005881d6 sp
>>> 7fec95c33910 error 4 in asterisk[40+2a2000] ? Any pointer ?
>>>
>>
>> Nope.  Not a thing.  Sorry.
>>
>>
>>
>>>
>>> 5. Suggestions ?
>>>
>>
>> If you can at least get the system to write a coredump file, there are
>> new utilities in /var/lib/asterisk/scripts, namely ast_coredumper which can
>> help create the backtraces if it can at least find the core file.  Just run
>> "./ast_coredumper --help" for more info.   You should also be able to use
>> those utilities with earlier Asterisk 13 versions.
>>
>>
>>
>
> Oh yeah, and it's on my list to publish instructions on how ot use those
> utilities but they were just released yesterday.
>
>
>
>>
>>> Best regards
>>>
>>>
>>> --
>>> _
>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>>>
>>> Check out the new Asterisk community forum at:
>>> https://community.asterisk.org/
>>>
>>> New to Asterisk? Start here:
>>>   https://wiki.asterisk.org/wiki/display/AST/Getting+Started
>>>
>>> asterisk-users mailing list
>>> To UNSUBSCRIBE or update options visit:
>>>http://lists.digium.com/mailman/listinfo/asterisk-users
>>>
>>
>>
>>
>> --
>> George Joseph
>> Digium, Inc. | Software Developer
>> 445 Jan Davis Drive NW - Huntsville, AL 35806 - US
>> Check us out at: www.digium.com & www.asterisk.org
>>
>>
>
>
> --
> George Joseph
> Digium, Inc. | Software Developer
> 445 Jan Davis Drive NW - Huntsville, AL 35806 - US
> Check us out at: www.digium.com & www.asterisk.org
>
>
> --
> _
> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>
> Check out the new Asterisk community forum at: https://community.asterisk.
> org/
>
> New to Asterisk? Start here:
>   https://wiki.asterisk.org/wiki/display/AST/Getting+Started
>
> asterisk-users mailing list
> To UNSUBSCRIBE or update options visit:
>http://lists.digium.com/mailman/listinfo/asterisk-users
>
-- 
_
-- Bandwidth and Colocation Provided by http://www.api-digital.com --

Check out the new Asterisk community forum at: 

Re: [asterisk-users] Advices when Asterisk segfaults and nothing useful in logs

[George Joseph]

On Tue, Feb 14, 2017 at 2:51 PM, George Joseph  wrote:

>
>
> On Tue, Feb 14, 2017 at 10:21 AM, Olivier  wrote:
>
>> Hello,
>>
>> I've got a 13.13.1 system using PJSIP stack on debian Jessie.
>> It runs from 50 to 100 simultaneous calls (so 100 to 200  PJSIP channels)
>> all day long.
>> From time to time, roughly meaning once a month, it segfaults  with lines
>> (from dmesg -T output) like this:
>> asterisk[1160]: segfault at 7efe ip 005881d6 sp
>> 7fec95c33910 error 4 in asterisk[40+2a2000]
>>
>>
>> Debug level was unfortunately not set in asterisk.conf but verbose level
>> was set to 5.
>> Asterisk runs with:
>> /usr/sbin/asterisk -U asterisk -G asterisk -g
>>
>> Asterisk is compiled with DONT_OPTIMIZE and BETTER_BACKTRACES options.
>>
>> "core show settings" outputs:
>> * Directories
>>   -
>>   Configuration file:
>>   Configuration directory: /etc/asterisk
>>   Module directory:/usr/lib/asterisk/modules
>>   Spool directory: /var/spool/asterisk
>>   Log directory:   /var/log/asterisk
>>   Run/Sockets directory:   /var/run/asterisk
>>   PID file:/var/run/asterisk/asterisk.pid
>>   VarLib directory:/var/lib/asterisk
>>   Data directory:  /var/lib/asterisk
>>   ASTDB:   /var/lib/asterisk/astdb
>>   IAX2 Keys directory: /var/lib/asterisk/keys
>>
>>
>>
>> 1. Am I correct to expect a coredump file to be produced anytime asterisk
>> segfaults ?
>>
>
> Yes if -g is set and the user that's running asterisk has permissions to
> set ulimit -c.
>
>
>>
>> 2. Does Asterisk prints any WARNING or ERROR message whenever it detects,
>> at startup preferably, that it has not required permissions to write a
>> coredump file ?
>>
>
> No because it's the system that determines where a coredump goes and
> actually writes it, not asterisk.
> It's the sysctl kernel.core_pattern setting.
>
>
>>
>> 3. Among above directories, which one is choosen to save coredump files ?
>> Is it something that can/should be configured in /etc/asterisk (I've seen
>> related options in some debian  /etc/default/asterisk files but I would be
>> curious to know if such things exist
>>
>
> See above.
>
>
>>
>> 4. Is there anything useful I can do with a line such as :
>> asterisk[1160]: segfault at 7efe ip 005881d6 sp
>> 7fec95c33910 error 4 in asterisk[40+2a2000] ? Any pointer ?
>>
>
> Nope.  Not a thing.  Sorry.
>
>
>
>>
>> 5. Suggestions ?
>>
>
> If you can at least get the system to write a coredump file, there are new
> utilities in /var/lib/asterisk/scripts, namely ast_coredumper which can
> help create the backtraces if it can at least find the core file.  Just run
> "./ast_coredumper --help" for more info.   You should also be able to use
> those utilities with earlier Asterisk 13 versions.
>
>
>

Oh yeah, and it's on my list to publish instructions on how ot use those
utilities but they were just released yesterday.



>
>> Best regards
>>
>>
>> --
>> _
>> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>>
>> Check out the new Asterisk community forum at:
>> https://community.asterisk.org/
>>
>> New to Asterisk? Start here:
>>   https://wiki.asterisk.org/wiki/display/AST/Getting+Started
>>
>> asterisk-users mailing list
>> To UNSUBSCRIBE or update options visit:
>>http://lists.digium.com/mailman/listinfo/asterisk-users
>>
>
>
>
> --
> George Joseph
> Digium, Inc. | Software Developer
> 445 Jan Davis Drive NW - Huntsville, AL 35806 - US
> Check us out at: www.digium.com & www.asterisk.org
>
>


-- 
George Joseph
Digium, Inc. | Software Developer
445 Jan Davis Drive NW - Huntsville, AL 35806 - US
Check us out at: www.digium.com & www.asterisk.org
-- 
_
-- Bandwidth and Colocation Provided by http://www.api-digital.com --

Check out the new Asterisk community forum at: https://community.asterisk.org/

New to Asterisk? Start here:
  https://wiki.asterisk.org/wiki/display/AST/Getting+Started

asterisk-users mailing list
To UNSUBSCRIBE or update options visit:
   http://lists.digium.com/mailman/listinfo/asterisk-users

Re: [asterisk-users] Advices when Asterisk segfaults and nothing useful in logs

[George Joseph]

On Tue, Feb 14, 2017 at 10:21 AM, Olivier  wrote:

> Hello,
>
> I've got a 13.13.1 system using PJSIP stack on debian Jessie.
> It runs from 50 to 100 simultaneous calls (so 100 to 200  PJSIP channels)
> all day long.
> From time to time, roughly meaning once a month, it segfaults  with lines
> (from dmesg -T output) like this:
> asterisk[1160]: segfault at 7efe ip 005881d6 sp
> 7fec95c33910 error 4 in asterisk[40+2a2000]
>
>
> Debug level was unfortunately not set in asterisk.conf but verbose level
> was set to 5.
> Asterisk runs with:
> /usr/sbin/asterisk -U asterisk -G asterisk -g
>
> Asterisk is compiled with DONT_OPTIMIZE and BETTER_BACKTRACES options.
>
> "core show settings" outputs:
> * Directories
>   -
>   Configuration file:
>   Configuration directory: /etc/asterisk
>   Module directory:/usr/lib/asterisk/modules
>   Spool directory: /var/spool/asterisk
>   Log directory:   /var/log/asterisk
>   Run/Sockets directory:   /var/run/asterisk
>   PID file:/var/run/asterisk/asterisk.pid
>   VarLib directory:/var/lib/asterisk
>   Data directory:  /var/lib/asterisk
>   ASTDB:   /var/lib/asterisk/astdb
>   IAX2 Keys directory: /var/lib/asterisk/keys
>
>
>
> 1. Am I correct to expect a coredump file to be produced anytime asterisk
> segfaults ?
>

Yes if -g is set and the user that's running asterisk has permissions to
set ulimit -c.


>
> 2. Does Asterisk prints any WARNING or ERROR message whenever it detects,
> at startup preferably, that it has not required permissions to write a
> coredump file ?
>

No because it's the system that determines where a coredump goes and
actually writes it, not asterisk.
It's the sysctl kernel.core_pattern setting.


>
> 3. Among above directories, which one is choosen to save coredump files ?
> Is it something that can/should be configured in /etc/asterisk (I've seen
> related options in some debian  /etc/default/asterisk files but I would be
> curious to know if such things exist
>

See above.


>
> 4. Is there anything useful I can do with a line such as :
> asterisk[1160]: segfault at 7efe ip 005881d6 sp
> 7fec95c33910 error 4 in asterisk[40+2a2000] ? Any pointer ?
>

Nope.  Not a thing.  Sorry.



>
> 5. Suggestions ?
>

If you can at least get the system to write a coredump file, there are new
utilities in /var/lib/asterisk/scripts, namely ast_coredumper which can
help create the backtraces if it can at least find the core file.  Just run
"./ast_coredumper --help" for more info.   You should also be able to use
those utilities with earlier Asterisk 13 versions.



>
> Best regards
>
>
> --
> _
> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>
> Check out the new Asterisk community forum at: https://community.asterisk.
> org/
>
> New to Asterisk? Start here:
>   https://wiki.asterisk.org/wiki/display/AST/Getting+Started
>
> asterisk-users mailing list
> To UNSUBSCRIBE or update options visit:
>http://lists.digium.com/mailman/listinfo/asterisk-users
>



-- 
George Joseph
Digium, Inc. | Software Developer
445 Jan Davis Drive NW - Huntsville, AL 35806 - US
Check us out at: www.digium.com & www.asterisk.org
-- 
_
-- Bandwidth and Colocation Provided by http://www.api-digital.com --

Check out the new Asterisk community forum at: https://community.asterisk.org/

New to Asterisk? Start here:
  https://wiki.asterisk.org/wiki/display/AST/Getting+Started

asterisk-users mailing list
To UNSUBSCRIBE or update options visit:
   http://lists.digium.com/mailman/listinfo/asterisk-users

[asterisk-users] Advices when Asterisk segfaults and nothing useful in logs

[Olivier]

Hello,

I've got a 13.13.1 system using PJSIP stack on debian Jessie.
It runs from 50 to 100 simultaneous calls (so 100 to 200  PJSIP channels)
all day long.
>From time to time, roughly meaning once a month, it segfaults  with lines
(from dmesg -T output) like this:
asterisk[1160]: segfault at 7efe ip 005881d6 sp
7fec95c33910 error 4 in asterisk[40+2a2000]


Debug level was unfortunately not set in asterisk.conf but verbose level
was set to 5.
Asterisk runs with:
/usr/sbin/asterisk -U asterisk -G asterisk -g

Asterisk is compiled with DONT_OPTIMIZE and BETTER_BACKTRACES options.

"core show settings" outputs:
* Directories
  -
  Configuration file:
  Configuration directory: /etc/asterisk
  Module directory:/usr/lib/asterisk/modules
  Spool directory: /var/spool/asterisk
  Log directory:   /var/log/asterisk
  Run/Sockets directory:   /var/run/asterisk
  PID file:/var/run/asterisk/asterisk.pid
  VarLib directory:/var/lib/asterisk
  Data directory:  /var/lib/asterisk
  ASTDB:   /var/lib/asterisk/astdb
  IAX2 Keys directory: /var/lib/asterisk/keys



1. Am I correct to expect a coredump file to be produced anytime asterisk
segfaults ?

2. Does Asterisk prints any WARNING or ERROR message whenever it detects,
at startup preferably, that it has not required permissions to write a
coredump file ?

3. Among above directories, which one is choosen to save coredump files ?
Is it something that can/should be configured in /etc/asterisk (I've seen
related options in some debian  /etc/default/asterisk files but I would be
curious to know if such things exist

4. Is there anything useful I can do with a line such as :
asterisk[1160]: segfault at 7efe ip 005881d6 sp
7fec95c33910 error 4 in asterisk[40+2a2000] ? Any pointer ?

5. Suggestions ?

Best regards
-- 
_
-- Bandwidth and Colocation Provided by http://www.api-digital.com --

Check out the new Asterisk community forum at: https://community.asterisk.org/

New to Asterisk? Start here:
  https://wiki.asterisk.org/wiki/display/AST/Getting+Started

asterisk-users mailing list
To UNSUBSCRIBE or update options visit:
   http://lists.digium.com/mailman/listinfo/asterisk-users