Re: [asterisk-users] Upgrading asterisk 13.7 to 13.11. Segfaults

[George Joseph]

On Tue, Sep 6, 2016 at 8:40 AM, Dmitriy Serov  wrote:

> 06.09.2016 17:08, George Joseph пишет:
>
>
>
> On Tue, Sep 6, 2016 at 7:58 AM, Dmitriy Serov  wrote:
>
>> 06.09.2016 16:42, George Joseph пишет:
>>
>>
>>
>> On Tue, Sep 6, 2016 at 7:32 AM, Dmitriy Serov 
>> wrote:
>>
>>> Hello.
>>>
>>> Several months server working on asterisk 13.7 and pjproject 2.5
>>> (installed separately). Once a day the server crashes or hangs and is
>>> familiar sores that written watchdogs.
>>>
>>> Yesterday I decided to upgrade to 13.11 and bundled pjproject (2.5.5).
>>> Solved all the problems with compilation I started asterisk several times
>>> and each time after 5-7 seconds was seg fault.
>>>
>>> So I didn't get to use the new version of asterisk. And I really wanted
>>> to be able to find and remove the cause. I would be grateful for any help.
>>>
>>>
>> All 3 of the backtraces are in different pjproject places which is
>> weird.  Makes me think there's still a library mismatch somewhere.  Is the
>> separately compiled pjproject still installed on both the build machine and
>> the server?  They *should* be ignored if --with-pjproject-bundled is
>> specified but you might want to remove them and try again.
>>
>>
>>
>>>
>> George, thank you that you responded. What I was hoping :)
>> I was also very surprised that SF can happen so quickly and so
>> consistently. With all this in completely different places.
>>
>> When installed separately pjproject source code with bundled not even
>> compiled. Of course, I uninstalled and cleaned pjproject according to this
>> instructions: https://wiki.asterisk.org/wiki
>> /display/AST/Building+and+Installing+pjproject
>>
>> At the moment I have restore separately pjproject and asterisk 13.7. But
>> there is a directory with configured and compiled asterisk 13.11. Happy to
>> provide the contents of the autoconfiguration files.
>>
>
> config.log and makeopts would be useful if you can zip them up.  Also,
> what distribution and version are you running?  As Jonathan suggested,
> could there have been any left over or third-party modules left in the
> asterisk modules directory?
>
>
> https://ruvoip.net/_other/voip/2016-09-05/makeopts_configlogs.zip
>


Went through the logs and configs and unfortunately nothing stands out.
The only things I can suggest at this point are to check that there really
are no libpj* files left in /usr/lib, no pj* header files left in
/usr/include, do a system update, make sure that all res_pj* modules are
loaded in modules.conf (maybe try autoload), etc.


>
>
> Slackware 14.1 x64 with some packages upgraded and some of libraries
> compiled from source.
>
> --
> _
> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>
> Join the Asterisk Community at the 13th AstriCon, September 27-29, 2016
>   http://www.asterisk.org/community/astricon-user-conference
>
> New to Asterisk? Start here:
>   https://wiki.asterisk.org/wiki/display/AST/Getting+Started
>
> asterisk-users mailing list
> To UNSUBSCRIBE or update options visit:
>http://lists.digium.com/mailman/listinfo/asterisk-users
>



-- 
George Joseph
Digium, Inc. | Software Developer
445 Jan Davis Drive NW - Huntsville, AL 35806 - US
Check us out at: www.digium.com & www.asterisk.org
-- 
_
-- Bandwidth and Colocation Provided by http://www.api-digital.com --

Join the Asterisk Community at the 13th AstriCon, September 27-29, 2016
  http://www.asterisk.org/community/astricon-user-conference

New to Asterisk? Start here:
  https://wiki.asterisk.org/wiki/display/AST/Getting+Started

asterisk-users mailing list
To UNSUBSCRIBE or update options visit:
   http://lists.digium.com/mailman/listinfo/asterisk-users

Re: [asterisk-users] Upgrading asterisk 13.7 to 13.11. Segfaults

[Dmitriy Serov]

06.09.2016 17:08, George Joseph пишет:



On Tue, Sep 6, 2016 at 7:58 AM, Dmitriy Serov > wrote:


06.09.2016 16:42, George Joseph пишет:



On Tue, Sep 6, 2016 at 7:32 AM, Dmitriy Serov
> wrote:

Hello.

Several months server working on asterisk 13.7 and pjproject
2.5 (installed separately). Once a day the server crashes or
hangs and is familiar sores that written watchdogs.

Yesterday I decided to upgrade to 13.11 and bundled pjproject
(2.5.5). Solved all the problems with compilation I started
asterisk several times and each time after 5-7 seconds was
seg fault.

So I didn't get to use the new version of asterisk. And I
really wanted to be able to find and remove the cause. I
would be grateful for any help.


All 3 of the backtraces are in different pjproject places which
is weird.  Makes me think there's still a library mismatch
somewhere.  Is the separately compiled pjproject still installed
on both the build machine and the server?  They /should/ be
ignored if --with-pjproject-bundled is specified but you might
want to remove them and try again.



George, thank you that you responded. What I was hoping :)
I was also very surprised that SF can happen so quickly and so
consistently. With all this in completely different places.

When installed separately pjproject source code with bundled not
even compiled. Of course, I uninstalled and cleaned pjproject
according to this instructions:
https://wiki.asterisk.org/wiki/display/AST/Building+and+Installing+pjproject



At the moment I have restore separately pjproject and asterisk
13.7. But there is a directory with configured and compiled
asterisk 13.11. Happy to provide the contents of the
autoconfiguration files.


config.log and makeopts would be useful if you can zip them up.  Also, 
what distribution and version are you running?  As Jonathan suggested, 
could there have been any left over or third-party modules left in the 
asterisk modules directory?




https://ruvoip.net/_other/voip/2016-09-05/makeopts_configlogs.zip

Slackware 14.1 x64 with some packages upgraded and some of libraries 
compiled from source.
-- 
_
-- Bandwidth and Colocation Provided by http://www.api-digital.com --

Join the Asterisk Community at the 13th AstriCon, September 27-29, 2016
  http://www.asterisk.org/community/astricon-user-conference

New to Asterisk? Start here:
  https://wiki.asterisk.org/wiki/display/AST/Getting+Started

asterisk-users mailing list
To UNSUBSCRIBE or update options visit:
   http://lists.digium.com/mailman/listinfo/asterisk-users

Re: [asterisk-users] Upgrading asterisk 13.7 to 13.11. Segfaults

[Dmitriy Serov]

06.09.2016 16:51, Jonathan H пишет:

All your libraries, kernel, headers and build tools up to date?
I can't be sure because I do not know the required versions of these 
libraries. When you configure and build asterisk, no error detected.
The server has not a new kernel 3.13.6 #4 SMP Wed Aug 20 17:52:10 MSK 
2014 x86_64




The other thing that might be worth noting is the warning along the
lines of "contains modules that were not installed by this version of
Asterisk".

Might be worth deleting anything that appears there, and then starting Asterisk.
I don't use the autoloading of modules. All the necessary modules 
written manually in modules.conf.
At the end of the installation it reported the extra modules (thanks) 
and they removed me as "garbage".



On 6 September 2016 at 14:42, George Joseph  wrote:


On Tue, Sep 6, 2016 at 7:32 AM, Dmitriy Serov  wrote:

Hello.

Several months server working on asterisk 13.7 and pjproject 2.5
(installed separately). Once a day the server crashes or hangs and is
familiar sores that written watchdogs.

Yesterday I decided to upgrade to 13.11 and bundled pjproject (2.5.5).
Solved all the problems with compilation I started asterisk several times
and each time after 5-7 seconds was seg fault.

So I didn't get to use the new version of asterisk. And I really wanted to
be able to find and remove the cause. I would be grateful for any help.


All 3 of the backtraces are in different pjproject places which is weird.
Makes me think there's still a library mismatch somewhere.  Is the
separately compiled pjproject still installed on both the build machine and
the server?  They should be ignored if --with-pjproject-bundled is specified
but you might want to remove them and try again.




first segfault:

Program terminated with signal 11, Segmentation fault.
#0  0x7fe19337357e in pj_atomic_dec_and_get (atomic_var=0x1a9a) at
../src/pj/os_core_unix.c:962
962pj_mutex_lock( atomic_var->mutex );

backtrace:
https://ruvoip.net/_other/voip/2016-09-05-1/backtrace-threads.txt

log (16 MB) : https://ruvoip.net/_other/voip/2016-09-05-1/full.txt

second segfault:

Program terminated with signal 11, Segmentation fault.
#0  0x7f5777412788 in pjsip_auth_clt_reinit_req (sess=0x7f56f4088a30,
rdata=0x7f56ec2b0c98, old_request=0x7f56f47775a8,
new_request=0x7f56d5b38980) at ../src/pjsip/sip_auth_client.c:1144
1144PJ_ASSERT_RETURN(old_request->msg->type == PJSIP_REQUEST_MSG,

backtrace:
https://ruvoip.net/_other/voip/2016-09-05-2/backtrace-threads.txt

log (15 MB) : https://ruvoip.net/_other/voip/2016-09-05-2/full.txt

third segfault:

Program terminated with signal 11, Segmentation fault.
#0  0x7f592bcad53d in pj_pool_alloc (pool=0x7f58002c, size=80) at
../include/pj/pool_i.h:60
60void *ptr = pj_pool_alloc_from_block(pool->block_list.next,
size);

backtrace:
https://ruvoip.net/_other/voip/2016-09-05-3/backtrace-threads.txt

log (11 MB) : https://ruvoip.net/_other/voip/2016-09-05-3/full.txt



--
_
-- Bandwidth and Colocation Provided by http://www.api-digital.com --

Join the Asterisk Community at the 13th AstriCon, September 27-29, 2016
  http://www.asterisk.org/community/astricon-user-conference

New to Asterisk? Start here:
  https://wiki.asterisk.org/wiki/display/AST/Getting+Started

asterisk-users mailing list
To UNSUBSCRIBE or update options visit:
   http://lists.digium.com/mailman/listinfo/asterisk-users




--
George Joseph
Digium, Inc. | Software Developer
445 Jan Davis Drive NW - Huntsville, AL 35806 - US
Check us out at: www.digium.com & www.asterisk.org


--
_
-- Bandwidth and Colocation Provided by http://www.api-digital.com --

Join the Asterisk Community at the 13th AstriCon, September 27-29, 2016
   http://www.asterisk.org/community/astricon-user-conference

New to Asterisk? Start here:
   https://wiki.asterisk.org/wiki/display/AST/Getting+Started

asterisk-users mailing list
To UNSUBSCRIBE or update options visit:
http://lists.digium.com/mailman/listinfo/asterisk-users




--
_
-- Bandwidth and Colocation Provided by http://www.api-digital.com --

Join the Asterisk Community at the 13th AstriCon, September 27-29, 2016
 http://www.asterisk.org/community/astricon-user-conference

New to Asterisk? Start here:
 https://wiki.asterisk.org/wiki/display/AST/Getting+Started

asterisk-users mailing list
To UNSUBSCRIBE or update options visit:
  http://lists.digium.com/mailman/listinfo/asterisk-users

Re: [asterisk-users] Upgrading asterisk 13.7 to 13.11. Segfaults

[George Joseph]

On Tue, Sep 6, 2016 at 7:58 AM, Dmitriy Serov  wrote:

> 06.09.2016 16:42, George Joseph пишет:
>
>
>
> On Tue, Sep 6, 2016 at 7:32 AM, Dmitriy Serov  wrote:
>
>> Hello.
>>
>> Several months server working on asterisk 13.7 and pjproject 2.5
>> (installed separately). Once a day the server crashes or hangs and is
>> familiar sores that written watchdogs.
>>
>> Yesterday I decided to upgrade to 13.11 and bundled pjproject (2.5.5).
>> Solved all the problems with compilation I started asterisk several times
>> and each time after 5-7 seconds was seg fault.
>>
>> So I didn't get to use the new version of asterisk. And I really wanted
>> to be able to find and remove the cause. I would be grateful for any help.
>>
>>
> All 3 of the backtraces are in different pjproject places which is weird.
> Makes me think there's still a library mismatch somewhere.  Is the
> separately compiled pjproject still installed on both the build machine and
> the server?  They *should* be ignored if --with-pjproject-bundled is
> specified but you might want to remove them and try again.
>
>
>
>>
> George, thank you that you responded. What I was hoping :)
> I was also very surprised that SF can happen so quickly and so
> consistently. With all this in completely different places.
>
> When installed separately pjproject source code with bundled not even
> compiled. Of course, I uninstalled and cleaned pjproject according to this
> instructions: https://wiki.asterisk.org/wiki/display/AST/Building+and+
> Installing+pjproject
>
> At the moment I have restore separately pjproject and asterisk 13.7. But
> there is a directory with configured and compiled asterisk 13.11. Happy to
> provide the contents of the autoconfiguration files.
>

config.log and makeopts would be useful if you can zip them up.  Also, what
distribution and version are you running?  As Jonathan suggested, could
there have been any left over or third-party modules left in the asterisk
modules directory?



>
-- 
_
-- Bandwidth and Colocation Provided by http://www.api-digital.com --

Join the Asterisk Community at the 13th AstriCon, September 27-29, 2016
  http://www.asterisk.org/community/astricon-user-conference

New to Asterisk? Start here:
  https://wiki.asterisk.org/wiki/display/AST/Getting+Started

asterisk-users mailing list
To UNSUBSCRIBE or update options visit:
   http://lists.digium.com/mailman/listinfo/asterisk-users

Re: [asterisk-users] Upgrading asterisk 13.7 to 13.11. Segfaults

[Dmitriy Serov]

06.09.2016 16:42, George Joseph пишет:



On Tue, Sep 6, 2016 at 7:32 AM, Dmitriy Serov > wrote:


Hello.

Several months server working on asterisk 13.7 and pjproject 2.5
(installed separately). Once a day the server crashes or hangs and
is familiar sores that written watchdogs.

Yesterday I decided to upgrade to 13.11 and bundled pjproject
(2.5.5). Solved all the problems with compilation I started
asterisk several times and each time after 5-7 seconds was seg fault.

So I didn't get to use the new version of asterisk. And I really
wanted to be able to find and remove the cause. I would be
grateful for any help.


All 3 of the backtraces are in different pjproject places which is 
weird.  Makes me think there's still a library mismatch somewhere.  Is 
the separately compiled pjproject still installed on both the build 
machine and the server?  They /should/ be ignored if 
--with-pjproject-bundled is specified but you might want to remove 
them and try again.




George, thank you that you responded. What I was hoping :)
I was also very surprised that SF can happen so quickly and so 
consistently. With all this in completely different places.


When installed separately pjproject source code with bundled not even 
compiled. Of course, I uninstalled and cleaned pjproject according to 
this instructions: 
https://wiki.asterisk.org/wiki/display/AST/Building+and+Installing+pjproject


At the moment I have restore separately pjproject and asterisk 13.7. But 
there is a directory with configured and compiled asterisk 13.11. Happy 
to provide the contents of the autoconfiguration files.
-- 
_
-- Bandwidth and Colocation Provided by http://www.api-digital.com --

Join the Asterisk Community at the 13th AstriCon, September 27-29, 2016
  http://www.asterisk.org/community/astricon-user-conference

New to Asterisk? Start here:
  https://wiki.asterisk.org/wiki/display/AST/Getting+Started

asterisk-users mailing list
To UNSUBSCRIBE or update options visit:
   http://lists.digium.com/mailman/listinfo/asterisk-users

Re: [asterisk-users] Upgrading asterisk 13.7 to 13.11. Segfaults

[Jonathan H]

All your libraries, kernel, headers and build tools up to date?

The other thing that might be worth noting is the warning along the
lines of "contains modules that were not installed by this version of
Asterisk".

Might be worth deleting anything that appears there, and then starting Asterisk.

On 6 September 2016 at 14:42, George Joseph  wrote:
>
>
> On Tue, Sep 6, 2016 at 7:32 AM, Dmitriy Serov  wrote:
>>
>> Hello.
>>
>> Several months server working on asterisk 13.7 and pjproject 2.5
>> (installed separately). Once a day the server crashes or hangs and is
>> familiar sores that written watchdogs.
>>
>> Yesterday I decided to upgrade to 13.11 and bundled pjproject (2.5.5).
>> Solved all the problems with compilation I started asterisk several times
>> and each time after 5-7 seconds was seg fault.
>>
>> So I didn't get to use the new version of asterisk. And I really wanted to
>> be able to find and remove the cause. I would be grateful for any help.
>>
>
> All 3 of the backtraces are in different pjproject places which is weird.
> Makes me think there's still a library mismatch somewhere.  Is the
> separately compiled pjproject still installed on both the build machine and
> the server?  They should be ignored if --with-pjproject-bundled is specified
> but you might want to remove them and try again.
>
>
>>
>>
>> first segfault:
>>
>> Program terminated with signal 11, Segmentation fault.
>> #0  0x7fe19337357e in pj_atomic_dec_and_get (atomic_var=0x1a9a) at
>> ../src/pj/os_core_unix.c:962
>> 962pj_mutex_lock( atomic_var->mutex );
>>
>> backtrace:
>> https://ruvoip.net/_other/voip/2016-09-05-1/backtrace-threads.txt
>>
>> log (16 MB) : https://ruvoip.net/_other/voip/2016-09-05-1/full.txt
>>
>> second segfault:
>>
>> Program terminated with signal 11, Segmentation fault.
>> #0  0x7f5777412788 in pjsip_auth_clt_reinit_req (sess=0x7f56f4088a30,
>> rdata=0x7f56ec2b0c98, old_request=0x7f56f47775a8,
>> new_request=0x7f56d5b38980) at ../src/pjsip/sip_auth_client.c:1144
>> 1144PJ_ASSERT_RETURN(old_request->msg->type == PJSIP_REQUEST_MSG,
>>
>> backtrace:
>> https://ruvoip.net/_other/voip/2016-09-05-2/backtrace-threads.txt
>>
>> log (15 MB) : https://ruvoip.net/_other/voip/2016-09-05-2/full.txt
>>
>> third segfault:
>>
>> Program terminated with signal 11, Segmentation fault.
>> #0  0x7f592bcad53d in pj_pool_alloc (pool=0x7f58002c, size=80) at
>> ../include/pj/pool_i.h:60
>> 60void *ptr = pj_pool_alloc_from_block(pool->block_list.next,
>> size);
>>
>> backtrace:
>> https://ruvoip.net/_other/voip/2016-09-05-3/backtrace-threads.txt
>>
>> log (11 MB) : https://ruvoip.net/_other/voip/2016-09-05-3/full.txt
>>
>>
>>
>> --
>> _
>> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>>
>> Join the Asterisk Community at the 13th AstriCon, September 27-29, 2016
>>  http://www.asterisk.org/community/astricon-user-conference
>>
>> New to Asterisk? Start here:
>>  https://wiki.asterisk.org/wiki/display/AST/Getting+Started
>>
>> asterisk-users mailing list
>> To UNSUBSCRIBE or update options visit:
>>   http://lists.digium.com/mailman/listinfo/asterisk-users
>
>
>
>
> --
> George Joseph
> Digium, Inc. | Software Developer
> 445 Jan Davis Drive NW - Huntsville, AL 35806 - US
> Check us out at: www.digium.com & www.asterisk.org
>
>
> --
> _
> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>
> Join the Asterisk Community at the 13th AstriCon, September 27-29, 2016
>   http://www.asterisk.org/community/astricon-user-conference
>
> New to Asterisk? Start here:
>   https://wiki.asterisk.org/wiki/display/AST/Getting+Started
>
> asterisk-users mailing list
> To UNSUBSCRIBE or update options visit:
>http://lists.digium.com/mailman/listinfo/asterisk-users

-- 
_
-- Bandwidth and Colocation Provided by http://www.api-digital.com --

Join the Asterisk Community at the 13th AstriCon, September 27-29, 2016
  http://www.asterisk.org/community/astricon-user-conference

New to Asterisk? Start here:
  https://wiki.asterisk.org/wiki/display/AST/Getting+Started

asterisk-users mailing list
To UNSUBSCRIBE or update options visit:
   http://lists.digium.com/mailman/listinfo/asterisk-users

Re: [asterisk-users] Upgrading asterisk 13.7 to 13.11. Segfaults

[George Joseph]

On Tue, Sep 6, 2016 at 7:32 AM, Dmitriy Serov  wrote:

> Hello.
>
> Several months server working on asterisk 13.7 and pjproject 2.5
> (installed separately). Once a day the server crashes or hangs and is
> familiar sores that written watchdogs.
>
> Yesterday I decided to upgrade to 13.11 and bundled pjproject (2.5.5).
> Solved all the problems with compilation I started asterisk several times
> and each time after 5-7 seconds was seg fault.
>
> So I didn't get to use the new version of asterisk. And I really wanted to
> be able to find and remove the cause. I would be grateful for any help.
>
>
All 3 of the backtraces are in different pjproject places which is weird.
Makes me think there's still a library mismatch somewhere.  Is the
separately compiled pjproject still installed on both the build machine and
the server?  They *should* be ignored if --with-pjproject-bundled is
specified but you might want to remove them and try again.



>
> first segfault:
>
> Program terminated with signal 11, Segmentation fault.
> #0  0x7fe19337357e in pj_atomic_dec_and_get (atomic_var=0x1a9a) at
> ../src/pj/os_core_unix.c:962
> 962pj_mutex_lock( atomic_var->mutex );
>
> backtrace: https://ruvoip.net/_other/voip/2016-09-05-1/backtrace-thread
> s.txt
>
> log (16 MB) : https://ruvoip.net/_other/voip/2016-09-05-1/full.txt
>
> second segfault:
>
> Program terminated with signal 11, Segmentation fault.
> #0  0x7f5777412788 in pjsip_auth_clt_reinit_req (sess=0x7f56f4088a30,
> rdata=0x7f56ec2b0c98, old_request=0x7f56f47775a8,
> new_request=0x7f56d5b38980) at ../src/pjsip/sip_auth_client.c:1144
> 1144PJ_ASSERT_RETURN(old_request->msg->type == PJSIP_REQUEST_MSG,
>
> backtrace: https://ruvoip.net/_other/voip/2016-09-05-2/backtrace-thread
> s.txt
>
> log (15 MB) : https://ruvoip.net/_other/voip/2016-09-05-2/full.txt
>
> third segfault:
>
> Program terminated with signal 11, Segmentation fault.
> #0  0x7f592bcad53d in pj_pool_alloc (pool=0x7f58002c, size=80) at
> ../include/pj/pool_i.h:60
> 60void *ptr = pj_pool_alloc_from_block(pool->block_list.next,
> size);
>
> backtrace: https://ruvoip.net/_other/voip/2016-09-05-3/backtrace-thread
> s.txt
>
> log (11 MB) : https://ruvoip.net/_other/voip/2016-09-05-3/full.txt
>
>
>
> --
> _
> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>
> Join the Asterisk Community at the 13th AstriCon, September 27-29, 2016
>  http://www.asterisk.org/community/astricon-user-conference
>
> New to Asterisk? Start here:
>  https://wiki.asterisk.org/wiki/display/AST/Getting+Started
>
> asterisk-users mailing list
> To UNSUBSCRIBE or update options visit:
>   http://lists.digium.com/mailman/listinfo/asterisk-users
>



-- 
George Joseph
Digium, Inc. | Software Developer
445 Jan Davis Drive NW - Huntsville, AL 35806 - US
Check us out at: www.digium.com & www.asterisk.org
-- 
_
-- Bandwidth and Colocation Provided by http://www.api-digital.com --

Join the Asterisk Community at the 13th AstriCon, September 27-29, 2016
  http://www.asterisk.org/community/astricon-user-conference

New to Asterisk? Start here:
  https://wiki.asterisk.org/wiki/display/AST/Getting+Started

asterisk-users mailing list
To UNSUBSCRIBE or update options visit:
   http://lists.digium.com/mailman/listinfo/asterisk-users