Re: [OpenSIPS-Devel] [opensips] [tm] OpenSIPS 2.1 Crash (#732)

[Muhammad Shahzad Shafi]

Details sent to your new email address.

---
Reply to this email directly or view it on GitHub:
https://github.com/OpenSIPS/opensips/issues/732#issuecomment-176400894___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

Re: [OpenSIPS-Devel] [opensips] OpenSIPS 2.1 Crash (#732)

[Muhammad Shahzad Shafi]

OK, i have setup a vm where crash is reproduce-able. I am sending you access 
credentials at your private email address listed here,

http://www.opensips.org/Profiles/Bogdan

Let me know if you need any assistance with this vm.

Thank you.



---
Reply to this email directly or view it on GitHub:
https://github.com/OpenSIPS/opensips/issues/732#issuecomment-175850654___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

Re: [OpenSIPS-Devel] [opensips] OpenSIPS 2.1 Crash (#732)

[Muhammad Shahzad Shafi]

OK. Let me setup a test virtual machine on another server.

Thank you.

---
Reply to this email directly or view it on GitHub:
https://github.com/OpenSIPS/opensips/issues/732#issuecomment-175767337___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

Re: [OpenSIPS-Devel] [opensips] OpenSIPS 2.1 Crash (#732)

[Muhammad Shahzad Shafi]

The core files are on private lan of a production system. However i can upload 
core dump along with binary executable files to another server where you may 
access them.

A few things about the machine where the crash is happening, not sure if these 
are of any help to you,

1. Crash happens on almost every call, even with zero load.
2. The machine is debian wheezy 32bit Xen VM with single cpu and 2 GB ram.
3. I reduced opensips children from 8 to 2 but no use. Timer partition was set 
to 4 (and never changed)
4. Binary listeners were enabled for userloc and dialog modules.
5. The opensips configuration is significantly complex with due to very complex 
network architecture with class 5 service features etc.

Thank you.

---
Reply to this email directly or view it on GitHub:
https://github.com/OpenSIPS/opensips/issues/732#issuecomment-174311659___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

Re: [OpenSIPS-Devel] [opensips] OpenSIPS 2.1 Crash (#732)

[Muhammad Shahzad Shafi]

After commenting out this parameter line in opensips.cfg, it seems to work fine 
and does not crash any more.

Let me know if you would like to debug this further.

Thank you.


---
Reply to this email directly or view it on GitHub:
https://github.com/OpenSIPS/opensips/issues/732#issuecomment-172882829___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

Re: [OpenSIPS-Devel] [opensips] OpenSIPS 2.1 Crash (#732)

[Muhammad Shahzad Shafi]

I now have opensips v2.1.2 rev. b602a7f. I still observe sigfault.

Here is the info you requested.

In first dump core we have,


GNU gdb (GDB) 7.4.1-debian
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i486-linux-gnu".
For bug reporting instructions, please see:
...
Reading symbols from /usr/local/adx-ipc/sbin/opensips...done.
[New LWP 7002]

warning: Can't read pathname for load map: Input/output error.
[Thread debugging using libthread_db enabled]
Using host libthread_db library 
"/lib/i386-linux-gnu/i686/cmov/libthread_db.so.1".
Core was generated by `/usr/local/adx-ipc/sbin/opensips -P 
/var/run/opensips/opensips.pid -m 64 -M 4 -'.
Program terminated with signal 11, Segmentation fault.
#0  0xb72038b6 in tsl (lock=0x0) at ../../mem/../fastlock.h:86
86  asm volatile(
(gdb) frame 4
#4  set_timer (new_tl=new_tl@entry=0xb2cf71b4, 
list_id=list_id@entry=RT_T1_TO_1, ext_timeout=0x0, ext_timeout@entry=0x1) at 
timer.c:884
884 void set_timer( struct timer_link *new_tl, enum lists list_id,
(gdb) print *new_tl
$1 = {next_tl = 0x0, prev_tl = 0x0, ld_tl = 0x0, time_out = 0, timer_list = 
0x0, deleted = 1, set = 65533}


In second dump core we have,


GNU gdb (GDB) 7.4.1-debian
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i486-linux-gnu".
For bug reporting instructions, please see:
...
Reading symbols from /usr/local/adx-ipc/sbin/opensips...done.
[New LWP 7003]

warning: Can't read pathname for load map: Input/output error.
[Thread debugging using libthread_db enabled]
Using host libthread_db library 
"/lib/i386-linux-gnu/i686/cmov/libthread_db.so.1".
Core was generated by `/usr/local/adx-ipc/sbin/opensips -P 
/var/run/opensips/opensips.pid -m 64 -M 4 -'.
Program terminated with signal 11, Segmentation fault.
#0  0xb72038b6 in tsl (lock=0x0) at ../../mem/../fastlock.h:86
86  asm volatile(
(gdb) frame 4
#4  set_timer (new_tl=new_tl@entry=0xb2cf71d0, 
list_id=list_id@entry=FR_INV_TIMER_LIST, 
ext_timeout=ext_timeout@entry=0xbfe01b68) at timer.c:884
884 void set_timer( struct timer_link *new_tl, enum lists list_id,
(gdb) print *new_tl
$1 = {next_tl = 0x0, prev_tl = 0x0, ld_tl = 0x0, time_out = 0, timer_list = 
0x0, deleted = 0, set = 65533}


Thank you.


---
Reply to this email directly or view it on GitHub:
https://github.com/OpenSIPS/opensips/issues/732#issuecomment-172875632___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

Re: [OpenSIPS-Devel] [opensips] OpenSIPS 2.1 Crash (#732)

[Muhammad Shahzad Shafi]

Yes, I guess. I have, 

modparam("tm", "timer_partitions", 4)

Should I disable it and test again?

Thank you.


---
Reply to this email directly or view it on GitHub:
https://github.com/OpenSIPS/opensips/issues/732#issuecomment-172877534___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

Re: [OpenSIPS-Devel] [opensips] OpenSIPS 2.1 Crash (#732)

[Muhammad Shahzad Shafi]

Actually i downloaded and installed opensips v2.1 from scratch using,

git clone https://github.com/OpenSIPS/opensips.git -b 2.1 opensips

I believe this would get updated git repo for version 2.1 which would include 
the patch for this crash problem. Let me know if i need to do something else.

I will do a fresh / latest opensips v2.1 download and install today and provide 
you with requested info regarding frame 4.

Thank you.


---
Reply to this email directly or view it on GitHub:
https://github.com/OpenSIPS/opensips/issues/732#issuecomment-172855672___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

Re: [OpenSIPS-Devel] [opensips] OpenSIPS 2.1 Crash (#732)

[Muhammad Shahzad Shafi]

I have updated opensips to v2.1.1 Git Rev. 4c775f2. I still have crash through 
frequency is a bit reduced now, previously it was crashing on every call, now 
it crashes every 2nd or 3rd call. I see two core dumps, here are their back 
traces.

Contents of core.opensips.8051.loadbalancer-dev.1452761363,

GNU gdb (GDB) 7.4.1-debian
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i486-linux-gnu".
For bug reporting instructions, please see:
...
Reading symbols from /usr/local/adx-ipc/sbin/opensips...done.
[New LWP 8051]

warning: Can't read pathname for load map: Input/output error.
[Thread debugging using libthread_db enabled]
Using host libthread_db library 
"/lib/i386-linux-gnu/i686/cmov/libthread_db.so.1".
Core was generated by `/usr/local/adx-ipc/sbin/opensips -P 
/var/run/opensips/opensips.pid -m 64 -M 4 -'.
Program terminated with signal 11, Segmentation fault.
#0  0xb712c9e2 in tsl (lock=0x0) at ../../mem/../fastlock.h:86
86  asm volatile(
(gdb) bt
#0  0xb712c9e2 in tsl (lock=0x0) at ../../mem/../fastlock.h:86
#1  get_lock (lock=) at ../../mem/../fastlock.h:176
#2  _lock (s=) at lock.h:100
#3  set_timer (ext_timeout=0xbfa9b7c8, list_id=FR_INV_TIMER_LIST, 
new_tl=0xb2c211d0) at timer.c:907
#4  set_timer (new_tl=new_tl@entry=0xb2c211d0, 
list_id=list_id@entry=FR_INV_TIMER_LIST, 
ext_timeout=ext_timeout@entry=0xbfa9b7c8) at timer.c:884
#5  0xb714b800 in reply_received (p_msg=0xb71c7300) at t_reply.c:1535
#6  0x080823e2 in forward_reply (msg=msg@entry=0xb71c7300) at forward.c:517
#7  0x080c4977 in receive_msg (
buf=0x827c720 "SIP/2.0 100 Trying\r\nVia: SIP/2.0/UDP 
172.16.171.21:5060;branch=z9hG4bKc4b9.a142e77.0;received=172.16.171.21\r\nVia: 
SIP/2.0/UDP 
192.168.1.21:6000;received=172.16.253.35;rport=6000;branch=z9hG4bK17684605"..., 
len=, rcv_info=rcv_info@entry=0xbfa9b930) at receive.c:243
#8  0x081c3227 in udp_read_req (si=0xb7177af8, bytes_read=0xbfa9b9f8) at 
net/proto_udp/proto_udp.c:190
#9  0x081b43c1 in handle_io (fm=, idx=, 
event_type=) at net/net_udp.c:260
#10 handle_io (fm=, idx=, event_type=) at net/net_udp.c:287
#11 io_wait_loop_epoll (h=, t=, repeat=) at net/../io_wait_loop.h:215
#12 udp_rcv_loop (si=si@entry=0xb7177af8) at net/net_udp.c:308
#13 0x081b6bf1 in udp_start_processes (chd_rank=chd_rank@entry=0x82400bc, 
startup_done=startup_done@entry=0x0) at net/net_udp.c:448
#14 0x080603f6 in main_loop () at main.c:723
#15 main (argc=11, argv=0xbfa9bca4) at main.c:1260


Contents of core.opensips.8052.loadbalancer-dev.1452761363,


GNU gdb (GDB) 7.4.1-debian
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i486-linux-gnu".
For bug reporting instructions, please see:
...
Reading symbols from /usr/local/adx-ipc/sbin/opensips...done.
[New LWP 8052]

warning: Can't read pathname for load map: Input/output error.
[Thread debugging using libthread_db enabled]
Using host libthread_db library 
"/lib/i386-linux-gnu/i686/cmov/libthread_db.so.1".
Core was generated by `/usr/local/adx-ipc/sbin/opensips -P 
/var/run/opensips/opensips.pid -m 64 -M 4 -'.
Program terminated with signal 11, Segmentation fault.
#0  0xb712c9e2 in tsl (lock=0x0) at ../../mem/../fastlock.h:86
86  asm volatile(
(gdb) bt
#0  0xb712c9e2 in tsl (lock=0x0) at ../../mem/../fastlock.h:86
#1  get_lock (lock=) at ../../mem/../fastlock.h:176
#2  _lock (s=) at lock.h:100
#3  set_timer (ext_timeout=0x0, list_id=RT_T1_TO_1, new_tl=0xb2c211b4) at 
timer.c:907
#4  set_timer (new_tl=new_tl@entry=0xb2c211b4, 
list_id=list_id@entry=RT_T1_TO_1, ext_timeout=0x0, ext_timeout@entry=0x1) at 
timer.c:884
#5  0xb71298b0 in _set_fr_retr (retr=, rb=0xb2c21180) at 
t_funcs.h:137
#6  start_retr (rb=0xb2c21180) at t_funcs.h:151
#7  t_forward_nonack (t=t@entry=0xb2c21068, p_msg=p_msg@entry=0xb71c6e44, 
proxy=proxy@entry=0x0) at t_fwd.c:736
#8  0xb712503d in t_relay_to (p_msg=p_msg@entry=0xb71c6e44, 
proxy=proxy@entry=0x0, flags=flags@entry=0) at t_funcs.c:247
#9  0xb713c244 in w_t_relay (p_msg=0xb71c6e44, proxy=0x0, flags=0x0) at 
tm.c:1213
#10 0x0806441c in do_action (a=a@entry=0xb7192344, msg=msg@entry=0xb71c6e44) at 
action.c:1862
#11 0x0806bd34 in run_action_list (a=0xb7192344, msg=msg@entry=0xb71c6e44) at 
action.c:169
#12 0x080dbb48 in eval_elem (e=0xb71923b8, msg=msg@entry=0xb71c6e44, 
val=val@entry=0x0) at route.c:1643
#13 0x080db439 in eval_expr (e=0xb71923b8, msg=msg@entry=0xb71c6e44, 
val=v

Re: [OpenSIPS-Devel] [opensips] OpenSIPS 2.1 Crash (#732)

[Muhammad Shahzad Shafi]

Thanks. I will test and update you guys later today.

---
Reply to this email directly or view it on GitHub:
https://github.com/OpenSIPS/opensips/issues/732#issuecomment-171283133___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

Re: [OpenSIPS-Devel] [opensips] OpenSIPS 2.1 Crash (#732)

[Muhammad Shahzad Shafi]

Guys, any update on this? I can register on opensips just fine but as soon as 
it receives an invite, it just crashes.

---
Reply to this email directly or view it on GitHub:
https://github.com/OpenSIPS/opensips/issues/732#issuecomment-166755913___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel

[OpenSIPS-Devel] [opensips] OpenSIPS 2.1 Crash (#732)

[Muhammad Shahzad Shafi]

Hi,

I recently installed opensips version 21 with git revision a173499 on a 32bit 
debian wheezy and it is keep getting crashed while acquiring lock in TM module 
Here is BT given by GDB,


Core was generated by '/usr/local/adx-ipc/sbin/opensips -P 
/var/run/opensips/opensipspid -m 256 -M 16'
Program terminated with signal 11, Segmentation fault
#0  0xb660a188 in tsl (lock=0x0) at //mem//fastlockh:86

warning: Source file is more recent than executable
▒▒/usr/src/svn-src/opensips/fastlockh:86:2617:beg:0xb660a188
(gdb) bt
#0  0xb660a188 in tsl (lock=0x0) at //mem//fastlockh:86
#1  get_lock (lock=) at //mem//fastlockh:176
#2  _lock (s=) at lockh:100
#3  set_timer (ext_timeout=0x0, list_id=RT_T1_TO_1, new_tl=0xa61072c0) at 
timerc:899
#4  set_timer (new_tl=new_tl@entry=0xa61072c0, 
list_id=list_id@entry=RT_T1_TO_1, ext_timeout=0x0, ext_timeout@entry=0x1) at 
timerc:876
#5  0xb66078db in _set_fr_retr (retr=, rb=0xa610728c) at 
t_funcsh:137
#6  start_retr (rb=0xa610728c) at t_funcsh:151
#7  t_forward_nonack (t=t@entry=0xa6107174, p_msg=p_msg@entry=0xb66aa29c, 
proxy=proxy@entry=0x0) at t_fwdc:728
#8  0xb6602f6d in t_relay_to (p_msg=p_msg@entry=0xb66aa29c, 
proxy=proxy@entry=0x0, flags=flags@entry=0) at t_funcsc:247
#9  0xb6619814 in w_t_relay (p_msg=0xb66aa29c, proxy=0x0, flags=0x0) at tmc:1213
#10 0x0806441c in do_action (a=a@entry=0xb66752d0, msg=msg@entry=0xb66aa29c) at 
actionc:1862
#11 0x0806bd34 in run_action_list (a=0xb66752d0, msg=msg@entry=0xb66aa29c) at 
actionc:169
#12 0x080dbb48 in eval_elem (e=0xb6675344, msg=msg@entry=0xb66aa29c, 
val=val@entry=0x0) at routec:1643
#13 0x080db439 in eval_expr (e=0xb6675344, msg=msg@entry=0xb66aa29c, 
val=val@entry=0x0) at routec:1988
#14 0x080db567 in eval_expr (e=0xb6675370, msg=msg@entry=0xb66aa29c, 
val=val@entry=0x0) at routec:2004
#15 0x080db421 in eval_expr (e=0xb667539c, msg=msg@entry=0xb66aa29c, 
val=val@entry=0x0) at routec:2009
#16 0x080646fa in do_action (a=a@entry=0xb66754c8, msg=msg@entry=0xb66aa29c) at 
actionc:
#17 0x0806b722 in run_action_list (msg=0xb66aa29c, a=) at 
actionc:169
#18 run_actions (a=a@entry=0xb6660dac, msg=msg@entry=0xb66aa29c) at actionc:134
#19 0x0806c106 in run_actions (msg=0xb66aa29c, a=0xb6660dac) at actionc:127
#20 run_top_route (a=a@entry=0xb6660dac, msg=msg@entry=0xb66aa29c) at 
actionc:209
#21 0x080c5182 in receive_msg (
buf=0x827c5e0 "INVITE sip:99710170@1721617121 SIP/20\r\nVia: SIP/20/UDP 
1721625335:5060;branch=z9hG4bKgQwBAxWsi;rport\r\nFrom: 
;tag=1~GfF8zam\r\nTo: sip:99710170@17216171", 
len=, rcv_info=rcv_info@entry=0xbfa16260) at receivec:196
#22 0x081c3137 in udp_read_req (si=0xb665aaf8, bytes_read=0xbfa16328) at 
net/proto_udp/proto_udpc:190
#23 0x081b42d1 in handle_io (fm=, idx=, 
event_type=) at net/net_udpc:260
#24 handle_io (fm=, idx=, event_type=) at net/net_udpc:287
#25 io_wait_loop_epoll (h=, t=, repeat=) at net//io_wait_looph:215
#26 udp_rcv_loop (si=si@entry=0xb665aaf8) at net/net_udpc:308
#27 0x081b6b01 in udp_start_processes (chd_rank=chd_rank@entry=0x823ff7c, 
startup_done=startup_done@entry=0x0) at net/net_udpc:448
#28 0x080603f6 in main_loop () at mainc:723
#29 main (argc=11, argv=0xbfa165d4) at mainc:1260


I remain available for any further info you may need regarding it

Thank you

---
Reply to this email directly or view it on GitHub:
https://github.com/OpenSIPS/opensips/issues/732___
Devel mailing list
Devel@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/devel