Re: [OpenSIPS-Users] OpenSIPS core dumps
Hi Ryan, On 16.11.2010, at 18:17, thrillerbee wrote: John, I appreciate the suggestion, but, unfortunately, I don't use t_newtran(). What led you to remove that? Hm, sorry… I was removing some functions from my opensips.cfg, then running various stress tests (REGISTER, INVITE etc.) and watching resident memory usage graph. Starting with 6.11.2010 (12:00) opensips working without t_newtrans() and everything fine: http://dev.sgu.ru/pub/without_tnewtrans.png (kKb == MB) Btw, now I use localcache for handling sip-request retransmits. With localcache resident memory usage also increases, but OpenSIPS does not crash… On Tue, Nov 16, 2010 at 9:08 AM, John Khvatov iva...@gmail.com wrote: Hello. On 10.11.2010, at 19:19, thrillerbee wrote: Well, I spoke too soon - it's not just an issue with the opensipsctl fifo calls - looks more like a memory leak. It crashed again today, but I did get some errors in the syslog this time right before the crash: Nov 10 15:42:32 core1 /usr/local/sbin/opensips[27044]: ERROR:db_flatstore:new_flat_id: no pkg memory left Nov 10 15:42:32 core1 kernel: [5508366.582447] opensips[27044]: segfault at 10 ip 7fa7ff74c21f sp 7fffdc101700 error 4 in db_flatstore.so[7fa7ff749000+5000] To be thorough, I've attached the backtrace output from print commands (although they're the same as before). I have the same issue. OpenSIPS crashes at db_flatstore.so due to lack of memory (according resident memory usage graph). Recently I removed the use of the t_newtran() function and localstore module in opensips.cfg and crash did not happens for 11 days. Earlier crash happens every two-tree days. Do you use t_newtran() in your config? -- WBR, John Khvatov ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users -- WBR, John Khvatov ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Re: [OpenSIPS-Users] OpenSIPS core dumps
Bogdan, Are these memlogs useful? Please let me know if I can provide any other info. Thanks again. On Sat, Nov 13, 2010 at 4:20 PM, thrillerbee thriller...@gmail.com wrote: Bogdan, Since I made those changes, it doesn't core dump anymore - it just runs out of memory and stops processing packets. It happened on both of my proxies - the mem dump can be downloaded here: http://www.starviewconnect.com/tmp/core1_mem_dump_20101112.gz http://www.starviewconnect.com/tmp/core2_mem_dump_20101112.gz Thanks. On Thu, Nov 11, 2010 at 12:43 PM, thrillerbee thriller...@gmail.comwrote: Bogdan, I had already increased the PKG_MEM_POOL_SIZE by *10. I'm making the changes suggested in the link you provided to try to narrow down the memory issue. I'll send over as soon as I have another crash with mem info. Thanks. On Thu, Nov 11, 2010 at 5:10 AM, Bogdan-Andrei Iancu bog...@voice-system.ro wrote: This last crash had the same bt as the one from previous email ? the prev fix I made reports this: ERROR:db_flatstore:flat_db_insert: uninitialized connection and does not crash, so this time the crash may be in a different place. Now, it seams after all that the root of your problem is the mem exhaustion. To see what causes this (if a mem leak or simply not enough mem), there is a doc - http://www.opensips.org/Resources/DocsTsMem (it is for shm, but the same for pkg). Maybe, first you should simply try to increase the pkg mem ( in config.h you have PKG_MEM_POOL_SIZE that you can increase) - you NEED to recompile and reinstall after that. If more pkg mem does not solve the problem, I can help you with using the memory debugger to see what is going on there. Regards, Bogdan thrillerbee wrote: My other proxy crashed as well with these ERRORs in the syslog: Nov 10 22:01:02 core2 /usr/local/sbin/opensips[22959]: ERROR:db_flatstore:get_name: pkg memory allocation failure Nov 10 22:01:02 core2 /usr/local/sbin/opensips[22959]: ERROR:db_flatstore:flat_reopen_connection: failed to get_name Nov 10 22:01:02 core2 /usr/local/sbin/opensips[22959]: ERROR:db_flatstore:flat_db_insert: uninitialized connection Nov 10 22:01:02 core2 /usr/local/sbin/opensips[22959]: ERROR:db_flatstore:flat_db_insert: uninitialized connection ... Nov 10 22:01:21 core2 /usr/local/sbin/opensips[22959]: ERROR:db_flatstore:flat_db_insert: uninitialized connection Nov 10 22:01:22 core2 /usr/local/sbin/opensips[22959]: ERROR:db_flatstore:flat_db_insert: uninitialized connection Nov 10 22:01:22 core2 /usr/local/sbin/opensips[22959]: ERROR:db_flatstore:new_flat_id: no pkg memory left Nov 10 22:01:22 core2 kernel: [4297088.404734] opensips[22959]: segfault at 10 ip 7f3db577e21f sp 7fffa260d640 error 4 in db_flatstore.so[7f3db577b000+5000] On Wed, Nov 10, 2010 at 10:19 AM, thrillerbee thriller...@gmail.commailto: thriller...@gmail.com wrote: Bogdan, Well, I spoke too soon - it's not just an issue with the opensipsctl fifo calls - looks more like a memory leak. It crashed again today, but I did get some errors in the syslog this time right before the crash: Nov 10 15:42:32 core1 /usr/local/sbin/opensips[27044]: ERROR:db_flatstore:new_flat_id: no pkg memory left Nov 10 15:42:32 core1 kernel: [5508366.582447] opensips[27044]: segfault at 10 ip 7fa7ff74c21f sp 7fffdc101700 error 4 in db_flatstore.so[7fa7ff749000+5000] To be thorough, I've attached the backtrace output from print commands (although they're the same as before). To answer your question, yes - I do use the flat_rotate MI command. Thanks again. On Wed, Nov 10, 2010 at 4:04 AM, Bogdan-Andrei Iancu bog...@voice-system.ro mailto:bog...@voice-system.ro wrote: Hi, opensipsctl takes care that each command takes a separate fifo reply, so here it should be no problem. But the problem may be when comes with sending multiple commands (via FIFO) in the same time - this translates into parallel writes to the same file and depends on the atomicity of the write op. But in the worst case, a mixture at the FIFO level may lead to bogus command and not in any kind of crashDo you use the flat_rotate MI command ? Regards, Bogdan thrillerbee wrote: Bogdan, It seems the issue is with 'opensipsctl fifo' - it's very sensitive to simultaneous calls. Basically, I've combined all my scripts to prevent 'opensipsctl fifo' from being called too frequently and that seems (so far) to have mitigated the issue. Is there anything one should know about how (not) to use /opensipsctl/? Thanks. On Mon, Nov 8, 2010 at 6:07 AM, Bogdan-Andrei Iancu bog...@voice-system.ro mailto:bog...@voice-system.ro mailto:bog...@voice-system.ro
Re: [OpenSIPS-Users] OpenSIPS core dumps
Hi, opensipsctl takes care that each command takes a separate fifo reply, so here it should be no problem. But the problem may be when comes with sending multiple commands (via FIFO) in the same time - this translates into parallel writes to the same file and depends on the atomicity of the write op. But in the worst case, a mixture at the FIFO level may lead to bogus command and not in any kind of crashDo you use the flat_rotate MI command ? Regards, Bogdan thrillerbee wrote: Bogdan, It seems the issue is with 'opensipsctl fifo' - it's very sensitive to simultaneous calls. Basically, I've combined all my scripts to prevent 'opensipsctl fifo' from being called too frequently and that seems (so far) to have mitigated the issue. Is there anything one should know about how (not) to use /opensipsctl/? Thanks. On Mon, Nov 8, 2010 at 6:07 AM, Bogdan-Andrei Iancu bog...@voice-system.ro mailto:bog...@voice-system.ro wrote: Hi, strange if you do not have any errors :( I just made a fix on both trunk and 1.6 to extend some checks in flatstore and prevent crashing (even if the DB op will not be executed). Could you update from SVN and see if stops crashing ? Regards, Bogdan thrillerbee wrote: Bogdan, I am not seeing any other errors in the logs. Is there anything else I can look at? Both proxies are crashing everyday. Thanks. On Wed, Nov 3, 2010 at 4:21 AM, Bogdan-Andrei Iancu bog...@voice-system.ro mailto:bog...@voice-system.ro mailto:bog...@voice-system.ro mailto:bog...@voice-system.ro wrote: I got some clue about what is happening - as you are using flatstore for acc, the acc module does not check the success of the use_table DB operation - in 90% of the cases (for mysql, postgres, etc) this function cannot fail, but it seams that for flatstore can. And if it fails, the h-tail is set to NULL, leading to crash. Now, before the crash itself, do you see any other ERROR messages (even long before the crash) related to flatstore module ? Try: grep ERROR opensips_log_file | grep flatstore Regards, Bogdan thrillerbee wrote: Bogdan, One more detail that may help - I added another OpenSIPS proxy in parallel with this one ( load balancing between the two) and now both OpenSIPS proxies crash at the same time (within a couple seconds). I can provide more core dumps if it will help. Thanks. On Tue, Nov 2, 2010 at 9:02 AM, thrillerbee thriller...@gmail.com mailto:thriller...@gmail.com mailto:thriller...@gmail.com mailto:thriller...@gmail.com mailto:thriller...@gmail.com mailto:thriller...@gmail.com mailto:thriller...@gmail.com mailto:thriller...@gmail.com wrote: Bogdan, Below is the info requested: (gdb) frame 0 #0 0x7f51999f221f in flat_db_insert (h=0x7f0978, k=0x7f5198732a60, v=0x7f5198732d20, n=19) at flatstore.c:165 165 f = CON_FILE(h); (gdb) print h $1 = (const db_con_t *) 0x7f0978 (gdb) print h-tail $2 = 0 (gdb) print ((struct flat_con*)(h-tail))-file Cannot access memory at address 0x10 Before each of the crashes yesterday, I saw these in the logs: Nov 1 14:17:40 core1 kernel: [4287745.452111] opensips[22141]: segfault at 10 ip 7f51999f221f sp 7fffbcd8d510 error 4 in db_flatstore.so[7f51999ef000+5000] Nov 1 23:52:58 core1 kernel: [4348562.990735] opensips[26978]: segfault at 10 ip 7f726cb9b21f sp 7083f6f0 error 4 in db_flatstore.so[7f726cb98000+5000] Are there any compiler flags I should use for debugging? (gdb) info locals f = value optimized out i = value optimized out l = value optimized out p = value optimized out __FUNCTION__ = flat_db_insert Thanks again. On Tue, Nov 2, 2010 at 4:18 AM, Bogdan-Andrei Iancu bog...@voice-system.ro mailto:bog...@voice-system.ro mailto:bog...@voice-system.ro mailto:bog...@voice-system.ro mailto:bog...@voice-system.ro mailto:bog...@voice-system.ro mailto:bog...@voice-system.ro mailto:bog...@voice-system.ro wrote: Hi, in frame 0, could you print: h h-tail ((struct
Re: [OpenSIPS-Users] OpenSIPS core dumps
Hello. On 10.11.2010, at 19:19, thrillerbee wrote: Well, I spoke too soon - it's not just an issue with the opensipsctl fifo calls - looks more like a memory leak. It crashed again today, but I did get some errors in the syslog this time right before the crash: Nov 10 15:42:32 core1 /usr/local/sbin/opensips[27044]: ERROR:db_flatstore:new_flat_id: no pkg memory left Nov 10 15:42:32 core1 kernel: [5508366.582447] opensips[27044]: segfault at 10 ip 7fa7ff74c21f sp 7fffdc101700 error 4 in db_flatstore.so[7fa7ff749000+5000] To be thorough, I've attached the backtrace output from print commands (although they're the same as before). I have the same issue. OpenSIPS crashes at db_flatstore.so due to lack of memory (according resident memory usage graph). Recently I removed the use of the t_newtran() function and localstore module in opensips.cfg and crash did not happens for 11 days. Earlier crash happens every two-tree days. Do you use t_newtran() in your config? -- WBR, John Khvatov ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Re: [OpenSIPS-Users] OpenSIPS core dumps
John, I appreciate the suggestion, but, unfortunately, I don't use t_newtran(). What led you to remove that? Thanks, Ryan On Tue, Nov 16, 2010 at 9:08 AM, John Khvatov iva...@gmail.com wrote: Hello. On 10.11.2010, at 19:19, thrillerbee wrote: Well, I spoke too soon - it's not just an issue with the opensipsctl fifo calls - looks more like a memory leak. It crashed again today, but I did get some errors in the syslog this time right before the crash: Nov 10 15:42:32 core1 /usr/local/sbin/opensips[27044]: ERROR:db_flatstore:new_flat_id: no pkg memory left Nov 10 15:42:32 core1 kernel: [5508366.582447] opensips[27044]: segfault at 10 ip 7fa7ff74c21f sp 7fffdc101700 error 4 in db_flatstore.so[7fa7ff749000+5000] To be thorough, I've attached the backtrace output from print commands (although they're the same as before). I have the same issue. OpenSIPS crashes at db_flatstore.so due to lack of memory (according resident memory usage graph). Recently I removed the use of the t_newtran() function and localstore module in opensips.cfg and crash did not happens for 11 days. Earlier crash happens every two-tree days. Do you use t_newtran() in your config? -- WBR, John Khvatov ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Re: [OpenSIPS-Users] OpenSIPS core dumps
Bogdan, Since I made those changes, it doesn't core dump anymore - it just runs out of memory and stops processing packets. It happened on both of my proxies - the mem dump can be downloaded here: http://www.starviewconnect.com/tmp/core1_mem_dump_20101112.gz http://www.starviewconnect.com/tmp/core2_mem_dump_20101112.gz Thanks. On Thu, Nov 11, 2010 at 12:43 PM, thrillerbee thriller...@gmail.com wrote: Bogdan, I had already increased the PKG_MEM_POOL_SIZE by *10. I'm making the changes suggested in the link you provided to try to narrow down the memory issue. I'll send over as soon as I have another crash with mem info. Thanks. On Thu, Nov 11, 2010 at 5:10 AM, Bogdan-Andrei Iancu bog...@voice-system.ro wrote: This last crash had the same bt as the one from previous email ? the prev fix I made reports this: ERROR:db_flatstore:flat_db_insert: uninitialized connection and does not crash, so this time the crash may be in a different place. Now, it seams after all that the root of your problem is the mem exhaustion. To see what causes this (if a mem leak or simply not enough mem), there is a doc - http://www.opensips.org/Resources/DocsTsMem (it is for shm, but the same for pkg). Maybe, first you should simply try to increase the pkg mem ( in config.h you have PKG_MEM_POOL_SIZE that you can increase) - you NEED to recompile and reinstall after that. If more pkg mem does not solve the problem, I can help you with using the memory debugger to see what is going on there. Regards, Bogdan thrillerbee wrote: My other proxy crashed as well with these ERRORs in the syslog: Nov 10 22:01:02 core2 /usr/local/sbin/opensips[22959]: ERROR:db_flatstore:get_name: pkg memory allocation failure Nov 10 22:01:02 core2 /usr/local/sbin/opensips[22959]: ERROR:db_flatstore:flat_reopen_connection: failed to get_name Nov 10 22:01:02 core2 /usr/local/sbin/opensips[22959]: ERROR:db_flatstore:flat_db_insert: uninitialized connection Nov 10 22:01:02 core2 /usr/local/sbin/opensips[22959]: ERROR:db_flatstore:flat_db_insert: uninitialized connection ... Nov 10 22:01:21 core2 /usr/local/sbin/opensips[22959]: ERROR:db_flatstore:flat_db_insert: uninitialized connection Nov 10 22:01:22 core2 /usr/local/sbin/opensips[22959]: ERROR:db_flatstore:flat_db_insert: uninitialized connection Nov 10 22:01:22 core2 /usr/local/sbin/opensips[22959]: ERROR:db_flatstore:new_flat_id: no pkg memory left Nov 10 22:01:22 core2 kernel: [4297088.404734] opensips[22959]: segfault at 10 ip 7f3db577e21f sp 7fffa260d640 error 4 in db_flatstore.so[7f3db577b000+5000] On Wed, Nov 10, 2010 at 10:19 AM, thrillerbee thriller...@gmail.commailto: thriller...@gmail.com wrote: Bogdan, Well, I spoke too soon - it's not just an issue with the opensipsctl fifo calls - looks more like a memory leak. It crashed again today, but I did get some errors in the syslog this time right before the crash: Nov 10 15:42:32 core1 /usr/local/sbin/opensips[27044]: ERROR:db_flatstore:new_flat_id: no pkg memory left Nov 10 15:42:32 core1 kernel: [5508366.582447] opensips[27044]: segfault at 10 ip 7fa7ff74c21f sp 7fffdc101700 error 4 in db_flatstore.so[7fa7ff749000+5000] To be thorough, I've attached the backtrace output from print commands (although they're the same as before). To answer your question, yes - I do use the flat_rotate MI command. Thanks again. On Wed, Nov 10, 2010 at 4:04 AM, Bogdan-Andrei Iancu bog...@voice-system.ro mailto:bog...@voice-system.ro wrote: Hi, opensipsctl takes care that each command takes a separate fifo reply, so here it should be no problem. But the problem may be when comes with sending multiple commands (via FIFO) in the same time - this translates into parallel writes to the same file and depends on the atomicity of the write op. But in the worst case, a mixture at the FIFO level may lead to bogus command and not in any kind of crashDo you use the flat_rotate MI command ? Regards, Bogdan thrillerbee wrote: Bogdan, It seems the issue is with 'opensipsctl fifo' - it's very sensitive to simultaneous calls. Basically, I've combined all my scripts to prevent 'opensipsctl fifo' from being called too frequently and that seems (so far) to have mitigated the issue. Is there anything one should know about how (not) to use /opensipsctl/? Thanks. On Mon, Nov 8, 2010 at 6:07 AM, Bogdan-Andrei Iancu bog...@voice-system.ro mailto:bog...@voice-system.ro mailto:bog...@voice-system.ro mailto:bog...@voice-system.ro wrote: Hi, strange if you do not have any errors :( I just made a fix on both trunk and 1.6 to extend some checks in
Re: [OpenSIPS-Users] OpenSIPS core dumps
Bogdan, I had already increased the PKG_MEM_POOL_SIZE by *10. I'm making the changes suggested in the link you provided to try to narrow down the memory issue. I'll send over as soon as I have another crash with mem info. Thanks. On Thu, Nov 11, 2010 at 5:10 AM, Bogdan-Andrei Iancu bog...@voice-system.ro wrote: This last crash had the same bt as the one from previous email ? the prev fix I made reports this: ERROR:db_flatstore:flat_db_insert: uninitialized connection and does not crash, so this time the crash may be in a different place. Now, it seams after all that the root of your problem is the mem exhaustion. To see what causes this (if a mem leak or simply not enough mem), there is a doc - http://www.opensips.org/Resources/DocsTsMem (it is for shm, but the same for pkg). Maybe, first you should simply try to increase the pkg mem ( in config.h you have PKG_MEM_POOL_SIZE that you can increase) - you NEED to recompile and reinstall after that. If more pkg mem does not solve the problem, I can help you with using the memory debugger to see what is going on there. Regards, Bogdan thrillerbee wrote: My other proxy crashed as well with these ERRORs in the syslog: Nov 10 22:01:02 core2 /usr/local/sbin/opensips[22959]: ERROR:db_flatstore:get_name: pkg memory allocation failure Nov 10 22:01:02 core2 /usr/local/sbin/opensips[22959]: ERROR:db_flatstore:flat_reopen_connection: failed to get_name Nov 10 22:01:02 core2 /usr/local/sbin/opensips[22959]: ERROR:db_flatstore:flat_db_insert: uninitialized connection Nov 10 22:01:02 core2 /usr/local/sbin/opensips[22959]: ERROR:db_flatstore:flat_db_insert: uninitialized connection ... Nov 10 22:01:21 core2 /usr/local/sbin/opensips[22959]: ERROR:db_flatstore:flat_db_insert: uninitialized connection Nov 10 22:01:22 core2 /usr/local/sbin/opensips[22959]: ERROR:db_flatstore:flat_db_insert: uninitialized connection Nov 10 22:01:22 core2 /usr/local/sbin/opensips[22959]: ERROR:db_flatstore:new_flat_id: no pkg memory left Nov 10 22:01:22 core2 kernel: [4297088.404734] opensips[22959]: segfault at 10 ip 7f3db577e21f sp 7fffa260d640 error 4 in db_flatstore.so[7f3db577b000+5000] On Wed, Nov 10, 2010 at 10:19 AM, thrillerbee thriller...@gmail.commailto: thriller...@gmail.com wrote: Bogdan, Well, I spoke too soon - it's not just an issue with the opensipsctl fifo calls - looks more like a memory leak. It crashed again today, but I did get some errors in the syslog this time right before the crash: Nov 10 15:42:32 core1 /usr/local/sbin/opensips[27044]: ERROR:db_flatstore:new_flat_id: no pkg memory left Nov 10 15:42:32 core1 kernel: [5508366.582447] opensips[27044]: segfault at 10 ip 7fa7ff74c21f sp 7fffdc101700 error 4 in db_flatstore.so[7fa7ff749000+5000] To be thorough, I've attached the backtrace output from print commands (although they're the same as before). To answer your question, yes - I do use the flat_rotate MI command. Thanks again. On Wed, Nov 10, 2010 at 4:04 AM, Bogdan-Andrei Iancu bog...@voice-system.ro mailto:bog...@voice-system.ro wrote: Hi, opensipsctl takes care that each command takes a separate fifo reply, so here it should be no problem. But the problem may be when comes with sending multiple commands (via FIFO) in the same time - this translates into parallel writes to the same file and depends on the atomicity of the write op. But in the worst case, a mixture at the FIFO level may lead to bogus command and not in any kind of crashDo you use the flat_rotate MI command ? Regards, Bogdan thrillerbee wrote: Bogdan, It seems the issue is with 'opensipsctl fifo' - it's very sensitive to simultaneous calls. Basically, I've combined all my scripts to prevent 'opensipsctl fifo' from being called too frequently and that seems (so far) to have mitigated the issue. Is there anything one should know about how (not) to use /opensipsctl/? Thanks. On Mon, Nov 8, 2010 at 6:07 AM, Bogdan-Andrei Iancu bog...@voice-system.ro mailto:bog...@voice-system.ro mailto:bog...@voice-system.ro mailto:bog...@voice-system.ro wrote: Hi, strange if you do not have any errors :( I just made a fix on both trunk and 1.6 to extend some checks in flatstore and prevent crashing (even if the DB op will not be executed). Could you update from SVN and see if stops crashing ? Regards, Bogdan ___ Users mailing list Users@lists.opensips.org
Re: [OpenSIPS-Users] OpenSIPS core dumps
Bogdan,
Well, I spoke too soon - it's not just an issue with the opensipsctl fifo
calls - looks more like a memory leak. It crashed again today, but I did
get some errors in the syslog this time right before the crash:
Nov 10 15:42:32 core1 /usr/local/sbin/opensips[27044]:
ERROR:db_flatstore:new_flat_id: no pkg memory left
Nov 10 15:42:32 core1 kernel: [5508366.582447] opensips[27044]: segfault at
10 ip 7fa7ff74c21f sp 7fffdc101700 error 4 in
db_flatstore.so[7fa7ff749000+5000]
To be thorough, I've attached the backtrace output from print commands
(although they're the same as before).
To answer your question, yes - I do use the flat_rotate MI command.
Thanks again.
On Wed, Nov 10, 2010 at 4:04 AM, Bogdan-Andrei Iancu bog...@voice-system.ro
wrote:
Hi,
opensipsctl takes care that each command takes a separate fifo reply, so
here it should be no problem. But the problem may be when comes with sending
multiple commands (via FIFO) in the same time - this translates into
parallel writes to the same file and depends on the atomicity of the write
op.
But in the worst case, a mixture at the FIFO level may lead to bogus
command and not in any kind of crashDo you use the flat_rotate MI
command ?
Regards,
Bogdan
thrillerbee wrote:
Bogdan,
It seems the issue is with 'opensipsctl fifo' - it's very sensitive to
simultaneous calls. Basically, I've combined all my scripts to prevent
'opensipsctl fifo' from being called too frequently and that seems (so far)
to have mitigated the issue. Is there anything one should know about how
(not) to use /opensipsctl/?
Thanks.
On Mon, Nov 8, 2010 at 6:07 AM, Bogdan-Andrei Iancu
bog...@voice-system.ro mailto:bog...@voice-system.ro wrote:
Hi,
strange if you do not have any errors :(
I just made a fix on both trunk and 1.6 to extend some checks in
flatstore and prevent crashing (even if the DB op will not be
executed).
Could you update from SVN and see if stops crashing ?
Regards,
Bogdan
Core was generated by `/usr/local/sbin/opensips -P
/var/run/opensips/opensips.pid -m 512 -u root -g ro'.
Program terminated with signal 11, Segmentation fault.
[New process 27044]
#0 0x7fa7ff74c21f in flat_db_insert (h=0x7efb38, k=0x7fa7fe48ca60,
v=0x7fa7fe48cd20, n=19) at flatstore.c:165
165 f = CON_FILE(h);
(gdb) bt full
#0 0x7fa7ff74c21f in flat_db_insert (h=0x7efb38, k=0x7fa7fe48ca60,
v=0x7fa7fe48cd20, n=19) at flatstore.c:165
f = value optimized out
i = value optimized out
l = value optimized out
p = value optimized out
__FUNCTION__ = flat_db_insert
#1 0x7fa7fe2737ef in acc_db_request (rq=0x7fa7de27a978, rpl=value
optimized out) at acc.c:364
m = 19
n = value optimized out
i = value optimized out
my_ps = (db_ps_t) 0x0
__FUNCTION__ = acc_db_request
#2 0x7fa7fe27869e in tmcb_func (t=value optimized out, type=value
optimized out, ps=value optimized out) at acc_logic.c:386
No locals.
#3 0x7fa7ff1014a2 in run_trans_callbacks (type=256, trans=0x7fa7df8a6e00,
req=0x7fa7de27a978, rpl=0x7efef0, code=200) at t_hooks.c:208
cbp = (struct tm_callback *) 0x7fa7dc84f6f0
backup = (struct usr_avp **) 0x771f48
trans_backup = (struct cell *) 0x7fa7df8a6e00
__FUNCTION__ = run_trans_callbacks
#4 0x7fa7ff1015c7 in run_trans_callbacks_locked (type=256,
trans=0x7fa7df8a6e00, req=0x7fa7de27a978, rpl=0x7efef0, code=200) at
t_hooks.c:254
No locals.
#5 0x7fa7ff11ea38 in relay_reply (t=0x7fa7df8a6e00, p_msg=value optimized
out, branch=1, msg_status=200, cancel_bitmap=0x7fffdc101ad8) at t_reply.c:1257
relay = 1
save_clone = 0
buf = 0x1184390 SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
24.121.80.36;branch=z9hG4bKadcb.35a7dc2.0\r\nVia: SIP/2.0/UDP
184.106.205.223;branch=z9hG4bKadcb.be5e7083.0\r\nVia: SIP/2.0/UDP
216.18.222.3;branch=z9hG4bKadcb.18d6b484.0\r...
res_len = 1391
relayed_code = 200
relayed_msg = (struct sip_msg *) 0x7efef0
bm = {to_tag_val = {s = 0x1 Address 0x1 out of bounds, len = 8322800}}
totag_retr = value optimized out
reply_status = RPS_COMPLETED
cb_s = {
s = 0x1184390 SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
24.121.80.36;branch=z9hG4bKadcb.35a7dc2.0\r\nVia: SIP/2.0/UDP
184.106.205.223;branch=z9hG4bKadcb.be5e7083.0\r\nVia: SIP/2.0/UDP
216.18.222.3;branch=z9hG4bKadcb.18d6b484.0\r..., len = 1391}
text = {s = 0x18f Address 0x18f out of bounds, len = 1452}
__FUNCTION__ = relay_reply
#6 0x7fa7ff11f548 in reply_received (p_msg=0x7efef0) at t_reply.c:1502
last_uac_status = value optimized out
branch = 1
reply_status = value optimized out
timer = value optimized out
cancel_bitmap = 0
t = (struct cell *) 0x7fa7df8a6e00
backup_list = value optimized out
has_reply_route = 3692375232
__FUNCTION__ = reply_received
Re: [OpenSIPS-Users] OpenSIPS core dumps
My other proxy crashed as well with these ERRORs in the syslog: Nov 10 22:01:02 core2 /usr/local/sbin/opensips[22959]: ERROR:db_flatstore:get_name: pkg memory allocation failure Nov 10 22:01:02 core2 /usr/local/sbin/opensips[22959]: ERROR:db_flatstore:flat_reopen_connection: failed to get_name Nov 10 22:01:02 core2 /usr/local/sbin/opensips[22959]: ERROR:db_flatstore:flat_db_insert: uninitialized connection Nov 10 22:01:02 core2 /usr/local/sbin/opensips[22959]: ERROR:db_flatstore:flat_db_insert: uninitialized connection ... Nov 10 22:01:21 core2 /usr/local/sbin/opensips[22959]: ERROR:db_flatstore:flat_db_insert: uninitialized connection Nov 10 22:01:22 core2 /usr/local/sbin/opensips[22959]: ERROR:db_flatstore:flat_db_insert: uninitialized connection Nov 10 22:01:22 core2 /usr/local/sbin/opensips[22959]: ERROR:db_flatstore:new_flat_id: no pkg memory left Nov 10 22:01:22 core2 kernel: [4297088.404734] opensips[22959]: segfault at 10 ip 7f3db577e21f sp 7fffa260d640 error 4 in db_flatstore.so[7f3db577b000+5000] On Wed, Nov 10, 2010 at 10:19 AM, thrillerbee thriller...@gmail.com wrote: Bogdan, Well, I spoke too soon - it's not just an issue with the opensipsctl fifo calls - looks more like a memory leak. It crashed again today, but I did get some errors in the syslog this time right before the crash: Nov 10 15:42:32 core1 /usr/local/sbin/opensips[27044]: ERROR:db_flatstore:new_flat_id: no pkg memory left Nov 10 15:42:32 core1 kernel: [5508366.582447] opensips[27044]: segfault at 10 ip 7fa7ff74c21f sp 7fffdc101700 error 4 in db_flatstore.so[7fa7ff749000+5000] To be thorough, I've attached the backtrace output from print commands (although they're the same as before). To answer your question, yes - I do use the flat_rotate MI command. Thanks again. On Wed, Nov 10, 2010 at 4:04 AM, Bogdan-Andrei Iancu bog...@voice-system.ro wrote: Hi, opensipsctl takes care that each command takes a separate fifo reply, so here it should be no problem. But the problem may be when comes with sending multiple commands (via FIFO) in the same time - this translates into parallel writes to the same file and depends on the atomicity of the write op. But in the worst case, a mixture at the FIFO level may lead to bogus command and not in any kind of crashDo you use the flat_rotate MI command ? Regards, Bogdan thrillerbee wrote: Bogdan, It seems the issue is with 'opensipsctl fifo' - it's very sensitive to simultaneous calls. Basically, I've combined all my scripts to prevent 'opensipsctl fifo' from being called too frequently and that seems (so far) to have mitigated the issue. Is there anything one should know about how (not) to use /opensipsctl/? Thanks. On Mon, Nov 8, 2010 at 6:07 AM, Bogdan-Andrei Iancu bog...@voice-system.ro mailto:bog...@voice-system.ro wrote: Hi, strange if you do not have any errors :( I just made a fix on both trunk and 1.6 to extend some checks in flatstore and prevent crashing (even if the DB op will not be executed). Could you update from SVN and see if stops crashing ? Regards, Bogdan ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Re: [OpenSIPS-Users] OpenSIPS core dumps
Bogdan, It seems the issue is with 'opensipsctl fifo' - it's very sensitive to simultaneous calls. Basically, I've combined all my scripts to prevent 'opensipsctl fifo' from being called too frequently and that seems (so far) to have mitigated the issue. Is there anything one should know about how (not) to use *opensipsctl*? Thanks. On Mon, Nov 8, 2010 at 6:07 AM, Bogdan-Andrei Iancu bog...@voice-system.rowrote: Hi, strange if you do not have any errors :( I just made a fix on both trunk and 1.6 to extend some checks in flatstore and prevent crashing (even if the DB op will not be executed). Could you update from SVN and see if stops crashing ? Regards, Bogdan thrillerbee wrote: Bogdan, I am not seeing any other errors in the logs. Is there anything else I can look at? Both proxies are crashing everyday. Thanks. On Wed, Nov 3, 2010 at 4:21 AM, Bogdan-Andrei Iancu bog...@voice-system.ro mailto:bog...@voice-system.ro wrote: I got some clue about what is happening - as you are using flatstore for acc, the acc module does not check the success of the use_table DB operation - in 90% of the cases (for mysql, postgres, etc) this function cannot fail, but it seams that for flatstore can. And if it fails, the h-tail is set to NULL, leading to crash. Now, before the crash itself, do you see any other ERROR messages (even long before the crash) related to flatstore module ? Try: grep ERROR opensips_log_file | grep flatstore Regards, Bogdan thrillerbee wrote: Bogdan, One more detail that may help - I added another OpenSIPS proxy in parallel with this one ( load balancing between the two) and now both OpenSIPS proxies crash at the same time (within a couple seconds). I can provide more core dumps if it will help. Thanks. On Tue, Nov 2, 2010 at 9:02 AM, thrillerbee thriller...@gmail.com mailto:thriller...@gmail.com mailto:thriller...@gmail.com mailto:thriller...@gmail.com wrote: Bogdan, Below is the info requested: (gdb) frame 0 #0 0x7f51999f221f in flat_db_insert (h=0x7f0978, k=0x7f5198732a60, v=0x7f5198732d20, n=19) at flatstore.c:165 165 f = CON_FILE(h); (gdb) print h $1 = (const db_con_t *) 0x7f0978 (gdb) print h-tail $2 = 0 (gdb) print ((struct flat_con*)(h-tail))-file Cannot access memory at address 0x10 Before each of the crashes yesterday, I saw these in the logs: Nov 1 14:17:40 core1 kernel: [4287745.452111] opensips[22141]: segfault at 10 ip 7f51999f221f sp 7fffbcd8d510 error 4 in db_flatstore.so[7f51999ef000+5000] Nov 1 23:52:58 core1 kernel: [4348562.990735] opensips[26978]: segfault at 10 ip 7f726cb9b21f sp 7083f6f0 error 4 in db_flatstore.so[7f726cb98000+5000] Are there any compiler flags I should use for debugging? (gdb) info locals f = value optimized out i = value optimized out l = value optimized out p = value optimized out __FUNCTION__ = flat_db_insert Thanks again. On Tue, Nov 2, 2010 at 4:18 AM, Bogdan-Andrei Iancu bog...@voice-system.ro mailto:bog...@voice-system.ro mailto:bog...@voice-system.ro mailto:bog...@voice-system.ro wrote: Hi, in frame 0, could you print: h h-tail ((struct flat_con*)(h-tail))-file Also, before crash, do you see in the logs any errors from the db_flatstore module ? Regards, Bogdan thrillerbee wrote: Bogdan, It crashed again tonight. I've attached the backtrace. Thanks. On Mon, Nov 1, 2010 at 9:32 AM, thrillerbee thriller...@gmail.com mailto:thriller...@gmail.com mailto:thriller...@gmail.com mailto:thriller...@gmail.com mailto:thriller...@gmail.com mailto:thriller...@gmail.com mailto:thriller...@gmail.com mailto:thriller...@gmail.com wrote: Bogdan, Yes, I've attached 2 to my response - one was a crash from 10/29. The other occurred a few minutes ago. Thanks. On Fri, Oct 22, 2010 at 11:41 AM, Bogdan-Andrei Iancu bog...@voice-system.ro
Re: [OpenSIPS-Users] OpenSIPS core dumps
Hi, strange if you do not have any errors :( I just made a fix on both trunk and 1.6 to extend some checks in flatstore and prevent crashing (even if the DB op will not be executed). Could you update from SVN and see if stops crashing ? Regards, Bogdan thrillerbee wrote: Bogdan, I am not seeing any other errors in the logs. Is there anything else I can look at? Both proxies are crashing everyday. Thanks. On Wed, Nov 3, 2010 at 4:21 AM, Bogdan-Andrei Iancu bog...@voice-system.ro mailto:bog...@voice-system.ro wrote: I got some clue about what is happening - as you are using flatstore for acc, the acc module does not check the success of the use_table DB operation - in 90% of the cases (for mysql, postgres, etc) this function cannot fail, but it seams that for flatstore can. And if it fails, the h-tail is set to NULL, leading to crash. Now, before the crash itself, do you see any other ERROR messages (even long before the crash) related to flatstore module ? Try: grep ERROR opensips_log_file | grep flatstore Regards, Bogdan thrillerbee wrote: Bogdan, One more detail that may help - I added another OpenSIPS proxy in parallel with this one ( load balancing between the two) and now both OpenSIPS proxies crash at the same time (within a couple seconds). I can provide more core dumps if it will help. Thanks. On Tue, Nov 2, 2010 at 9:02 AM, thrillerbee thriller...@gmail.com mailto:thriller...@gmail.com mailto:thriller...@gmail.com mailto:thriller...@gmail.com wrote: Bogdan, Below is the info requested: (gdb) frame 0 #0 0x7f51999f221f in flat_db_insert (h=0x7f0978, k=0x7f5198732a60, v=0x7f5198732d20, n=19) at flatstore.c:165 165 f = CON_FILE(h); (gdb) print h $1 = (const db_con_t *) 0x7f0978 (gdb) print h-tail $2 = 0 (gdb) print ((struct flat_con*)(h-tail))-file Cannot access memory at address 0x10 Before each of the crashes yesterday, I saw these in the logs: Nov 1 14:17:40 core1 kernel: [4287745.452111] opensips[22141]: segfault at 10 ip 7f51999f221f sp 7fffbcd8d510 error 4 in db_flatstore.so[7f51999ef000+5000] Nov 1 23:52:58 core1 kernel: [4348562.990735] opensips[26978]: segfault at 10 ip 7f726cb9b21f sp 7083f6f0 error 4 in db_flatstore.so[7f726cb98000+5000] Are there any compiler flags I should use for debugging? (gdb) info locals f = value optimized out i = value optimized out l = value optimized out p = value optimized out __FUNCTION__ = flat_db_insert Thanks again. On Tue, Nov 2, 2010 at 4:18 AM, Bogdan-Andrei Iancu bog...@voice-system.ro mailto:bog...@voice-system.ro mailto:bog...@voice-system.ro mailto:bog...@voice-system.ro wrote: Hi, in frame 0, could you print: h h-tail ((struct flat_con*)(h-tail))-file Also, before crash, do you see in the logs any errors from the db_flatstore module ? Regards, Bogdan thrillerbee wrote: Bogdan, It crashed again tonight. I've attached the backtrace. Thanks. On Mon, Nov 1, 2010 at 9:32 AM, thrillerbee thriller...@gmail.com mailto:thriller...@gmail.com mailto:thriller...@gmail.com mailto:thriller...@gmail.com mailto:thriller...@gmail.com mailto:thriller...@gmail.com mailto:thriller...@gmail.com mailto:thriller...@gmail.com wrote: Bogdan, Yes, I've attached 2 to my response - one was a crash from 10/29. The other occurred a few minutes ago. Thanks. On Fri, Oct 22, 2010 at 11:41 AM, Bogdan-Andrei Iancu bog...@voice-system.ro mailto:bog...@voice-system.ro mailto:bog...@voice-system.ro mailto:bog...@voice-system.ro mailto:bog...@voice-system.ro mailto:bog...@voice-system.ro mailto:bog...@voice-system.ro mailto:bog...@voice-system.ro wrote: Any chance with the backtraces ? Regards, Bogdan Anca Vamanu wrote: Hi, You need to inspect them with gdb,
Re: [OpenSIPS-Users] OpenSIPS core dumps
Bogdan, I am not seeing any other errors in the logs. Is there anything else I can look at? Both proxies are crashing everyday. Thanks. On Wed, Nov 3, 2010 at 4:21 AM, Bogdan-Andrei Iancu bog...@voice-system.rowrote: I got some clue about what is happening - as you are using flatstore for acc, the acc module does not check the success of the use_table DB operation - in 90% of the cases (for mysql, postgres, etc) this function cannot fail, but it seams that for flatstore can. And if it fails, the h-tail is set to NULL, leading to crash. Now, before the crash itself, do you see any other ERROR messages (even long before the crash) related to flatstore module ? Try: grep ERROR opensips_log_file | grep flatstore Regards, Bogdan thrillerbee wrote: Bogdan, One more detail that may help - I added another OpenSIPS proxy in parallel with this one ( load balancing between the two) and now both OpenSIPS proxies crash at the same time (within a couple seconds). I can provide more core dumps if it will help. Thanks. On Tue, Nov 2, 2010 at 9:02 AM, thrillerbee thriller...@gmail.com mailto:thriller...@gmail.com wrote: Bogdan, Below is the info requested: (gdb) frame 0 #0 0x7f51999f221f in flat_db_insert (h=0x7f0978, k=0x7f5198732a60, v=0x7f5198732d20, n=19) at flatstore.c:165 165 f = CON_FILE(h); (gdb) print h $1 = (const db_con_t *) 0x7f0978 (gdb) print h-tail $2 = 0 (gdb) print ((struct flat_con*)(h-tail))-file Cannot access memory at address 0x10 Before each of the crashes yesterday, I saw these in the logs: Nov 1 14:17:40 core1 kernel: [4287745.452111] opensips[22141]: segfault at 10 ip 7f51999f221f sp 7fffbcd8d510 error 4 in db_flatstore.so[7f51999ef000+5000] Nov 1 23:52:58 core1 kernel: [4348562.990735] opensips[26978]: segfault at 10 ip 7f726cb9b21f sp 7083f6f0 error 4 in db_flatstore.so[7f726cb98000+5000] Are there any compiler flags I should use for debugging? (gdb) info locals f = value optimized out i = value optimized out l = value optimized out p = value optimized out __FUNCTION__ = flat_db_insert Thanks again. On Tue, Nov 2, 2010 at 4:18 AM, Bogdan-Andrei Iancu bog...@voice-system.ro mailto:bog...@voice-system.ro wrote: Hi, in frame 0, could you print: h h-tail ((struct flat_con*)(h-tail))-file Also, before crash, do you see in the logs any errors from the db_flatstore module ? Regards, Bogdan thrillerbee wrote: Bogdan, It crashed again tonight. I've attached the backtrace. Thanks. On Mon, Nov 1, 2010 at 9:32 AM, thrillerbee thriller...@gmail.com mailto:thriller...@gmail.com mailto:thriller...@gmail.com mailto:thriller...@gmail.com wrote: Bogdan, Yes, I've attached 2 to my response - one was a crash from 10/29. The other occurred a few minutes ago. Thanks. On Fri, Oct 22, 2010 at 11:41 AM, Bogdan-Andrei Iancu bog...@voice-system.ro mailto:bog...@voice-system.ro mailto:bog...@voice-system.ro mailto:bog...@voice-system.ro wrote: Any chance with the backtraces ? Regards, Bogdan Anca Vamanu wrote: Hi, You need to inspect them with gdb, run: gdb path_to_opensips_executable path_to_corefile, and then run 'bt full' and send the output. Regards, -- Anca Vamanu www.voice-system.ro http://www.voice-system.ro http://www.voice-system.ro On 10/14/2010 10:12 PM, thrillerbee wrote: I have this info from dmesg: [1985853.285221] opensips[30865]: segfault at 10 ip 7f43899ce21f sp 7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000] [1985856.379671] opensips[30858]: segfault at 10 ip 7f43899ce21f sp 7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000] [1985896.961279] opensips[30868]: segfault at 10 ip 7f43899ce21f sp 7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000]
Re: [OpenSIPS-Users] OpenSIPS core dumps
I got some clue about what is happening - as you are using flatstore for acc, the acc module does not check the success of the use_table DB operation - in 90% of the cases (for mysql, postgres, etc) this function cannot fail, but it seams that for flatstore can. And if it fails, the h-tail is set to NULL, leading to crash. Now, before the crash itself, do you see any other ERROR messages (even long before the crash) related to flatstore module ? Try: grep ERROR opensips_log_file | grep flatstore Regards, Bogdan thrillerbee wrote: Bogdan, One more detail that may help - I added another OpenSIPS proxy in parallel with this one ( load balancing between the two) and now both OpenSIPS proxies crash at the same time (within a couple seconds). I can provide more core dumps if it will help. Thanks. On Tue, Nov 2, 2010 at 9:02 AM, thrillerbee thriller...@gmail.com mailto:thriller...@gmail.com wrote: Bogdan, Below is the info requested: (gdb) frame 0 #0 0x7f51999f221f in flat_db_insert (h=0x7f0978, k=0x7f5198732a60, v=0x7f5198732d20, n=19) at flatstore.c:165 165 f = CON_FILE(h); (gdb) print h $1 = (const db_con_t *) 0x7f0978 (gdb) print h-tail $2 = 0 (gdb) print ((struct flat_con*)(h-tail))-file Cannot access memory at address 0x10 Before each of the crashes yesterday, I saw these in the logs: Nov 1 14:17:40 core1 kernel: [4287745.452111] opensips[22141]: segfault at 10 ip 7f51999f221f sp 7fffbcd8d510 error 4 in db_flatstore.so[7f51999ef000+5000] Nov 1 23:52:58 core1 kernel: [4348562.990735] opensips[26978]: segfault at 10 ip 7f726cb9b21f sp 7083f6f0 error 4 in db_flatstore.so[7f726cb98000+5000] Are there any compiler flags I should use for debugging? (gdb) info locals f = value optimized out i = value optimized out l = value optimized out p = value optimized out __FUNCTION__ = flat_db_insert Thanks again. On Tue, Nov 2, 2010 at 4:18 AM, Bogdan-Andrei Iancu bog...@voice-system.ro mailto:bog...@voice-system.ro wrote: Hi, in frame 0, could you print: h h-tail ((struct flat_con*)(h-tail))-file Also, before crash, do you see in the logs any errors from the db_flatstore module ? Regards, Bogdan thrillerbee wrote: Bogdan, It crashed again tonight. I've attached the backtrace. Thanks. On Mon, Nov 1, 2010 at 9:32 AM, thrillerbee thriller...@gmail.com mailto:thriller...@gmail.com mailto:thriller...@gmail.com mailto:thriller...@gmail.com wrote: Bogdan, Yes, I've attached 2 to my response - one was a crash from 10/29. The other occurred a few minutes ago. Thanks. On Fri, Oct 22, 2010 at 11:41 AM, Bogdan-Andrei Iancu bog...@voice-system.ro mailto:bog...@voice-system.ro mailto:bog...@voice-system.ro mailto:bog...@voice-system.ro wrote: Any chance with the backtraces ? Regards, Bogdan Anca Vamanu wrote: Hi, You need to inspect them with gdb, run: gdb path_to_opensips_executable path_to_corefile, and then run 'bt full' and send the output. Regards, -- Anca Vamanu www.voice-system.ro http://www.voice-system.ro http://www.voice-system.ro On 10/14/2010 10:12 PM, thrillerbee wrote: I have this info from dmesg: [1985853.285221] opensips[30865]: segfault at 10 ip 7f43899ce21f sp 7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000] [1985856.379671] opensips[30858]: segfault at 10 ip 7f43899ce21f sp 7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000] [1985896.961279] opensips[30868]: segfault at 10 ip 7f43899ce21f sp 7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000] [2000131.245512] opensips[17672]: segfault at 10 ip 7fd0f21fb21f sp 7fff3a3b4f00 error 4 in db_flatstore.so[7fd0f21f8000+5000] [2000161.735962] opensips[17668]: segfault at 10 ip 7fd0f21fb21f sp 7fff3a3b4e40 error 4 in db_flatstore.so[7fd0f21f8000+5000]
Re: [OpenSIPS-Users] OpenSIPS core dumps
Hi, in frame 0, could you print: h h-tail ((struct flat_con*)(h-tail))-file Also, before crash, do you see in the logs any errors from the db_flatstore module ? Regards, Bogdan thrillerbee wrote: Bogdan, It crashed again tonight. I've attached the backtrace. Thanks. On Mon, Nov 1, 2010 at 9:32 AM, thrillerbee thriller...@gmail.com mailto:thriller...@gmail.com wrote: Bogdan, Yes, I've attached 2 to my response - one was a crash from 10/29. The other occurred a few minutes ago. Thanks. On Fri, Oct 22, 2010 at 11:41 AM, Bogdan-Andrei Iancu bog...@voice-system.ro mailto:bog...@voice-system.ro wrote: Any chance with the backtraces ? Regards, Bogdan Anca Vamanu wrote: Hi, You need to inspect them with gdb, run: gdb path_to_opensips_executable path_to_corefile, and then run 'bt full' and send the output. Regards, -- Anca Vamanu www.voice-system.ro http://www.voice-system.ro On 10/14/2010 10:12 PM, thrillerbee wrote: I have this info from dmesg: [1985853.285221] opensips[30865]: segfault at 10 ip 7f43899ce21f sp 7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000] [1985856.379671] opensips[30858]: segfault at 10 ip 7f43899ce21f sp 7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000] [1985896.961279] opensips[30868]: segfault at 10 ip 7f43899ce21f sp 7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000] [2000131.245512] opensips[17672]: segfault at 10 ip 7fd0f21fb21f sp 7fff3a3b4f00 error 4 in db_flatstore.so[7fd0f21f8000+5000] [2000161.735962] opensips[17668]: segfault at 10 ip 7fd0f21fb21f sp 7fff3a3b4e40 error 4 in db_flatstore.so[7fd0f21f8000+5000] [2000167.299402] opensips[17670]: segfault at 10 ip 7fd0f21fb21f sp 7fff3a3b4f00 error 4 in db_flatstore.so[7fd0f21f8000+5000] On Thu, Oct 14, 2010 at 1:57 PM, thrillerbee thriller...@gmail.com mailto:thriller...@gmail.com mailto:thriller...@gmail.com mailto:thriller...@gmail.com wrote: When OpenSIPS crashes, three corefiles are generated that are 2.1GB in size. How do I use these files to understand what's causing the crash? Thanks. ___ Users mailing list Users@lists.opensips.org mailto:Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users -- Bogdan-Andrei Iancu OpenSIPS Bootcamp 15 - 19 November 2010, Edison, New Jersey, USA www.voice-system.ro http://www.voice-system.ro ___ Users mailing list Users@lists.opensips.org mailto:Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users -- Bogdan-Andrei Iancu OpenSIPS Bootcamp 15 - 19 November 2010, Edison, New Jersey, USA www.voice-system.ro ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Re: [OpenSIPS-Users] OpenSIPS core dumps
Bogdan, One more detail that may help - I added another OpenSIPS proxy in parallel with this one ( load balancing between the two) and now both OpenSIPS proxies crash at the same time (within a couple seconds). I can provide more core dumps if it will help. Thanks. On Tue, Nov 2, 2010 at 9:02 AM, thrillerbee thriller...@gmail.com wrote: Bogdan, Below is the info requested: (gdb) frame 0 #0 0x7f51999f221f in flat_db_insert (h=0x7f0978, k=0x7f5198732a60, v=0x7f5198732d20, n=19) at flatstore.c:165 165 f = CON_FILE(h); (gdb) print h $1 = (const db_con_t *) 0x7f0978 (gdb) print h-tail $2 = 0 (gdb) print ((struct flat_con*)(h-tail))-file Cannot access memory at address 0x10 Before each of the crashes yesterday, I saw these in the logs: Nov 1 14:17:40 core1 kernel: [4287745.452111] opensips[22141]: segfault at 10 ip 7f51999f221f sp 7fffbcd8d510 error 4 in db_flatstore.so[7f51999ef000+5000] Nov 1 23:52:58 core1 kernel: [4348562.990735] opensips[26978]: segfault at 10 ip 7f726cb9b21f sp 7083f6f0 error 4 in db_flatstore.so[7f726cb98000+5000] Are there any compiler flags I should use for debugging? (gdb) info locals f = value optimized out i = value optimized out l = value optimized out p = value optimized out __FUNCTION__ = flat_db_insert Thanks again. On Tue, Nov 2, 2010 at 4:18 AM, Bogdan-Andrei Iancu bog...@voice-system.ro wrote: Hi, in frame 0, could you print: h h-tail ((struct flat_con*)(h-tail))-file Also, before crash, do you see in the logs any errors from the db_flatstore module ? Regards, Bogdan thrillerbee wrote: Bogdan, It crashed again tonight. I've attached the backtrace. Thanks. On Mon, Nov 1, 2010 at 9:32 AM, thrillerbee thriller...@gmail.com mailto:thriller...@gmail.com wrote: Bogdan, Yes, I've attached 2 to my response - one was a crash from 10/29. The other occurred a few minutes ago. Thanks. On Fri, Oct 22, 2010 at 11:41 AM, Bogdan-Andrei Iancu bog...@voice-system.ro mailto:bog...@voice-system.ro wrote: Any chance with the backtraces ? Regards, Bogdan Anca Vamanu wrote: Hi, You need to inspect them with gdb, run: gdb path_to_opensips_executable path_to_corefile, and then run 'bt full' and send the output. Regards, -- Anca Vamanu www.voice-system.ro http://www.voice-system.ro On 10/14/2010 10:12 PM, thrillerbee wrote: I have this info from dmesg: [1985853.285221] opensips[30865]: segfault at 10 ip 7f43899ce21f sp 7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000] [1985856.379671] opensips[30858]: segfault at 10 ip 7f43899ce21f sp 7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000] [1985896.961279] opensips[30868]: segfault at 10 ip 7f43899ce21f sp 7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000] [2000131.245512] opensips[17672]: segfault at 10 ip 7fd0f21fb21f sp 7fff3a3b4f00 error 4 in db_flatstore.so[7fd0f21f8000+5000] [2000161.735962] opensips[17668]: segfault at 10 ip 7fd0f21fb21f sp 7fff3a3b4e40 error 4 in db_flatstore.so[7fd0f21f8000+5000] [2000167.299402] opensips[17670]: segfault at 10 ip 7fd0f21fb21f sp 7fff3a3b4f00 error 4 in db_flatstore.so[7fd0f21f8000+5000] On Thu, Oct 14, 2010 at 1:57 PM, thrillerbee thriller...@gmail.com mailto:thriller...@gmail.com mailto:thriller...@gmail.com mailto:thriller...@gmail.com wrote: When OpenSIPS crashes, three corefiles are generated that are 2.1GB in size. How do I use these files to understand what's causing the crash? Thanks. ___ Users mailing list Users@lists.opensips.org mailto:Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users -- Bogdan-Andrei Iancu OpenSIPS Bootcamp 15 - 19 November 2010, Edison, New Jersey, USA www.voice-system.ro http://www.voice-system.ro ___ Users mailing list Users@lists.opensips.org mailto:Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Re: [OpenSIPS-Users] OpenSIPS core dumps
Bogdan,
Yes, I've attached 2 to my response - one was a crash from 10/29. The
other occurred a few minutes ago.
Thanks.
On Fri, Oct 22, 2010 at 11:41 AM, Bogdan-Andrei Iancu
bog...@voice-system.ro wrote:
Any chance with the backtraces ?
Regards,
Bogdan
Anca Vamanu wrote:
Hi,
You need to inspect them with gdb, run: gdb
path_to_opensips_executable path_to_corefile, and then run 'bt full'
and send the output.
Regards,
--
Anca Vamanu
www.voice-system.ro
On 10/14/2010 10:12 PM, thrillerbee wrote:
I have this info from dmesg:
[1985853.285221] opensips[30865]: segfault at 10 ip 7f43899ce21f sp
7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000]
[1985856.379671] opensips[30858]: segfault at 10 ip 7f43899ce21f sp
7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000]
[1985896.961279] opensips[30868]: segfault at 10 ip 7f43899ce21f sp
7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000]
[2000131.245512] opensips[17672]: segfault at 10 ip 7fd0f21fb21f sp
7fff3a3b4f00 error 4 in db_flatstore.so[7fd0f21f8000+5000]
[2000161.735962] opensips[17668]: segfault at 10 ip 7fd0f21fb21f sp
7fff3a3b4e40 error 4 in db_flatstore.so[7fd0f21f8000+5000]
[2000167.299402] opensips[17670]: segfault at 10 ip 7fd0f21fb21f sp
7fff3a3b4f00 error 4 in db_flatstore.so[7fd0f21f8000+5000]
On Thu, Oct 14, 2010 at 1:57 PM, thrillerbee thriller...@gmail.com
mailto:thriller...@gmail.com wrote:
When OpenSIPS crashes, three corefiles are generated that are
2.1GB in size. How do I use these files to understand what's
causing the crash?
Thanks.
___
Users mailing list
Users@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
--
Bogdan-Andrei Iancu
OpenSIPS Bootcamp
15 - 19 November 2010, Edison, New Jersey, USA
www.voice-system.ro
___
Users mailing list
Users@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Core was generated by `/usr/local/sbin/opensips -P
/var/run/opensips/opensips.pid -m 512 -u root -g ro'.
Program terminated with signal 11, Segmentation fault.
[New process 22242]
#0 0x7f66e354821f in flat_db_insert (h=0x7ea710, k=0x7f66e2288a60,
v=0x7f66e2288d20, n=19) at flatstore.c:165
165 f = CON_FILE(h);
(gdb) bt full
#0 0x7f66e354821f in flat_db_insert (h=0x7ea710, k=0x7f66e2288a60,
v=0x7f66e2288d20, n=19) at flatstore.c:165
f = value optimized out
i = value optimized out
l = value optimized out
p = value optimized out
__FUNCTION__ = flat_db_insert
#1 0x7f66e206f7ef in acc_db_request (rq=0x7f66c34266b8, rpl=value
optimized out) at acc.c:364
m = 19
n = value optimized out
i = value optimized out
my_ps = (db_ps_t) 0x0
__FUNCTION__ = acc_db_request
#2 0x7f66e207469e in tmcb_func (t=value optimized out, type=value
optimized out, ps=value optimized out) at acc_logic.c:386
No locals.
#3 0x7f66e2efd4a2 in run_trans_callbacks (type=256, trans=0x7f66c314eba0,
req=0x7f66c34266b8, rpl=0x7eaac8, code=200) at t_hooks.c:208
cbp = (struct tm_callback *) 0x7f66c1892ac8
backup = (struct usr_avp **) 0x771f48
trans_backup = (struct cell *) 0x7f66c314eba0
__FUNCTION__ = run_trans_callbacks
#4 0x7f66e2efd5c7 in run_trans_callbacks_locked (type=256,
trans=0x7f66c314eba0, req=0x7f66c34266b8, rpl=0x7eaac8, code=200) at
t_hooks.c:254
No locals.
#5 0x7f66e2f1aa38 in relay_reply (t=0x7f66c314eba0, p_msg=value optimized
out, branch=1, msg_status=200, cancel_bitmap=0x7fff3aa31ca8) at t_reply.c:1257
relay = 1
save_clone = 0
buf = 0x11846d0 SIP/2.0 200 OK\r\nRecord-Route:
sip:24.121.80.38;lr;ftag=gK075a5ad6;did=d9.c581cf61\r\nRecord-Route:
sip:24.121.80.36;lr;ftag=gK075a5ad6\r\nVia: SIP/2.0/UDP
24.121.80.36;branch=z9hG4bK13cd.038ec486.0\r\nV...
res_len = 685
relayed_code = 200
relayed_msg = (struct sip_msg *) 0x7eaac8
bm = {to_tag_val = {s = 0x1 Address 0x1 out of bounds, len = 8301256}}
totag_retr = value optimized out
reply_status = RPS_COMPLETED
cb_s = {
s = 0x11846d0 SIP/2.0 200 OK\r\nRecord-Route:
sip:24.121.80.38;lr;ftag=gK075a5ad6;did=d9.c581cf61\r\nRecord-Route:
sip:24.121.80.36;lr;ftag=gK075a5ad6\r\nVia: SIP/2.0/UDP
24.121.80.36;branch=z9hG4bK13cd.038ec486.0\r\nV..., len = 685}
text = {s = 0x16c Address 0x16c out of bounds, len = 746}
__FUNCTION__ = relay_reply
#6 0x7f66e2f1b548 in reply_received (p_msg=0x7eaac8) at t_reply.c:1502
last_uac_status = value optimized out
branch = 1
reply_status = value optimized out
timer = value optimized out
cancel_bitmap = 0
t = (struct cell *)
Re: [OpenSIPS-Users] OpenSIPS core dumps
Bogdan,
It crashed again tonight. I've attached the backtrace.
Thanks.
On Mon, Nov 1, 2010 at 9:32 AM, thrillerbee thriller...@gmail.com wrote:
Bogdan,
Yes, I've attached 2 to my response - one was a crash from 10/29. The
other occurred a few minutes ago.
Thanks.
On Fri, Oct 22, 2010 at 11:41 AM, Bogdan-Andrei Iancu
bog...@voice-system.ro wrote:
Any chance with the backtraces ?
Regards,
Bogdan
Anca Vamanu wrote:
Hi,
You need to inspect them with gdb, run: gdb
path_to_opensips_executable path_to_corefile, and then run 'bt full'
and send the output.
Regards,
--
Anca Vamanu
www.voice-system.ro
On 10/14/2010 10:12 PM, thrillerbee wrote:
I have this info from dmesg:
[1985853.285221] opensips[30865]: segfault at 10 ip 7f43899ce21f sp
7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000]
[1985856.379671] opensips[30858]: segfault at 10 ip 7f43899ce21f sp
7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000]
[1985896.961279] opensips[30868]: segfault at 10 ip 7f43899ce21f sp
7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000]
[2000131.245512] opensips[17672]: segfault at 10 ip 7fd0f21fb21f sp
7fff3a3b4f00 error 4 in db_flatstore.so[7fd0f21f8000+5000]
[2000161.735962] opensips[17668]: segfault at 10 ip 7fd0f21fb21f sp
7fff3a3b4e40 error 4 in db_flatstore.so[7fd0f21f8000+5000]
[2000167.299402] opensips[17670]: segfault at 10 ip 7fd0f21fb21f sp
7fff3a3b4f00 error 4 in db_flatstore.so[7fd0f21f8000+5000]
On Thu, Oct 14, 2010 at 1:57 PM, thrillerbee thriller...@gmail.com
mailto:thriller...@gmail.com wrote:
When OpenSIPS crashes, three corefiles are generated that are
2.1GB in size. How do I use these files to understand what's
causing the crash?
Thanks.
___
Users mailing list
Users@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
--
Bogdan-Andrei Iancu
OpenSIPS Bootcamp
15 - 19 November 2010, Edison, New Jersey, USA
www.voice-system.ro
___
Users mailing list
Users@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Core was generated by `/usr/local/sbin/opensips -P
/var/run/opensips/opensips.pid -m 512 -u root -g ro'.
Program terminated with signal 11, Segmentation fault.
[New process 26978]
#0 0x7f726cb9b21f in flat_db_insert (h=0x7f09a8, k=0x7f726b8dba60,
v=0x7f726b8dbd20, n=19) at flatstore.c:165
165 f = CON_FILE(h);
(gdb) bt full
#0 0x7f726cb9b21f in flat_db_insert (h=0x7f09a8, k=0x7f726b8dba60,
v=0x7f726b8dbd20, n=19) at flatstore.c:165
f = value optimized out
i = value optimized out
l = value optimized out
p = value optimized out
__FUNCTION__ = flat_db_insert
#1 0x7f726b6c27ef in acc_db_request (rq=0x7f726c77f160, rpl=value
optimized out) at acc.c:364
m = 19
n = value optimized out
i = value optimized out
my_ps = (db_ps_t) 0x0
__FUNCTION__ = acc_db_request
#2 0x7f726b6c7e28 in tmcb_func (t=0x7f724cea4fd8, type=value optimized
out, ps=0x4ccf5229) at acc_logic.c:319
No locals.
#3 0x7f726c5504a2 in run_trans_callbacks (type=64, trans=0x7f724cea4fd8,
req=0x7f726c77f160, rpl=0x7f0d60, code=404) at t_hooks.c:208
cbp = (struct tm_callback *) 0x7f724ab5c220
backup = (struct usr_avp **) 0x7f724cea6808
trans_backup = (struct cell *) 0x7f724cea4fd8
__FUNCTION__ = run_trans_callbacks
#4 0x7f726c56c9b4 in t_should_relay_response (Trans=0x7f724cea4fd8,
new_code=value optimized out, branch=1, should_store=0x7083fa18,
should_relay=0x7083fa1c, cancel_bitmap=0x7083fae8, reply=0x7f0d60)
at t_reply.c:611
branch_cnt = 2
inv_through = value optimized out
do_cancel = value optimized out
__FUNCTION__ = t_should_relay_response
#5 0x7f726c56cfe2 in relay_reply (t=0x7f09a8, p_msg=0x7f0d60,
branch=1288655401, msg_status=404, cancel_bitmap=0x7083fae8)
at t_reply.c:1133
relay = value optimized out
save_clone = value optimized out
buf = value optimized out
res_len = 0
relayed_code = 404
relayed_msg = value optimized out
bm = {to_tag_val = {s = 0x1 Address 0x1 out of bounds, len = 8326496}}
totag_retr = value optimized out
reply_status = value optimized out
cb_s = {
s = 0x76202d From: \Office\
sip:14064263...@8.20.212.252;tag=Uvyr85XQ8FNvj\r\nTo:
sip:30201#16606592...@24.121.80.36;tag=B2B.2064.650\r\nCall-ID:
91b97e04-f697-4827-bc02-d76b56b579b5\r\nCSeq: 3991724 INVITE\r\nContact...,
len = 8187368}
text = {
s = 0x76206e To:
sip:30201#16606592...@24.121.80.36;tag=B2B.2064.650\r\nCall-ID:
91b97e04-f697-4827-bc02-d76b56b579b5\r\nCSeq: 3991724
Re: [OpenSIPS-Users] OpenSIPS core dumps
Any chance with the backtraces ? Regards, Bogdan Anca Vamanu wrote: Hi, You need to inspect them with gdb, run: gdb path_to_opensips_executable path_to_corefile, and then run 'bt full' and send the output. Regards, -- Anca Vamanu www.voice-system.ro On 10/14/2010 10:12 PM, thrillerbee wrote: I have this info from dmesg: [1985853.285221] opensips[30865]: segfault at 10 ip 7f43899ce21f sp 7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000] [1985856.379671] opensips[30858]: segfault at 10 ip 7f43899ce21f sp 7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000] [1985896.961279] opensips[30868]: segfault at 10 ip 7f43899ce21f sp 7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000] [2000131.245512] opensips[17672]: segfault at 10 ip 7fd0f21fb21f sp 7fff3a3b4f00 error 4 in db_flatstore.so[7fd0f21f8000+5000] [2000161.735962] opensips[17668]: segfault at 10 ip 7fd0f21fb21f sp 7fff3a3b4e40 error 4 in db_flatstore.so[7fd0f21f8000+5000] [2000167.299402] opensips[17670]: segfault at 10 ip 7fd0f21fb21f sp 7fff3a3b4f00 error 4 in db_flatstore.so[7fd0f21f8000+5000] On Thu, Oct 14, 2010 at 1:57 PM, thrillerbee thriller...@gmail.com mailto:thriller...@gmail.com wrote: When OpenSIPS crashes, three corefiles are generated that are 2.1GB in size. How do I use these files to understand what's causing the crash? Thanks. ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users -- Bogdan-Andrei Iancu OpenSIPS Bootcamp 15 - 19 November 2010, Edison, New Jersey, USA www.voice-system.ro ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users
