Hi Tolga, Thanks for the info.
What exact OpenSIPs version/revision are you using ? I need to correlate logs with sources. Regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developer http://www.opensips-solutions.com On 05/22/2013 11:02 PM, Tolga Tarhan wrote: > Sorry for the self-reply -- here's the (same) stacktraces with line > numbers and params: > > #0 0x0000003564c328a5 in raise () from /lib64/libc.so.6 > #1 0x0000003564c34085 in abort () from /lib64/libc.so.6 > #2 0x000000000049d370 in qm_free (qm=<value optimized out>, p=<value > optimized out>, file=0x7ffccb25df64 "logic.c", func=<value optimized > out>, line=755) at mem/q_malloc.c:450 > #3 0x00007ffccb253804 in process_bridge_200OK (msg=0x7ffcccdd2600, > extra_headers=0xd5, body=<value optimized out>, tuple=0x7ffcc9518de0, > entity=<value optimized out>) at logic.c:755 > #4 0x00007ffccb254ba2 in b2b_logic_notify_reply (src=<value optimized > out>, msg=0x7ffcccdd2600, key=<value optimized out>, > body=0x7fffc2ce15d0, extra_headers=0x7fffc2ce15c0, > b2bl_key=0x7fffc2ce23f0, hash_index=649, local_index=1) > at logic.c:1133 > #5 0x00007ffccb2565e1 in b2b_logic_notify (src=1, msg=0x7ffcccdd2600, > key=0x7ffcc9525e40, type=1, param=0x7fffc2ce23f0) at logic.c:2040 > #6 0x00007ffccb47a7a7 in b2b_tm_cback (t=0x7ffcc951c110, > htable=0x7ffcc94f38d0, ps=<value optimized out>) at dlg.c:2678 > #7 0x00007ffccc744b71 in run_trans_callbacks (type=256, > trans=0x7ffcc951c110, req=<value optimized out>, rpl=<value optimized > out>, code=<value optimized out>) at t_hooks.c:212 > #8 0x00007ffccc74fa12 in local_reply (t=0x7ffcc951c110, p_msg=<value > optimized out>, branch=<value optimized out>, msg_status=<value > optimized out>, cancel_bitmap=<value optimized out>) at t_reply.c:1391 > #9 0x00007ffccc750cc5 in reply_received (p_msg=0x7ffcccdd2600) at > t_reply.c:1540 > #10 0x00000000004266da in forward_reply (msg=0x7ffcccdd2600) at > forward.c:574 > #11 0x0000000000452ad8 in receive_msg (buf=<value optimized out>, > len=<value optimized out>, rcv_info=0x7fffc2ce28c0) at receive.c:207 > #12 0x0000000000496c95 in udp_rcv_loop () at udp_server.c:424 > #13 0x000000000042d763 in main_loop (argc=<value optimized out>, > argv=<value optimized out>) at main.c:884 > #14 main (argc=<value optimized out>, argv=<value optimized out>) at > main.c:1557 > > #0 0x0000003564c328a5 in raise () from /lib64/libc.so.6 > #1 0x0000003564c34085 in abort () from /lib64/libc.so.6 > #2 0x000000000049d370 in qm_free (qm=<value optimized out>, p=<value > optimized out>, file=0x7ffccb262d75 "records.c", func=<value optimized > out>, line=595) at mem/q_malloc.c:450 > #3 0x00007ffccb25a003 in b2bl_delete (tuple=0x7ffcc9518de0, > hash_index=<value optimized out>, not_del_b2be=1) at records.c:595 > #4 0x00007ffccb25a3d5 in destroy_b2bl_htable () at records.c:705 > #5 0x000000000046dbf2 in destroy_modules () at sr_module.c:371 > #6 0x00000000004298a1 in cleanup (show_status=1) at main.c:348 > #7 0x000000000042a360 in handle_sigs () at main.c:549 > #8 0x000000000042db66 in main_loop (argc=<value optimized out>, > argv=<value optimized out>) at main.c:1024 > #9 main (argc=<value optimized out>, argv=<value optimized out>) at > main.c:1557 > > Thanks, > Tolga > > > On Wed, May 22, 2013 at 1:00 PM, Tolga Tarhan <[email protected] > <mailto:[email protected]>> wrote: > > Thank you -- I've recompiled and enabled the memory debug. I have > the log file from the whole experience available here: > > http://netbrains-misc.s3.amazonaws.com/opensips/opensips.log > > (note - real phone numbers and domain names in the log have been > replaced with placeholders) > > The key item seems to be: > > CRITICAL:core:qm_free: freeing already freed pointer, first free: > logic.c: process_bridge_200OK(740) - aborting > > Although this appears to be after we've already decided we're > going to crash, as I see "INFO:core:cleanup: cleanup" and > "NFO:core:handle_sigs: child process 24788 exited by a signal 6" > above this part of the log. > > Also worth noting is the existance of > "CRITICAL:b2b_logic:b2bl_drop_entity: we should never end up here" > throughout the log. > > Also, here's the stack trace at crash time. Note that there were > two core files generated for the same crash, so this is the > backtrace from each: > > #0 0x0000003564c328a5 in raise () from /lib64/libc.so.6 > #1 0x0000003564c34085 in abort () from /lib64/libc.so.6 > #2 0x000000000049d370 in qm_free () > #3 0x00007ffccb253804 in process_bridge_200OK () from > /usr/lib64/opensips/modules/b2b_logic.so > #4 0x00007ffccb254ba2 in b2b_logic_notify_reply () from > /usr/lib64/opensips/modules/b2b_logic.so > #5 0x00007ffccb2565e1 in b2b_logic_notify () from > /usr/lib64/opensips/modules/b2b_logic.so > #6 0x00007ffccb47a7a7 in b2b_tm_cback () from > /usr/lib64/opensips/modules/b2b_entities.so > #7 0x00007ffccc744b71 in run_trans_callbacks () from > /usr/lib64/opensips/modules/tm.so > #8 0x00007ffccc74fa12 in local_reply () from > /usr/lib64/opensips/modules/tm.so > #9 0x00007ffccc750cc5 in reply_received () from > /usr/lib64/opensips/modules/tm.so > #10 0x00000000004266da in forward_reply () > #11 0x0000000000452ad8 in receive_msg () > #12 0x0000000000496c95 in udp_rcv_loop () > #13 0x000000000042d763 in main () > > #0 0x0000003564c328a5 in raise () from /lib64/libc.so.6 > #1 0x0000003564c34085 in abort () from /lib64/libc.so.6 > #2 0x000000000049d370 in qm_free () > #3 0x00007ffccb25a003 in b2bl_delete () from > /usr/lib64/opensips/modules/b2b_logic.so > #4 0x00007ffccb25a3d5 in destroy_b2bl_htable () from > /usr/lib64/opensips/modules/b2b_logic.so > #5 0x000000000046dbf2 in destroy_modules () > #6 0x00000000004298a1 in cleanup () > #7 0x000000000042a360 in handle_sigs () > #8 0x000000000042db66 in main () > > I am unfamiliar with this codebase. Can anyone garner anything > useful from the logs? > > Thanks, > Tolga > > > > On Wed, May 22, 2013 at 9:02 AM, Bogdan-Andrei Iancu > <[email protected] <mailto:[email protected]>> wrote: > > Hello Tolga, > > The crash seems to be in the memory manager, most probably > because of memory corruption. To troubleshoot such issues you > need to compile-in the memory debugger - see > http://www.opensips.org/Documentation/TroubleShooting-OutOfMem . > > Using memlog=1 + memdump=10 you should get a lot of logs > related to mem ops, including a final report + abort() when > the corruption is detected. > > Regards, > > Bogdan-Andrei Iancu > OpenSIPS Founder and Developer > http://www.opensips-solutions.com > > > On 05/22/2013 01:29 AM, Tolga Tarhan wrote: >> Hello, >> >> While using the B2BUA module in OpenSIPS 1.9.0, >> we've encountered a consistent segfault. We are using a refer >> scenario just like the one in the B2BUA sample docs, and >> after several REFERs for the same call (to different >> destinations), OpenSIPS crashes with a segfault. The core >> file indicates the following backtrace: >> >> #0 0x000000000049a334 in fm_malloc () >> #1 0x00007fdaecd96230 in shm_malloc_unsafe (type=B2B_CLIENT, >> entity_id=0x7fdaee8ec750, to_uri=0x7fff2d346360, >> from_uri=0x7fff2d346320, from_dname=0x0, ssid=<value >> optimized out>, msg=0x0) at ../../mem/shm_mem.h:248 >> #2 shm_malloc (type=B2B_CLIENT, entity_id=0x7fdaee8ec750, >> to_uri=0x7fff2d346360, from_uri=0x7fff2d346320, >> from_dname=0x0, ssid=<value optimized out>, msg=0x0) at >> ../../mem/shm_mem.h:258 >> #3 b2bl_create_new_entity (type=B2B_CLIENT, >> entity_id=0x7fdaee8ec750, to_uri=0x7fff2d346360, >> from_uri=0x7fff2d346320, from_dname=0x0, ssid=<value >> optimized out>, msg=0x0) at logic.c:293 >> #4 0x00007fdaecd96882 in b2bl_new_client (to_uri=<value >> optimized out>, from_uri=<value optimized out>, tuple=<value >> optimized out>, ssid=0x7fdaeb026c00, msg=<value optimized >> out>) at logic.c:607 >> #5 0x00007fdaecda3579 in process_bridge_200OK >> (msg=0x7fdaee8e8b30, extra_headers=0x7fdaeb03d578, >> body=<value optimized out>, tuple=0x7fdaeb01ada8, >> entity=<value optimized out>) at logic.c:816 >> #6 0x00007fdaecda46c2 in b2b_logic_notify_reply (src=<value >> optimized out>, msg=0x7fdaee8e8b30, key=<value optimized >> out>, body=0x7fff2d3468b0, extra_headers=0x7fff2d3468a0, >> b2bl_key=0x7fff2d3476d0, hash_index=649, local_index=0) >> at logic.c:1133 >> #7 0x00007fdaecda6081 in b2b_logic_notify (src=1, >> msg=0x7fdaee8e8b30, key=0x7fdaeb03d500, type=1, >> param=0x7fff2d3476d0) at logic.c:2040 >> #8 0x00007fdaecfca7ad in b2b_tm_cback (t=0x7fdaeb054118, >> htable=0x7fdaeb014630, ps=<value optimized out>) at dlg.c:2678 >> #9 0x00007fdaee291441 in run_trans_callbacks (type=256, >> trans=0x7fdaeb054118, req=<value optimized out>, rpl=<value >> optimized out>, code=<value optimized out>) at t_hooks.c:212 >> #10 0x00007fdaee29c0e2 in local_reply (t=0x7fdaeb054118, >> p_msg=<value optimized out>, branch=<value optimized out>, >> msg_status=<value optimized out>, cancel_bitmap=<value >> optimized out>) at t_reply.c:1391 >> #11 0x00007fdaee29d31d in reply_received >> (p_msg=0x7fdaee8e8b30) at t_reply.c:1540 >> #12 0x000000000042625a in forward_reply () >> #13 0x0000000000451c28 in receive_msg () >> #14 0x0000000000494e45 in udp_rcv_loop () >> #15 0x000000000042d1a3 in main () >> >> I'm not really sure how to diagnose this one. Any >> hints/fixes/suggestions would be very appreciated. >> >> Thanks, >> Tolga >> >> >> _______________________________________________ >> Users mailing list >> [email protected] <mailto:[email protected]> >> http://lists.opensips.org/cgi-bin/mailman/listinfo/users > > >
_______________________________________________ Users mailing list [email protected] http://lists.opensips.org/cgi-bin/mailman/listinfo/users
