On a related note, however, is this of any concern: May 27 15:35:02 b2b-dev1 /usr/sbin/opensips[19455]: CRITICAL:b2b_logic:b2bl_drop_entity: we should never end up here
Or for that matter, any of these: May 27 15:35:22 b2b-dev1 /usr/sbin/opensips[19455]: WARNING:b2b_logic:b2bl_delete_entity: entity [0x7f8840c171d8]->[] not found for tuple [383.0] May 27 15:35:19 b2b-dev1 /usr/sbin/opensips[19458]: ERROR:b2b_logic:process_bridge_negreply: unexpected entity_no [2] for tuple [0x7f8840c19378] May 27 15:35:19 b2b-dev1 /usr/sbin/opensips[19458]: ERROR:b2b_logic:b2b_logic_notify_reply: Failed to process negative reply while in bridging state -- Tolga On Mon, May 27, 2013 at 3:32 PM, Tolga Tarhan <[email protected]> wrote: > Bogdan, > > We've tried out the patch and it seems to resolve the issue. We will > continue to test to see if we can reproduce the problem, but for now, it > appears fixed. > > Thanks so much for your help on this. > > -- > Tolga > > > On Mon, May 27, 2013 at 2:31 PM, Bogdan-Andrei Iancu > <[email protected]>wrote: > >> Hi Tolga, >> >> Yes the patch is correct, please give it a try. >> >> Thanks and regards, >> Bogdan >> >> >> Sent from Samsung Mobile >> >> >> >> -------- Original message -------- >> From: Tolga Tarhan <[email protected]> >> Date: >> To: Bogdan-Andrei Iancu <[email protected]> >> Cc: OpenSIPS users mailling list <[email protected]> >> Subject: Re: [OpenSIPS-Users] B2BUA Segfault >> >> >> Bogdan, >> >> The patch doesn't build: >> >> Compiling logic.c >> logic.c: In function 'process_bridge_200OK': >> logic.c:741: error: invalid operands to binary == (have 'char *' and >> 'str') >> logic.c:742: error: incompatible types when assigning to type 'str' from >> type 'int' >> make[1]: *** [logic.o] Error 1 >> >> I've attached what I believe is a corrected patch. Let me know if this is >> what you intended. >> >> I'm trying the (corrected) patch now to see if it resolves the original >> issue. I'll let you know on my findings shortly. >> >> Thanks, >> Tolga >> >> >> >> On Mon, May 27, 2013 at 6:27 AM, Bogdan-Andrei Iancu <[email protected] >> > wrote: >> >>> ** >>> Hello Tolga, >>> >>> Please test the attached patch. >>> >>> Regards, >>> >>> Bogdan-Andrei Iancu >>> OpenSIPS Founder and Developerhttp://www.opensips-solutions.com >>> >>> >>> On 05/24/2013 08:33 PM, Tolga Tarhan wrote: >>> >>> Sure, here you go: >>> >>> version: opensips 1.9.0-tls (x86_64/linux) >>> flags: STATS: Off, USE_IPV6, USE_TCP, USE_TLS, DISABLE_NAGLE, USE_MCAST, >>> SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC, FAST_LOCK-ADAPTIVE_WAIT >>> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, >>> MAX_URI_SIZE 1024, BUF_SIZE 65535 >>> poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. >>> svnrevision: unknown >>> @(#) $Id: main.c 9790 2013-02-15 10:14:34Z bogdan_iancu $ >>> main.c compiled on 12:39:26 May 22 2013 with gcc 4.4.7 >>> >>> It also has the couple patches that I've previously submitted (and >>> you've since merged) in the build (the three related to GRUU bugs). They >>> aren't a factor, as this particular instance is not acting as a registrar >>> at all. >>> >>> Thanks, >>> Tolga >>> >>> >>> >>> On Fri, May 24, 2013 at 9:33 AM, Bogdan-Andrei Iancu < >>> [email protected]> wrote: >>> >>>> Hi Tolga, >>>> >>>> Thanks for the info. >>>> >>>> What exact OpenSIPs version/revision are you using ? I need to >>>> correlate logs with sources. >>>> >>>> Regards, >>>> >>>> Bogdan-Andrei Iancu >>>> OpenSIPS Founder and Developerhttp://www.opensips-solutions.com >>>> >>>> >>>> On 05/22/2013 11:02 PM, Tolga Tarhan wrote: >>>> >>>> Sorry for the self-reply -- here's the (same) stacktraces with line >>>> numbers and params: >>>> >>>> #0 0x0000003564c328a5 in raise () from /lib64/libc.so.6 >>>> #1 0x0000003564c34085 in abort () from /lib64/libc.so.6 >>>> #2 0x000000000049d370 in qm_free (qm=<value optimized out>, p=<value >>>> optimized out>, file=0x7ffccb25df64 "logic.c", func=<value optimized out>, >>>> line=755) at mem/q_malloc.c:450 >>>> #3 0x00007ffccb253804 in process_bridge_200OK (msg=0x7ffcccdd2600, >>>> extra_headers=0xd5, body=<value optimized out>, tuple=0x7ffcc9518de0, >>>> entity=<value optimized out>) at logic.c:755 >>>> #4 0x00007ffccb254ba2 in b2b_logic_notify_reply (src=<value optimized >>>> out>, msg=0x7ffcccdd2600, key=<value optimized out>, body=0x7fffc2ce15d0, >>>> extra_headers=0x7fffc2ce15c0, b2bl_key=0x7fffc2ce23f0, hash_index=649, >>>> local_index=1) >>>> at logic.c:1133 >>>> #5 0x00007ffccb2565e1 in b2b_logic_notify (src=1, msg=0x7ffcccdd2600, >>>> key=0x7ffcc9525e40, type=1, param=0x7fffc2ce23f0) at logic.c:2040 >>>> #6 0x00007ffccb47a7a7 in b2b_tm_cback (t=0x7ffcc951c110, >>>> htable=0x7ffcc94f38d0, ps=<value optimized out>) at dlg.c:2678 >>>> #7 0x00007ffccc744b71 in run_trans_callbacks (type=256, >>>> trans=0x7ffcc951c110, req=<value optimized out>, rpl=<value optimized out>, >>>> code=<value optimized out>) at t_hooks.c:212 >>>> #8 0x00007ffccc74fa12 in local_reply (t=0x7ffcc951c110, p_msg=<value >>>> optimized out>, branch=<value optimized out>, msg_status=<value optimized >>>> out>, cancel_bitmap=<value optimized out>) at t_reply.c:1391 >>>> #9 0x00007ffccc750cc5 in reply_received (p_msg=0x7ffcccdd2600) at >>>> t_reply.c:1540 >>>> #10 0x00000000004266da in forward_reply (msg=0x7ffcccdd2600) at >>>> forward.c:574 >>>> #11 0x0000000000452ad8 in receive_msg (buf=<value optimized out>, >>>> len=<value optimized out>, rcv_info=0x7fffc2ce28c0) at receive.c:207 >>>> #12 0x0000000000496c95 in udp_rcv_loop () at udp_server.c:424 >>>> #13 0x000000000042d763 in main_loop (argc=<value optimized out>, >>>> argv=<value optimized out>) at main.c:884 >>>> #14 main (argc=<value optimized out>, argv=<value optimized out>) at >>>> main.c:1557 >>>> >>>> #0 0x0000003564c328a5 in raise () from /lib64/libc.so.6 >>>> #1 0x0000003564c34085 in abort () from /lib64/libc.so.6 >>>> #2 0x000000000049d370 in qm_free (qm=<value optimized out>, p=<value >>>> optimized out>, file=0x7ffccb262d75 "records.c", func=<value optimized >>>> out>, line=595) at mem/q_malloc.c:450 >>>> #3 0x00007ffccb25a003 in b2bl_delete (tuple=0x7ffcc9518de0, >>>> hash_index=<value optimized out>, not_del_b2be=1) at records.c:595 >>>> #4 0x00007ffccb25a3d5 in destroy_b2bl_htable () at records.c:705 >>>> #5 0x000000000046dbf2 in destroy_modules () at sr_module.c:371 >>>> #6 0x00000000004298a1 in cleanup (show_status=1) at main.c:348 >>>> #7 0x000000000042a360 in handle_sigs () at main.c:549 >>>> #8 0x000000000042db66 in main_loop (argc=<value optimized out>, >>>> argv=<value optimized out>) at main.c:1024 >>>> #9 main (argc=<value optimized out>, argv=<value optimized out>) at >>>> main.c:1557 >>>> >>>> Thanks, >>>> Tolga >>>> >>>> >>>> On Wed, May 22, 2013 at 1:00 PM, Tolga Tarhan <[email protected]>wrote: >>>> >>>>> Thank you -- I've recompiled and enabled the memory debug. I have the >>>>> log file from the whole experience available here: >>>>> >>>>> http://netbrains-misc.s3.amazonaws.com/opensips/opensips.log >>>>> >>>>> (note - real phone numbers and domain names in the log have been >>>>> replaced with placeholders) >>>>> >>>>> The key item seems to be: >>>>> >>>>> CRITICAL:core:qm_free: freeing already freed pointer, first free: >>>>> logic.c: process_bridge_200OK(740) - aborting >>>>> >>>>> Although this appears to be after we've already decided we're going >>>>> to crash, as I see "INFO:core:cleanup: cleanup" and "NFO:core:handle_sigs: >>>>> child process 24788 exited by a signal 6" above this part of the log. >>>>> >>>>> Also worth noting is the existance of >>>>> "CRITICAL:b2b_logic:b2bl_drop_entity: we should never end up here" >>>>> throughout the log. >>>>> >>>>> Also, here's the stack trace at crash time. Note that there were two >>>>> core files generated for the same crash, so this is the backtrace from >>>>> each: >>>>> >>>>> #0 0x0000003564c328a5 in raise () from /lib64/libc.so.6 >>>>> #1 0x0000003564c34085 in abort () from /lib64/libc.so.6 >>>>> #2 0x000000000049d370 in qm_free () >>>>> #3 0x00007ffccb253804 in process_bridge_200OK () from >>>>> /usr/lib64/opensips/modules/b2b_logic.so >>>>> #4 0x00007ffccb254ba2 in b2b_logic_notify_reply () from >>>>> /usr/lib64/opensips/modules/b2b_logic.so >>>>> #5 0x00007ffccb2565e1 in b2b_logic_notify () from >>>>> /usr/lib64/opensips/modules/b2b_logic.so >>>>> #6 0x00007ffccb47a7a7 in b2b_tm_cback () from >>>>> /usr/lib64/opensips/modules/b2b_entities.so >>>>> #7 0x00007ffccc744b71 in run_trans_callbacks () from >>>>> /usr/lib64/opensips/modules/tm.so >>>>> #8 0x00007ffccc74fa12 in local_reply () from >>>>> /usr/lib64/opensips/modules/tm.so >>>>> #9 0x00007ffccc750cc5 in reply_received () from >>>>> /usr/lib64/opensips/modules/tm.so >>>>> #10 0x00000000004266da in forward_reply () >>>>> #11 0x0000000000452ad8 in receive_msg () >>>>> #12 0x0000000000496c95 in udp_rcv_loop () >>>>> #13 0x000000000042d763 in main () >>>>> >>>>> #0 0x0000003564c328a5 in raise () from /lib64/libc.so.6 >>>>> #1 0x0000003564c34085 in abort () from /lib64/libc.so.6 >>>>> #2 0x000000000049d370 in qm_free () >>>>> #3 0x00007ffccb25a003 in b2bl_delete () from >>>>> /usr/lib64/opensips/modules/b2b_logic.so >>>>> #4 0x00007ffccb25a3d5 in destroy_b2bl_htable () from >>>>> /usr/lib64/opensips/modules/b2b_logic.so >>>>> #5 0x000000000046dbf2 in destroy_modules () >>>>> #6 0x00000000004298a1 in cleanup () >>>>> #7 0x000000000042a360 in handle_sigs () >>>>> #8 0x000000000042db66 in main () >>>>> >>>>> I am unfamiliar with this codebase. Can anyone garner anything >>>>> useful from the logs? >>>>> >>>>> Thanks, >>>>> Tolga >>>>> >>>>> >>>>> >>>>> On Wed, May 22, 2013 at 9:02 AM, Bogdan-Andrei Iancu < >>>>> [email protected]> wrote: >>>>> >>>>>> Hello Tolga, >>>>>> >>>>>> The crash seems to be in the memory manager, most probably because of >>>>>> memory corruption. To troubleshoot such issues you need to compile-in the >>>>>> memory debugger - see >>>>>> http://www.opensips.org/Documentation/TroubleShooting-OutOfMem . >>>>>> >>>>>> Using memlog=1 + memdump=10 you should get a lot of logs related to >>>>>> mem ops, including a final report + abort() when the corruption is >>>>>> detected. >>>>>> >>>>>> Regards, >>>>>> >>>>>> Bogdan-Andrei Iancu >>>>>> OpenSIPS Founder and Developerhttp://www.opensips-solutions.com >>>>>> >>>>>> >>>>>> On 05/22/2013 01:29 AM, Tolga Tarhan wrote: >>>>>> >>>>>> Hello, >>>>>> >>>>>> While using the B2BUA module in OpenSIPS 1.9.0, we've encountered a >>>>>> consistent segfault. We are using a refer scenario just like the one in >>>>>> the >>>>>> B2BUA sample docs, and after several REFERs for the same call (to >>>>>> different >>>>>> destinations), OpenSIPS crashes with a segfault. The core file indicates >>>>>> the following backtrace: >>>>>> >>>>>> #0 0x000000000049a334 in fm_malloc () >>>>>> #1 0x00007fdaecd96230 in shm_malloc_unsafe (type=B2B_CLIENT, >>>>>> entity_id=0x7fdaee8ec750, to_uri=0x7fff2d346360, from_uri=0x7fff2d346320, >>>>>> from_dname=0x0, ssid=<value optimized out>, msg=0x0) at >>>>>> ../../mem/shm_mem.h:248 >>>>>> #2 shm_malloc (type=B2B_CLIENT, entity_id=0x7fdaee8ec750, >>>>>> to_uri=0x7fff2d346360, from_uri=0x7fff2d346320, from_dname=0x0, >>>>>> ssid=<value >>>>>> optimized out>, msg=0x0) at ../../mem/shm_mem.h:258 >>>>>> #3 b2bl_create_new_entity (type=B2B_CLIENT, >>>>>> entity_id=0x7fdaee8ec750, to_uri=0x7fff2d346360, from_uri=0x7fff2d346320, >>>>>> from_dname=0x0, ssid=<value optimized out>, msg=0x0) at logic.c:293 >>>>>> #4 0x00007fdaecd96882 in b2bl_new_client (to_uri=<value optimized >>>>>> out>, from_uri=<value optimized out>, tuple=<value optimized out>, >>>>>> ssid=0x7fdaeb026c00, msg=<value optimized out>) at logic.c:607 >>>>>> #5 0x00007fdaecda3579 in process_bridge_200OK (msg=0x7fdaee8e8b30, >>>>>> extra_headers=0x7fdaeb03d578, body=<value optimized out>, >>>>>> tuple=0x7fdaeb01ada8, entity=<value optimized out>) at logic.c:816 >>>>>> #6 0x00007fdaecda46c2 in b2b_logic_notify_reply (src=<value >>>>>> optimized out>, msg=0x7fdaee8e8b30, key=<value optimized out>, >>>>>> body=0x7fff2d3468b0, extra_headers=0x7fff2d3468a0, >>>>>> b2bl_key=0x7fff2d3476d0, >>>>>> hash_index=649, local_index=0) >>>>>> at logic.c:1133 >>>>>> #7 0x00007fdaecda6081 in b2b_logic_notify (src=1, >>>>>> msg=0x7fdaee8e8b30, key=0x7fdaeb03d500, type=1, param=0x7fff2d3476d0) at >>>>>> logic.c:2040 >>>>>> #8 0x00007fdaecfca7ad in b2b_tm_cback (t=0x7fdaeb054118, >>>>>> htable=0x7fdaeb014630, ps=<value optimized out>) at dlg.c:2678 >>>>>> #9 0x00007fdaee291441 in run_trans_callbacks (type=256, >>>>>> trans=0x7fdaeb054118, req=<value optimized out>, rpl=<value optimized >>>>>> out>, >>>>>> code=<value optimized out>) at t_hooks.c:212 >>>>>> #10 0x00007fdaee29c0e2 in local_reply (t=0x7fdaeb054118, p_msg=<value >>>>>> optimized out>, branch=<value optimized out>, msg_status=<value optimized >>>>>> out>, cancel_bitmap=<value optimized out>) at t_reply.c:1391 >>>>>> #11 0x00007fdaee29d31d in reply_received (p_msg=0x7fdaee8e8b30) at >>>>>> t_reply.c:1540 >>>>>> #12 0x000000000042625a in forward_reply () >>>>>> #13 0x0000000000451c28 in receive_msg () >>>>>> #14 0x0000000000494e45 in udp_rcv_loop () >>>>>> #15 0x000000000042d1a3 in main () >>>>>> >>>>>> I'm not really sure how to diagnose this one. Any >>>>>> hints/fixes/suggestions would be very appreciated. >>>>>> >>>>>> Thanks, >>>>>> Tolga >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Users mailing >>>>>> [email protected]http://lists.opensips.org/cgi-bin/mailman/listinfo/users >>>>>> >>>>>> >>>>> >>>> >>> >> >
_______________________________________________ Users mailing list [email protected] http://lists.opensips.org/cgi-bin/mailman/listinfo/users
