Hi Paul, It looks like it could have been monit that killed the bono process. Can you look in the monit logs (in /var/log/monit.log) to see if they indicate what has happened?
Also, is this problem reproducible? If so, can you let me know the exact scenario you're testing with? Thanks, Ellie -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Paul Sun Sent: 13 November 2014 04:08 To: [email protected] Subject: [Clearwater] Bono crached on I-BCF Hi While testing with I-BCF, it is found that it is crashed during establishing the call. Any idea? - PS SIP/2.0 200 OK^M Via: SIP/2.0/TCP 192.168.1.228;rport=55776;received=192.168.1.228;branch=z9hG4bK-153331^M Call-ID: poll-sip-153331^M From: "poll-sip" <sip:[email protected]>;tag=153331^M To: <sip:[email protected]>;tag=z9hG4bK-153331^M CSeq: 153331 OPTIONS^M Content-Length: 0^M ^M --end msg-- 13-11-2014 04:03:49.320 Debug pjsip: tdta0x7fd0c400 Destroying txdata Response msg 200/OPTIONS/cseq=153331 (tdta0x7fd0c4000eb0) 13-11-2014 04:03:49.320 Debug stack.cpp:191: Worker thread completed processing message 0x7fd0cc078628 13-11-2014 04:03:49.320 Debug stack.cpp:197: Request latency = 633us 13-11-2014 04:03:49.321 Verbose pjsip: tcps0x7fd0cc07 TCP connection closed 13-11-2014 04:03:49.321 Debug connection_tracker.cpp:91: Connection 0x7fd0cc074228 has been destroyed 13-11-2014 04:03:49.321 Verbose pjsip: tcps0x7fd0cc07 TCP transport destroyed with reason 70016: End of file (PJ_EEOF) Signal 6 caught Basic stack dump: /usr/share/clearwater/bin/bono[0x428c1d] /usr/share/clearwater/bin/bono[0x4cddcf] /usr/share/clearwater/bin/bono[0x5058d9] /lib/x86_64-linux-gnu/libc.so.6(+0x364a0)[0x7fd0f9e844a0] /lib/x86_64-linux-gnu/libpthread.so.0(sem_wait+0x2e)[0x7fd0fac8bfce] /usr/share/clearwater/bin/bono[0x427c4d] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed)[0x7fd0f9e6f76d] /usr/share/clearwater/bin/bono[0x428ab9] Advanced stack dump (requires gdb): [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". [New Thread 0x7fd0e57fa700 (LWP 1573)] [New Thread 0x7fd0e5ffb700 (LWP 1572)] [New Thread 0x7fd0e67fc700 (LWP 1571)] [New Thread 0x7fd0e6ffd700 (LWP 1570)] [New Thread 0x7fd0e77fe700 (LWP 1569)] [New Thread 0x7fd0e7fff700 (LWP 1568)] [New Thread 0x7fd0ecf44700 (LWP 1567)] [New Thread 0x7fd0ed745700 (LWP 1566)] [New Thread 0x7fd0edf46700 (LWP 1565)] [New Thread 0x7fd0ee747700 (LWP 1564)] [New Thread 0x7fd0eef48700 (LWP 1563)] [New Thread 0x7fd0ef749700 (LWP 1562)] [New Thread 0x7fd0eff4a700 (LWP 1560)] [New Thread 0x7fd0f074b700 (LWP 1558)] [New Thread 0x7fd0f3988700 (LWP 1556)] [New Thread 0x7fd0f4189700 (LWP 1555)] warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fff1d3fe000 0x00007fd0f9f0dccd in waitpid () from /lib/x86_64-linux-gnu/libc.so.6 Thread 17 (Thread 0x7fd0f4189700 (LWP 1555)): #0 0x00007fd0fac8bfd0 in sem_wait () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00000000004f5b34 in ?? () #2 0x00007fd0fac85e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #3 0x00007fd0f9f423fd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #4 0x0000000000000000 in ?? () Thread 16 (Thread 0x7fd0f3988700 (LWP 1556)): #0 0x00007fd0fac8bfd0 in sem_wait () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x000000000050588a in ?? () #2 0x00007fd0fac85e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #3 0x00007fd0f9f423fd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #4 0x0000000000000000 in ?? () Thread 15 (Thread 0x7fd0f074b700 (LWP 1558)): #0 0x00007fd0f9f42a93 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007fd0fb502207 in ?? () from /usr/lib/x86_64-linux-gnu/libzmq.so.3 #2 0x00007fd0fb51eef6 in ?? () from /usr/lib/x86_64-linux-gnu/libzmq.so.3 #3 0x00007fd0fac85e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #4 0x00007fd0f9f423fd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #5 0x0000000000000000 in ?? () Thread 14 (Thread 0x7fd0eff4a700 (LWP 1560)): #0 0x00007fd0f9f42a93 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007fd0fb502207 in ?? () from /usr/lib/x86_64-linux-gnu/libzmq.so.3 #2 0x00007fd0fb51eef6 in ?? () from /usr/lib/x86_64-linux-gnu/libzmq.so.3 #3 0x00007fd0fac85e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #4 0x00007fd0f9f423fd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #5 0x0000000000000000 in ?? () Thread 13 (Thread 0x7fd0ef749700 (LWP 1562)): #0 0x00007fd0f9f36a43 in poll () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007fd0fb526546 in zmq_poll () from /usr/lib/x86_64-linux-gnu/libzmq.so.3 #2 0x00000000004d3d1e in ?? () #3 0x00000000004d4519 in ?? () #4 0x00007fd0fac85e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fd0f9f423fd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 12 (Thread 0x7fd0eef48700 (LWP 1563)): #0 0x00007fd0fac89d84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00000000004d1761 in ?? () #2 0x00000000004d1969 in ?? () #3 0x00007fd0fac85e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #4 0x00007fd0f9f423fd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #5 0x0000000000000000 in ?? () Thread 11 (Thread 0x7fd0ee747700 (LWP 1564)): #0 0x00007fd0fac89d84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00000000004d1761 in ?? () #2 0x00000000004d1969 in ?? () #3 0x00007fd0fac85e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #4 0x00007fd0f9f423fd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #5 0x0000000000000000 in ?? () Thread 10 (Thread 0x7fd0edf46700 (LWP 1565)): #0 0x00007fd0fac89d84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00000000004d1761 in ?? () #2 0x00000000004d1969 in ?? () #3 0x00007fd0fac85e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #4 0x00007fd0f9f423fd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #5 0x0000000000000000 in ?? () Thread 9 (Thread 0x7fd0ed745700 (LWP 1566)): #0 0x00007fd0fac89d84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00000000004d1761 in ?? () #2 0x00000000004d1969 in ?? () #3 0x00007fd0fac85e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #4 0x00007fd0f9f423fd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #5 0x0000000000000000 in ?? () Thread 8 (Thread 0x7fd0ecf44700 (LWP 1567)): #0 0x00007fd0fac89d84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00000000004d1761 in ?? () #2 0x00000000004d1969 in ?? () #3 0x00007fd0fac85e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #4 0x00007fd0f9f423fd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #5 0x0000000000000000 in ?? () Thread 7 (Thread 0x7fd0e7fff700 (LWP 1568)): #0 0x00007fd0fac89d84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00000000004d1761 in ?? () #2 0x00000000004d1969 in ?? () #3 0x00007fd0fac85e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #4 0x00007fd0f9f423fd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #5 0x0000000000000000 in ?? () Thread 6 (Thread 0x7fd0e77fe700 (LWP 1569)): #0 0x00007fd0fac89d84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00000000004d1761 in ?? () #2 0x00000000004d1969 in ?? () #3 0x00007fd0fac85e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #4 0x00007fd0f9f423fd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #5 0x0000000000000000 in ?? () Thread 5 (Thread 0x7fd0e6ffd700 (LWP 1570)): #0 0x00007fd0f9f0e08d in nanosleep () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007fd0f9f0df2c in sleep () from /lib/x86_64-linux-gnu/libc.so.6 #2 0x000000000046faa2 in ?? () #3 0x000000000046fb89 in ?? () #4 0x00000000005630b8 in ?? () #5 0x00007fd0fac85e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007fd0f9f423fd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x0000000000000000 in ?? () Thread 4 (Thread 0x7fd0e67fc700 (LWP 1571)): #0 0x00007fd0f9f42a93 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x0000000000486204 in ?? () #2 0x000000000048a03d in ?? () #3 0x000000000049a494 in ?? () #4 0x0000000000481392 in ?? () #5 0x00000000005630b8 in ?? () #6 0x00007fd0fac85e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #7 0x00007fd0f9f423fd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #8 0x0000000000000000 in ?? () Thread 3 (Thread 0x7fd0e5ffb700 (LWP 1572)): #0 0x00007fd0fac89d84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x000000000042a250 in ?? () #2 0x00000000005630b8 in ?? () #3 0x00007fd0fac85e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #4 0x00007fd0f9f423fd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #5 0x0000000000000000 in ?? () Thread 2 (Thread 0x7fd0e57fa700 (LWP 1573)): #0 0x00007fd0f9f42a93 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00000000005623ea in ?? () #2 0x000000000052cd69 in ?? () #3 0x000000000052ce16 in ?? () #4 0x000000000042a3df in ?? () #5 0x00000000005630b8 in ?? () #6 0x00007fd0fac85e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #7 0x00007fd0f9f423fd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #8 0x0000000000000000 in ?? () Thread 1 (Thread 0x7fd0fc6f3740 (LWP 1553)): #0 0x00007fd0f9f0dccd in waitpid () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007fd0f9e9329e in ?? () from /lib/x86_64-linux-gnu/libc.so.6 #2 0x00007fd0f9e936b0 in system () from /lib/x86_64-linux-gnu/libc.so.6 #3 0x0000000000428cd3 in ?? () #4 0x00000000004cddcf in ?? () #5 0x00000000005058d9 in ?? () #6 <signal handler called> #7 0x00007fd0fac8bfce in sem_wait () from /lib/x86_64-linux-gnu/libpthread.so.0 #8 0x0000000000427c4d in ?? () #9 0x00007fd0f9e6f76d in __libc_start_main () from /lib/x86_64-linux-gnu/libc.so.6 #10 0x0000000000428ab9 in ?? () #11 0x00007fff1d3fb588 in ?? () #12 0x000000000000001c in ?? () #13 0x000000000000001d in ?? () #14 0x00007fff1d3fcd13 in ?? () #15 0x00007fff1d3fcd32 in ?? () #16 0x00007fff1d3fcd3b in ?? () #17 0x00007fff1d3fcd50 in ?? () #18 0x00007fff1d3fcd5c in ?? () #19 0x00007fff1d3fcd84 in ?? () #20 0x00007fff1d3fcd8c in ?? () #21 0x00007fff1d3fcd9a in ?? () #22 0x00007fff1d3fcda2 in ?? () #23 0x00007fff1d3fcdac in ?? () #24 0x00007fff1d3fcdba in ?? () #25 0x00007fff1d3fcdbf in ?? () #26 0x00007fff1d3fcdcf in ?? () #27 0x00007fff1d3fcdf7 in ?? () #28 0x00007fff1d3fcdfe in ?? () #29 0x00007fff1d3fce1e in ?? () #30 0x00007fff1d3fce24 in ?? () #31 0x00007fff1d3fce4b in ?? () #32 0x00007fff1d3fce5b in ?? () #33 0x00007fff1d3fce5d in ?? () #34 0x00007fff1d3fce6e in ?? () #35 0x00007fff1d3fce70 in ?? () #36 0x00007fff1d3fce73 in ?? () #37 0x00007fff1d3fce81 in ?? () #38 0x00007fff1d3fce84 in ?? () #39 0x00007fff1d3fce92 in ?? () #40 0x00007fff1d3fce95 in ?? () #41 0x00007fff1d3fce97 in ?? () #42 0x00007fff1d3fce9e in ?? () #43 0x0000000000000000 in ?? () _______________________________________________ Clearwater mailing list [email protected] http://lists.projectclearwater.org/listinfo/clearwater _______________________________________________ Clearwater mailing list [email protected] http://lists.projectclearwater.org/listinfo/clearwater
