Hi Developers of Clearwater: I posted a thread talking about Bono and Sprout crashing when running stress test. Now I collected more information about the crashing.
My test environment now consists of 2 Bono nodes, 1 Sprout node and 1 Homestead node. And I'm running Clearwater stress test on this small cluster. Each node is a KVM virtual machine running Ubuntu 12.04LTS OS. I modified the original stress test script (call_load2.xml) a little bit and increase the registration expiration time of sprout to 1 hour so that the test will first finish all the registration and then only do the call. So far the registration is all fine, but during the period when all subscribers try to call, I observe following things: *1. If the load of Sprout node is high (for a Sprout node with 2-core CPU, the CPU usage of Sprout is larger than 150%), Sprout will gradually use 80%-90% of the whole system memory before it receives a signal 6 and abort. Also, if I shut down the traffic generation nodes during the middle of a stress test. Then the memory used by both Bono and Sprout can not be free even after a rather long time (at least half an hour.).* *2. If I allocate Bono node with a 1-core CPU, then Bono node will run perfectly under even very high load (CPU usage approaches 95%). But if I allocate Bono node's CPU with more than one cores, then Bono will constantly crash itself even when the load is not very high. The following is the complete stack trace on a Bono node with a 4-core CPU* 17-01-2015 15:25:19.628 UTC Error pjsip: Assert failed: ../src/pj/os_core_unix.c:1287 mutex->owner == pj_thread_this() 17-01-2015 15:25:19.628 UTC Error pjsip: Assert failed: ../src/pj/os_core_unix.c:1287 mutex->owner == pj_thread_this() 17-01-2015 15:25:20.310 UTC Error pjsip: Assert failed: ../src/pj/os_core_unix.c:1287 mutex->owner == pj_thread_this() 17-01-2015 15:25:20.311 UTC Error pjsip: Assert failed: ../src/pj/os_core_unix.c:1287 mutex->owner == pj_thread_this() 17-01-2015 15:25:21.103 UTC Error pjsip: Assert failed: ../src/pj/os_core_unix.c:1287 mutex->owner == pj_thread_this() 17-01-2015 15:25:21.103 UTC Error pjsip: Assert failed: ../src/pj/os_core_unix.c:1287 mutex->owner == pj_thread_this() 17-01-2015 15:25:21.646 UTC Error pjsip: Assert failed: ../src/pj/os_core_unix.c:1287 mutex->owner == pj_thread_this() 17-01-2015 15:25:21.646 UTC Error pjsip: Assert failed: ../src/pj/os_core_unix.c:1287 mutex->owner == pj_thread_this() 17-01-2015 15:25:22.069 UTC Error pjsip: Assert failed: ../src/pj/os_core_unix.c:1287 mutex->owner == pj_thread_this() 17-01-2015 15:25:22.070 UTC Error pjsip: Assert failed: ../src/pj/os_core_unix.c:1287 mutex->owner == pj_thread_this() 17-01-2015 15:25:22.188 UTC Error pjsip: Assert failed: ../src/pj/os_core_unix.c:1287 mutex->owner == pj_thread_this() 17-01-2015 15:25:22.188 UTC Error pjsip: Assert failed: ../src/pj/os_core_unix.c:1287 mutex->owner == pj_thread_this() 17-01-2015 15:25:23.418 UTC Error pjsip: Assert failed: ../src/pj/os_core_unix.c:1287 mutex->owner == pj_thread_this() 17-01-2015 15:25:23.418 UTC Error pjsip: Assert failed: ../src/pj/os_core_unix.c:1287 mutex->owner == pj_thread_this() 17-01-2015 15:25:24.852 UTC Error pjsip: Assert failed: ../src/pj/os_core_unix.c:1287 mutex->owner == pj_thread_this() 17-01-2015 15:25:24.852 UTC Error pjsip: Assert failed: ../src/pj/os_core_unix.c:1287 mutex->owner == pj_thread_this() 17-01-2015 15:25:25.435 UTC Error pjsip: Assert failed: ../src/pj/os_core_unix.c:1287 mutex->owner == pj_thread_this() 17-01-2015 15:25:25.435 UTC Error pjsip: Assert failed: ../src/pj/os_core_unix.c:1287 mutex->owner == pj_thread_this() Signal 6 caught Basic stack dump: /usr/share/clearwater/bin/bono(_ZN6Logger9backtraceEPKc+0x6d)[0x49fd0d] /usr/share/clearwater/bin/bono(_ZN3Log9backtraceEPKcz+0x10d)[0x55b9bd] /usr/share/clearwater/bin/bono(_Z17exception_handleri+0x42)[0x5a8e52] /lib/x86_64-linux-gnu/libc.so.6(+0x36150)[0x7f65cadba150] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35)[0x7f65cadba0d5] /lib/x86_64-linux-gnu/libc.so.6(abort+0x17b)[0x7f65cadbd83b] /lib/x86_64-linux-gnu/libc.so.6(+0x7404e)[0x7f65cadf804e] /lib/x86_64-linux-gnu/libc.so.6(+0x7e846)[0x7f65cae02846] /lib/x86_64-linux-gnu/libc.so.6(+0x7fb1b)[0x7f65cae03b1b] /usr/share/clearwater/bin/bono[0x60dc70] /usr/share/clearwater/bin/bono(pj_pool_destroy_int+0x73)[0x61571a] /usr/share/clearwater/bin/bono[0x615f1c] /usr/share/clearwater/bin/bono(pj_pool_release+0x3d)[0x6150ed] /usr/share/clearwater/bin/bono(pjsip_endpt_release_pool+0x21)[0x5d59b5] /usr/share/clearwater/bin/bono[0x5ec1ef] /usr/share/clearwater/bin/bono[0x61379e] /usr/share/clearwater/bin/bono[0x613c46] /usr/share/clearwater/bin/bono(pj_grp_lock_dec_ref+0x18)[0x613cc5] /usr/share/clearwater/bin/bono[0x6136d0] /usr/share/clearwater/bin/bono(pj_grp_lock_release+0x18)[0x613a23] /usr/share/clearwater/bin/bono(pjsip_tsx_recv_msg+0xc1)[0x5ed9ca] /usr/share/clearwater/bin/bono[0x5ebc11] /usr/share/clearwater/bin/bono(pjsip_endpt_process_rx_data+0x1f6)[0x5d5e13] /usr/share/clearwater/bin/bono[0x4a19c1] /usr/share/clearwater/bin/bono[0x60ae84] /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a)[0x7f65cb95ce9a] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f65cae782ed] Advanced stack dump (requires gdb): [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". [New Thread 0x7f65a9ffb700 (LWP 1171)] [New Thread 0x7f65aa7fc700 (LWP 1170)] [New Thread 0x7f65aaffd700 (LWP 1169)] [New Thread 0x7f65ab7fe700 (LWP 1168)] [New Thread 0x7f65abfff700 (LWP 1167)] [New Thread 0x7f65b8ff9700 (LWP 1165)] [New Thread 0x7f65b97fa700 (LWP 1164)] [New Thread 0x7f65b9ffb700 (LWP 1153)] [New Thread 0x7f65ba7fc700 (LWP 1152)] [New Thread 0x7f65baffd700 (LWP 1151)] [New Thread 0x7f65bb7fe700 (LWP 1150)] [New Thread 0x7f65bbfff700 (LWP 1149)] [New Thread 0x7f65c08c2700 (LWP 1147)] [New Thread 0x7f65c10c3700 (LWP 1146)] [New Thread 0x7f65c18c4700 (LWP 1145)] [New Thread 0x7f65c20c5700 (LWP 1144)] [New Thread 0x7f65c50fa700 (LWP 1112)] [New Thread 0x7f65c58fb700 (LWP 1111)] warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fffebddd000 0x00007f65cb962fd0 in sem_wait () from /lib/x86_64-linux-gnu/libpthread.so.0 Thread 19 (Thread 0x7f65c58fb700 (LWP 1111)): #0 0x00007f65cb962fd0 in sem_wait () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000583fe4 in SignalHandler<1>::dispatcher (p=<optimized out>) at /var/lib/jenkins/workspace/sprout/modules/cpp-common/include/signalhandler.h:147 #2 0x00007f65cb95ce9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #3 0x00007f65cae782ed in clone () from /lib/x86_64-linux-gnu/libc.so.6 #4 0x0000000000000000 in ?? () Thread 18 (Thread 0x7f65c50fa700 (LWP 1112)): #0 0x00007f65cb962fd0 in sem_wait () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00000000005a5c3a in quiesce_unquiesce_thread_func (dummy=<optimized out>) at main.cpp:916 #2 0x00007f65cb95ce9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #3 0x00007f65cae782ed in clone () from /lib/x86_64-linux-gnu/libc.so.6 #4 0x0000000000000000 in ?? () Thread 17 (Thread 0x7f65c20c5700 (LWP 1144)): #0 0x00007f65cae78983 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007f65cc87b207 in ?? () from /usr/lib/x86_64-linux-gnu/libzmq.so.3 #2 0x00007f65cc897ef6 in ?? () from /usr/lib/x86_64-linux-gnu/libzmq.so.3 #3 0x00007f65cb95ce9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #4 0x00007f65cae782ed in clone () from /lib/x86_64-linux-gnu/libc.so.6 #5 0x0000000000000000 in ?? () Thread 16 (Thread 0x7f65c18c4700 (LWP 1145)): #0 0x00007f65cae78983 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007f65cc87b207 in ?? () from /usr/lib/x86_64-linux-gnu/libzmq.so.3 #2 0x00007f65cc897ef6 in ?? () from /usr/lib/x86_64-linux-gnu/libzmq.so.3 #3 0x00007f65cb95ce9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #4 0x00007f65cae782ed in clone () from /lib/x86_64-linux-gnu/libc.so.6 #5 0x0000000000000000 in ?? () Thread 15 (Thread 0x7f65c10c3700 (LWP 1146)): #0 0x00007f65cae6c933 in poll () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007f65cc89f546 in zmq_poll () from /usr/lib/x86_64-linux-gnu/libzmq.so.3 #2 0x00000000005677cf in LastValueCache::run (this=0x2b61720) at /var/lib/jenkins/workspace/sprout/modules/cpp-common/src/zmq_lvc.cpp:165 #3 0x00000000005680a9 in LastValueCache::last_value_cache_entry_func (lvc=<optimized out>) at /var/lib/jenkins/workspace/sprout/modules/cpp-common/src/zmq_lvc.cpp:301 #4 0x00007f65cb95ce9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007f65cae782ed in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 14 (Thread 0x7f65c08c2700 (LWP 1147)): #0 0x00007f65cb960d84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00000000005651a1 in pop (item=..., this=0x2b77520) at /var/lib/jenkins/workspace/sprout/modules/cpp-common/include/eventq.h:279 #2 Statistic::reporter (this=0x2b77508) at /var/lib/jenkins/workspace/sprout/modules/cpp-common/src/statistic.cpp:97 #3 0x0000000000565359 in Statistic::reporter_thread (p=<optimized out>) at /var/lib/jenkins/workspace/sprout/modules/cpp-common/src/statistic.cpp:132 #4 0x00007f65cb95ce9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007f65cae782ed in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 13 (Thread 0x7f65bbfff700 (LWP 1149)): #0 0x00007f65cb960d84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00000000005651a1 in pop (item=..., this=0x2b77a40) at /var/lib/jenkins/workspace/sprout/modules/cpp-common/include/eventq.h:279 #2 Statistic::reporter (this=0x2b77a28) at /var/lib/jenkins/workspace/sprout/modules/cpp-common/src/statistic.cpp:97 #3 0x0000000000565359 in Statistic::reporter_thread (p=<optimized out>) at /var/lib/jenkins/workspace/sprout/modules/cpp-common/src/statistic.cpp:132 #4 0x00007f65cb95ce9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007f65cae782ed in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 12 (Thread 0x7f65bb7fe700 (LWP 1150)): #0 0x00007f65cb960d84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00000000005651a1 in pop (item=..., this=0x2b77f60) at /var/lib/jenkins/workspace/sprout/modules/cpp-common/include/eventq.h:279 #2 Statistic::reporter (this=0x2b77f48) at /var/lib/jenkins/workspace/sprout/modules/cpp-common/src/statistic.cpp:97 #3 0x0000000000565359 in Statistic::reporter_thread (p=<optimized out>) at /var/lib/jenkins/workspace/sprout/modules/cpp-common/src/statistic.cpp:132 #4 0x00007f65cb95ce9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007f65cae782ed in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 11 (Thread 0x7f65baffd700 (LWP 1151)): #0 0x00007f65cb960d84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00000000005651a1 in pop (item=..., this=0x2b78480) at /var/lib/jenkins/workspace/sprout/modules/cpp-common/include/eventq.h:279 #2 Statistic::reporter (this=0x2b78468) at /var/lib/jenkins/workspace/sprout/modules/cpp-common/src/statistic.cpp:97 #3 0x0000000000565359 in Statistic::reporter_thread (p=<optimized out>) at /var/lib/jenkins/workspace/sprout/modules/cpp-common/src/statistic.cpp:132 #4 0x00007f65cb95ce9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007f65cae782ed in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 10 (Thread 0x7f65ba7fc700 (LWP 1152)): #0 0x00007f65cb960d84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00000000005651a1 in pop (item=..., this=0x2b78ad8) at /var/lib/jenkins/workspace/sprout/modules/cpp-common/include/eventq.h:279 #2 Statistic::reporter (this=0x2b78ac0) at /var/lib/jenkins/workspace/sprout/modules/cpp-common/src/statistic.cpp:97 #3 0x0000000000565359 in Statistic::reporter_thread (p=<optimized out>) at /var/lib/jenkins/workspace/sprout/modules/cpp-common/src/statistic.cpp:132 #4 0x00007f65cb95ce9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007f65cae782ed in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 9 (Thread 0x7f65b9ffb700 (LWP 1153)): #0 0x00007f65cb960d84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00000000005651a1 in pop (item=..., this=0x2b79020) at /var/lib/jenkins/workspace/sprout/modules/cpp-common/include/eventq.h:279 #2 Statistic::reporter (this=0x2b79008) at /var/lib/jenkins/workspace/sprout/modules/cpp-common/src/statistic.cpp:97 #3 0x0000000000565359 in Statistic::reporter_thread (p=<optimized out>) at /var/lib/jenkins/workspace/sprout/modules/cpp-common/src/statistic.cpp:132 #4 0x00007f65cb95ce9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007f65cae782ed in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 8 (Thread 0x7f65b97fa700 (LWP 1164)): #0 0x00007f65cae43dbd in nanosleep () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007f65cae43c5c in sleep () from /lib/x86_64-linux-gnu/libc.so.6 #2 0x00000000004ed3a2 in ConnectionPool::recycle_connections (this=0x2b78f40) at connection_pool.cpp:433 #3 0x00000000004ed489 in ConnectionPool::recycle_thread (p=<optimized out>) at connection_pool.cpp:470 #4 0x000000000060ae84 in thread_main (param=0x247c020) at ../src/pj/os_core_unix.c:523 #5 0x00007f65cb95ce9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007f65cae782ed in clone () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x0000000000000000 in ?? () Thread 7 (Thread 0x7f65b8ff9700 (LWP 1165)): #0 0x00007f65cae78983 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x0000000000507264 in boost::asio::detail::epoll_reactor::run (this=0x7f65b0000a60, block=<optimized out>, ops=...) at /usr/include/boost/asio/detail/impl/epoll_reactor.ipp:240 #2 0x000000000050b09d in do_one (this_idle_thread=<optimized out>, lock=..., this=<optimized out>) at /usr/include/boost/asio/detail/impl/task_io_service.ipp:264 #3 boost::asio::detail::task_io_service::run (this=0x7f65b0000980, ec=...) at /usr/include/boost/asio/detail/impl/task_io_service.ipp:130 #4 0x000000000051b504 in run (this=<optimized out>) at /usr/include/boost/asio/impl/io_service.ipp:57 #5 run_internal (this=0x7f65b8ff8bd0) at /var/lib/jenkins/workspace/sprout/usr/include/websocketpp/endpoint.hpp:53 #6 websocketpp::role::server<websocketpp::endpoint<websocketpp::role::server, websocketpp::socket::plain, websocketpp::log::logger> >::start_listen (this=0x7f65b8ff8be0, e=..., num_threads=0) at /var/lib/jenkins/workspace/sprout/usr/include/websocketpp/roles/server.hpp:312 #7 0x00000000005023f2 in listen (num_threads=1, e=..., this=0x7f65b8ff8be0) at /var/lib/jenkins/workspace/sprout/usr/include/websocketpp/roles/server.hpp:227 #8 websocket_thread (p=<optimized out>) at websockets.cpp:515 #9 0x000000000060ae84 in thread_main (param=0x247c068) at ../src/pj/os_core_unix.c:523 #10 0x00007f65cb95ce9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #11 0x00007f65cae782ed in clone () from /lib/x86_64-linux-gnu/libc.so.6 #12 0x0000000000000000 in ?? () Thread 6 (Thread 0x7f65abfff700 (LWP 1167)): #0 0x00007f65cae439fd in waitpid () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007f65cadc8f4e in ?? () from /lib/x86_64-linux-gnu/libc.so.6 #2 0x00007f65cadc9360 in system () from /lib/x86_64-linux-gnu/libc.so.6 #3 0x000000000049fdc3 in Logger::backtrace (this=0x24797c0, data=<optimized out>) at /var/lib/jenkins/workspace/sprout/modules/cpp-common/src/logger.cpp:279 #4 0x000000000055b9bd in Log::backtrace (fmt=<optimized out>) at /var/lib/jenkins/workspace/sprout/modules/cpp-common/src/log.cpp:185 #5 0x00000000005a8e52 in exception_handler (sig=6) at main.cpp:834 #6 <signal handler called> #7 0x00007f65cadba0d5 in raise () from /lib/x86_64-linux-gnu/libc.so.6 #8 0x00007f65cadbd83b in abort () from /lib/x86_64-linux-gnu/libc.so.6 #9 0x00007f65cadf804e in ?? () from /lib/x86_64-linux-gnu/libc.so.6 #10 0x00007f65cae02846 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 #11 0x00007f65cae03b1b in ?? () from /lib/x86_64-linux-gnu/libc.so.6 #12 0x000000000060dc70 in default_block_free (factory=0x89cec8, mem=0x7f6590694910, size=2048) at ../src/pj/pool_policy_malloc.c:78 #13 0x000000000061571a in pj_pool_destroy_int (pool=0x7f6590694910) at ../src/pj/pool.c:296 #14 0x0000000000615f1c in cpool_release_pool (pf=0x89cec8, pool=0x7f6590694910) at ../src/pj/pool_caching.c:238 #15 0x00000000006150ed in pj_pool_release (pool=0x7f6590694910) at ../include/pj/pool_i.h:92 #16 0x00000000005d59b5 in pjsip_endpt_release_pool (endpt=0x7f65c35e70b8, pool=0x7f6590694910) at ../src/pjsip/sip_endpoint.c:687 #17 0x00000000005ec1ef in tsx_on_destroy (arg=0x7f65906949b8) at ../src/pjsip/sip_transaction.c:1028 #18 0x000000000061379e in grp_lock_destroy (p=0x7f6590e53778) at ../src/pj/lock.c:365 #19 0x0000000000613c46 in grp_lock_dec_ref (glock=0x7f6590e53778) at ../src/pj/lock.c:527 #20 0x0000000000613cc5 in pj_grp_lock_dec_ref (glock=0x7f6590e53778) at ../src/pj/lock.c:603 #21 0x00000000006136d0 in grp_lock_release (p=0x7f6590e53778) at ../src/pj/lock.c:335 #22 0x0000000000613a23 in pj_grp_lock_release (grp_lock=0x7f6590e53778) at ../src/pj/lock.c:447 #23 0x00000000005ed9ca in pjsip_tsx_recv_msg (tsx=0x7f65906949b8, rdata=0x7f6578ddbab8) at ../src/pjsip/sip_transaction.c:1755 #24 0x00000000005ebc11 in mod_tsx_layer_on_rx_request (rdata=0x7f6578ddbab8) at ../src/pjsip/sip_transaction.c:813 #25 0x00000000005d5e13 in pjsip_endpt_process_rx_data (endpt=0x7f65c35e70b8, rdata=0x7f6578ddbab8, p=0x7f65abffee00, p_handled=0x0) at ../src/pjsip/sip_endpoint.c:885 #26 0x00000000004a19c1 in worker_thread (p=<optimized out>) at stack.cpp:207 #27 0x000000000060ae84 in thread_main (param=0x247c0b0) at ../src/pj/os_core_unix.c:523 #28 0x00007f65cb95ce9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #29 0x00007f65cae782ed in clone () from /lib/x86_64-linux-gnu/libc.so.6 #30 0x0000000000000000 in ?? () Thread 5 (Thread 0x7f65ab7fe700 (LWP 1168)): #0 0x00007f65cb960d84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00000000004a1b38 in pop (item=<synthetic pointer>, this=0x89cda0) at /var/lib/jenkins/workspace/sprout/modules/cpp-common/include/eventq.h:279 #2 worker_thread (p=<optimized out>) at stack.cpp:201 #3 0x000000000060ae84 in thread_main (param=0x247c0f8) at ../src/pj/os_core_unix.c:523 #4 0x00007f65cb95ce9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007f65cae782ed in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 4 (Thread 0x7f65aaffd700 (LWP 1169)): #0 0x00007f65cb960d84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00000000004a1b38 in pop (item=<synthetic pointer>, this=0x89cda0) at /var/lib/jenkins/workspace/sprout/modules/cpp-common/include/eventq.h:279 #2 worker_thread (p=<optimized out>) at stack.cpp:201 #3 0x000000000060ae84 in thread_main (param=0x247c140) at ../src/pj/os_core_unix.c:523 #4 0x00007f65cb95ce9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007f65cae782ed in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 3 (Thread 0x7f65aa7fc700 (LWP 1170)): #0 0x00007f65cb960d84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00000000004a1b38 in pop (item=<synthetic pointer>, this=0x89cda0) at /var/lib/jenkins/workspace/sprout/modules/cpp-common/include/eventq.h:279 #2 worker_thread (p=<optimized out>) at stack.cpp:201 #3 0x000000000060ae84 in thread_main (param=0x247c188) at ../src/pj/os_core_unix.c:523 #4 0x00007f65cb95ce9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007f65cae782ed in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 2 (Thread 0x7f65a9ffb700 (LWP 1171)): #0 0x00007f65cb96389c in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007f65cb95f065 in _L_lock_858 () from /lib/x86_64-linux-gnu/libpthread.so.0 #2 0x00007f65cb95eeba in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0 #3 0x000000000060bafb in pj_mutex_lock (mutex=0x89d118) at ../src/pj/os_core_unix.c:1243 #4 0x0000000000613239 in pj_lock_acquire (lock=0x89d0f0) at ../src/pj/lock.c:180 #5 0x0000000000615c53 in cpool_create_pool (pf=0x89cec8, name=0x64b17c "rtd%p", initial_size=4000, increment_sz=4000, callback=0) at ../src/pj/pool_caching.c:131 #6 0x00000000006150ae in pj_pool_create (f=0x89cec8, name=0x64b17c "rtd%p", initial_size=4000, increment_size=4000, callback=0) at ../include/pj/pool_i.h:86 #7 0x00000000005dd727 in pjsip_rx_data_clone (src=0x7f659c761298, flags=0, p_rdata=0x7f65a9ffa368) at ../src/pjsip/sip_transport.c:652 #8 0x00000000004a986e in on_rx_msg (rdata=0x7f659c761298) at stack.cpp:452 #9 0x00000000005d5e13 in pjsip_endpt_process_rx_data (endpt=0x7f65c35e70b8, rdata=0x7f659c761298, p=0x7f65a9ffa870, p_handled=0x7f65a9ffa8ac) at ../src/pjsip/sip_endpoint.c:885 #10 0x00000000005d6131 in endpt_on_rx_msg (endpt=0x7f65c35e70b8, status=0, rdata=0x7f659c761298) at ../src/pjsip/sip_endpoint.c:1032 #11 0x00000000005dfc3c in pjsip_tpmgr_receive_packet (mgr=0x7f65c2640550, rdata=0x7f659c761298) at ../src/pjsip/sip_transport.c:1777 #12 0x00000000005e4c72 in on_data_read (asock=0x7f659c760e50, data=0x7f659c761600, size=1771, status=0, remainder=0x7f65a9ffab40) at ../src/pjsip/sip_transport_tcp.c:1331 #13 0x000000000060fa06 in ioqueue_on_read_complete (key=0x7f65c3732f68, op_key=0x7f659c7639d8, bytes_read=1771) at ../src/pj/activesock.c:492 #14 0x00000000006083fe in ioqueue_dispatch_read_event (ioqueue=0x7f65c35f04a8, h=0x7f65c3732f68) at ../src/pj/ioqueue_common_abs.c:589 #15 0x000000000060a5f6 in pj_ioqueue_poll (ioqueue=0x7f65c35f04a8, timeout=0x7f65a9ffae00) at ../src/pj/ioqueue_epoll.c:810 #16 0x00000000005d5ad9 in pjsip_endpt_handle_events2 (endpt=0x7f65c35e70b8, max_timeout=0x7f65a9ffae50, p_count=0x0) at ../src/pjsip/sip_endpoint.c:740 #17 0x00000000005d5b86 in pjsip_endpt_handle_events (endpt=0x7f65c35e70b8, max_timeout=0x7f65a9ffae50) at ../src/pjsip/sip_endpoint.c:768 #18 0x00000000004a1cbf in pjsip_thread (p=<optimized out>) at stack.cpp:178 #19 0x000000000060ae84 in thread_main (param=0x247c1d0) at ../src/pj/os_core_unix.c:523 #20 0x00007f65cb95ce9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #21 0x00007f65cae782ed in clone () from /lib/x86_64-linux-gnu/libc.so.6 #22 0x0000000000000000 in ?? () Thread 1 (Thread 0x7f65cdc71740 (LWP 1080)): #0 0x00007f65cb962fd0 in sem_wait () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x000000000049e54e in main (argc=<optimized out>, argv=<optimized out>) at main.cpp:1630 Could you please help me with these problems? Thank you very much! Best, Duan Jingpu _______________________________________________ Clearwater mailing list [email protected] http://lists.projectclearwater.org/listinfo/clearwater
