michael goulish created DISPATCH-2014:
-----------------------------------------
Summary: Router TCP Adapter crash with high thread count and load
Key: DISPATCH-2014
URL: https://issues.apache.org/jira/browse/DISPATCH-2014
Project: Qpid Dispatch
Issue Type: Bug
Components: Protocol Adaptors
Reporter: michael goulish
Using latest proton and dispatch master code as of 3 hours ago.
Testing router TCP adapter on a machine with 32 cores / 64 threads.
I gave the router 64 worker threads, then used 'hey' load generator to send it
HTTP requests to a TCP listener which router forwarded to Nginx on same
machine.
Multiple tests with increasing number of parallel senders: 10, 20, 30,...Each
sender throttled to 10 messages per second.
It survived many tests, but crashed around test with 200 senders.
I believe this is easily repeatable – I will go check that now.
Here is the thread that crashed:
{color:#de350b} #0 0x00007f33186a0684 in pthread_mutex_lock () from
/lib64/libpthread.so.0{color}
{color:#de350b} #1 0x00007f33186e2848 in lock (m=<optimized out>){color}
{color:#de350b} at
/home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326{color}
{color:#de350b} #2 process (tsk=<optimized out>){color}
{color:#de350b} at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2248{color}
{color:#de350b} #3 next_event_batch (p=0x10ed970, can_block=true){color}
{color:#de350b} at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2423{color}
{color:#de350b} #4 0x00007f33187c192f in thread_run (arg=0x10f6e40){color}
{color:#de350b} at /home/mick/latest/qpid-dispatch/src/server.c:1107{color}
{color:#de350b} #5 0x00007f331869e3f9 in start_thread () from
/lib64/libpthread.so.0{color}
{color:#de350b} #6 0x00007f33181b2b53 in clone () from /lib64/libc.so.6{color}
{color:#172b4d}And here are all the threads:{color}
{color:#de350b}(gdb) thread apply all bt{color}
{color:#de350b}Thread 65 (Thread 0x7f3244ff9640 (LWP 36500)):{color}
{color:#de350b}#0 0x00007f33186a7ea0 in __lll_lock_wait () from
/lib64/libpthread.so.0{color}
{color:#de350b}#1 0x00007f33186a08f5 in pthread_mutex_lock () from
/lib64/libpthread.so.0{color}
{color:#de350b}#2 0x00007f33186dfc5f in lock (m=0x10edc90) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326{color}
{color:#de350b}#3 pni_raw_connection_done (rc=0x10ed3b8) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll_raw_connection.c:423{color}
{color:#de350b}#4 pn_proactor_done (batch=0x10ed970, p=0x10ed970) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2696{color}
{color:#de350b}#5 pn_proactor_done (p=0x10ed970,
batch=batch@entry=0x7f326811a578) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2676{color}
{color:#de350b}#6 0x00007f33187c1a11 in thread_run (arg=0x10f6e40) at
/home/mick/latest/qpid-dispatch/src/server.c:1140{color}
{color:#de350b}#7 0x00007f331869e3f9 in start_thread () from
/lib64/libpthread.so.0{color}
{color:#de350b}#8 0x00007f33181b2b53 in clone () from /lib64/libc.so.6{color}
{color:#de350b}Thread 64 (Thread 0x7f327ffff640 (LWP 36481)):{color}
{color:#de350b}#0 0x00007f33186a7ea0 in __lll_lock_wait () from
/lib64/libpthread.so.0{color}
{color:#de350b}#1 0x00007f33186a08f5 in pthread_mutex_lock () from
/lib64/libpthread.so.0{color}
{color:#de350b}#2 0x00007f33186e2b7e in lock (m=0x10edc90) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326{color}
{color:#de350b}#3 process (tsk=<optimized out>) at
/home/mick/latest/qpid-proton/c/src/proacto--Type <RET> for more, q to quit, c
to continue without paging--{color}
{color:#de350b}r/epoll.c:2248{color}
{color:#de350b}#4 next_event_batch (p=<optimized out>, can_block=true) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2423{color}
{color:#de350b}#5 0x00007f33187c192f in thread_run (arg=0x10f6e40) at
/home/mick/latest/qpid-dispatch/src/server.c:1107{color}
{color:#de350b}#6 0x00007f331869e3f9 in start_thread () from
/lib64/libpthread.so.0{color}
{color:#de350b}#7 0x00007f33181b2b53 in clone () from /lib64/libc.so.6{color}
{color:#de350b}Thread 63 (Thread 0x7f322f7fe640 (LWP 36502)):{color}
{color:#de350b}#0 0x00007f33186a7ea0 in __lll_lock_wait () from
/lib64/libpthread.so.0{color}
{color:#de350b}#1 0x00007f33186a08f5 in pthread_mutex_lock () from
/lib64/libpthread.so.0{color}
{color:#de350b}#2 0x00007f33186dfc5f in lock (m=0x10edc90) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326{color}
{color:#de350b}#3 pni_raw_connection_done (rc=0x10ed3b8) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll_raw_connection.c:423{color}
{color:#de350b}#4 pn_proactor_done (batch=0x10ed970, p=0x10ed970) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2696{color}
{color:#de350b}#5 pn_proactor_done (p=0x10ed970,
batch=batch@entry=0x7f32c8063af8) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2676{color}
{color:#de350b}#6 0x00007f33187c1a11 in thread_run (arg=0x10f6e40) at
/home/mick/latest/qpid-dispatch/src/server.c:1140{color}
{color:#de350b}#7 0x00007f331869e3f9 in start_thread () from
/lib64/libpthread.so.0{color}
{color:#de350b}#8 0x00007f33181b2b53 in clone () from /lib64/libc.so.6{color}
{color:#de350b}--Type <RET> for more, q to quit, c to continue without
paging--c{color}
{color:#de350b}Thread 62 (Thread 0x7f322d7fa640 (LWP 36506)):{color}
{color:#de350b}#0 0x00007f33186a7ea0 in __lll_lock_wait () from
/lib64/libpthread.so.0{color}
{color:#de350b}#1 0x00007f33186a08f5 in pthread_mutex_lock () from
/lib64/libpthread.so.0{color}
{color:#de350b}#2 0x00007f33186e2509 in lock (m=0x10edc90) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326{color}
{color:#de350b}#3 next_event_batch (p=0x10ed970, can_block=true) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2411{color}
{color:#de350b}#4 0x00007f33187c192f in thread_run (arg=0x10f6e40) at
/home/mick/latest/qpid-dispatch/src/server.c:1107{color}
{color:#de350b}#5 0x00007f331869e3f9 in start_thread () from
/lib64/libpthread.so.0{color}
{color:#de350b}#6 0x00007f33181b2b53 in clone () from /lib64/libc.so.6{color}
{color:#de350b}Thread 61 (Thread 0x7f322dffb640 (LWP 36505)):{color}
{color:#de350b}#0 0x00007f33186a7ea0 in __lll_lock_wait () from
/lib64/libpthread.so.0{color}
{color:#de350b}#1 0x00007f33186a08f5 in pthread_mutex_lock () from
/lib64/libpthread.so.0{color}
{color:#de350b}#2 0x00007f33186e2b7e in lock (m=0x10edc90) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326{color}
{color:#de350b}#3 process (tsk=<optimized out>) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2248{color}
{color:#de350b}#4 next_event_batch (p=<optimized out>, can_block=true) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2423{color}
{color:#de350b}#5 0x00007f33187c192f in thread_run (arg=0x10f6e40) at
/home/mick/latest/qpid-dispatch/src/server.c:1107{color}
{color:#de350b}#6 0x00007f331869e3f9 in start_thread () from
/lib64/libpthread.so.0{color}
{color:#de350b}#7 0x00007f33181b2b53 in clone () from /lib64/libc.so.6{color}
{color:#de350b}Thread 60 (Thread 0x7f32d17fa640 (LWP 36471)):{color}
{color:#de350b}#0 0x00007f33186a7ea0 in __lll_lock_wait () from
/lib64/libpthread.so.0{color}
{color:#de350b}#1 0x00007f33186a08f5 in pthread_mutex_lock () from
/lib64/libpthread.so.0{color}
{color:#de350b}#2 0x00007f33186dfc5f in lock (m=0x10edc90) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326{color}
{color:#de350b}#3 pni_raw_connection_done (rc=0x10ed3b8) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll_raw_connection.c:423{color}
{color:#de350b}#4 pn_proactor_done (batch=0x10ed970, p=0x10ed970) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2696{color}
{color:#de350b}#5 pn_proactor_done (p=0x10ed970,
batch=batch@entry=0x7f32880d36a8) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2676{color}
{color:#de350b}#6 0x00007f33187c1a11 in thread_run (arg=0x10f6e40) at
/home/mick/latest/qpid-dispatch/src/server.c:1140{color}
{color:#de350b}#7 0x00007f331869e3f9 in start_thread () from
/lib64/libpthread.so.0{color}
{color:#de350b}#8 0x00007f33181b2b53 in clone () from /lib64/libc.so.6{color}
{color:#de350b}Thread 59 (Thread 0x7f322effd640 (LWP 36503)):{color}
{color:#de350b}#0 0x00007f33186a7ea0 in __lll_lock_wait () from
/lib64/libpthread.so.0{color}
{color:#de350b}#1 0x00007f33186a08f5 in pthread_mutex_lock () from
/lib64/libpthread.so.0{color}
{color:#de350b}#2 0x00007f33186dfc5f in lock (m=0x10edc90) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326{color}
{color:#de350b}#3 pni_raw_connection_done (rc=0x10ed3b8) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll_raw_connection.c:423{color}
{color:#de350b}#4 pn_proactor_done (batch=0x10ed970, p=0x10ed970) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2696{color}
{color:#de350b}#5 pn_proactor_done (p=0x10ed970,
batch=batch@entry=0x7f325c074f68) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2676{color}
{color:#de350b}#6 0x00007f33187c1a11 in thread_run (arg=0x10f6e40) at
/home/mick/latest/qpid-dispatch/src/server.c:1140{color}
{color:#de350b}#7 0x00007f331869e3f9 in start_thread () from
/lib64/libpthread.so.0{color}
{color:#de350b}#8 0x00007f33181b2b53 in clone () from /lib64/libc.so.6{color}
{color:#de350b}Thread 58 (Thread 0x7f327e7fc640 (LWP 36484)):{color}
{color:#de350b}#0 0x00007f33186a7ea0 in __lll_lock_wait () from
/lib64/libpthread.so.0{color}
{color:#de350b}#1 0x00007f33186a08f5 in pthread_mutex_lock () from
/lib64/libpthread.so.0{color}
{color:#de350b}#2 0x00007f33186dfc5f in lock (m=0x10edc90) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326{color}
{color:#de350b}#3 pni_raw_connection_done (rc=0x10ed3b8) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll_raw_connection.c:423{color}
{color:#de350b}#4 pn_proactor_done (batch=0x10ed970, p=0x10ed970) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2696{color}
{color:#de350b}#5 pn_proactor_done (p=0x10ed970,
batch=batch@entry=0x7f32f0087df8) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2676{color}
{color:#de350b}#6 0x00007f33187c1a11 in thread_run (arg=0x10f6e40) at
/home/mick/latest/qpid-dispatch/src/server.c:1140{color}
{color:#de350b}#7 0x00007f331869e3f9 in start_thread () from
/lib64/libpthread.so.0{color}
{color:#de350b}#8 0x00007f33181b2b53 in clone () from /lib64/libc.so.6{color}
{color:#de350b}Thread 57 (Thread 0x7f3276ffd640 (LWP 36489)):{color}
{color:#de350b}#0 0x00007f33186a7ea0 in __lll_lock_wait () from
/lib64/libpthread.so.0{color}
{color:#de350b}#1 0x00007f33186a08f5 in pthread_mutex_lock () from
/lib64/libpthread.so.0{color}
{color:#de350b}#2 0x00007f33186dfc5f in lock (m=0x10edc90) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326{color}
{color:#de350b}#3 pni_raw_connection_done (rc=0x10ed3b8) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll_raw_connection.c:423{color}
{color:#de350b}#4 pn_proactor_done (batch=0x10ed970, p=0x10ed970) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2696{color}
{color:#de350b}#5 pn_proactor_done (p=0x10ed970,
batch=batch@entry=0x7f328c0e1e48) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2676{color}
{color:#de350b}#6 0x00007f33187c1a11 in thread_run (arg=0x10f6e40) at
/home/mick/latest/qpid-dispatch/src/server.c:1140{color}
{color:#de350b}#7 0x00007f331869e3f9 in start_thread () from
/lib64/libpthread.so.0{color}
{color:#de350b}#8 0x00007f33181b2b53 in clone () from /lib64/libc.so.6{color}
{color:#de350b}Thread 56 (Thread 0x7f32757fa640 (LWP 36492)):{color}
{color:#de350b}#0 0x00007f33186a7ea0 in __lll_lock_wait () from
/lib64/libpthread.so.0{color}
{color:#de350b}#1 0x00007f33186a08f5 in pthread_mutex_lock () from
/lib64/libpthread.so.0{color}
{color:#de350b}#2 0x00007f33186e2509 in lock (m=0x10edc90) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326{color}
{color:#de350b}#3 next_event_batch (p=0x10ed970, can_block=true) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2411{color}
{color:#de350b}#4 0x00007f33187c192f in thread_run (arg=0x10f6e40) at
/home/mick/latest/qpid-dispatch/src/server.c:1107{color}
{color:#de350b}#5 0x00007f331869e3f9 in start_thread () from
/lib64/libpthread.so.0{color}
{color:#de350b}#6 0x00007f33181b2b53 in clone () from /lib64/libc.so.6{color}
{color:#de350b}Thread 55 (Thread 0x7f32477fe640 (LWP 36495)):{color}
{color:#de350b}#0 0x00007f33186a7ea0 in __lll_lock_wait () from
/lib64/libpthread.so.0{color}
{color:#de350b}#1 0x00007f33186a08f5 in pthread_mutex_lock () from
/lib64/libpthread.so.0{color}
{color:#de350b}#2 0x00007f33186e2b7e in lock (m=0x10edc90) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326{color}
{color:#de350b}#3 process (tsk=<optimized out>) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2248{color}
{color:#de350b}#4 next_event_batch (p=<optimized out>, can_block=true) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2423{color}
{color:#de350b}#5 0x00007f33187c192f in thread_run (arg=0x10f6e40) at
/home/mick/latest/qpid-dispatch/src/server.c:1107{color}
{color:#de350b}#6 0x00007f331869e3f9 in start_thread () from
/lib64/libpthread.so.0{color}
{color:#de350b}#7 0x00007f33181b2b53 in clone () from /lib64/libc.so.6{color}
{color:#de350b}Thread 54 (Thread 0x7f32d2ffd640 (LWP 36468)):{color}
{color:#de350b}#0 0x00007f33186a7ea0 in __lll_lock_wait () from
/lib64/libpthread.so.0{color}
{color:#de350b}#1 0x00007f33186a08f5 in pthread_mutex_lock () from
/lib64/libpthread.so.0{color}
{color:#de350b}#2 0x00007f33186e2b7e in lock (m=0x10edc90) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326{color}
{color:#de350b}#3 process (tsk=<optimized out>) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2248{color}
{color:#de350b}#4 next_event_batch (p=<optimized out>, can_block=true) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2423{color}
{color:#de350b}#5 0x00007f33187c192f in thread_run (arg=0x10f6e40) at
/home/mick/latest/qpid-dispatch/src/server.c:1107{color}
{color:#de350b}#6 0x00007f331869e3f9 in start_thread () from
/lib64/libpthread.so.0{color}
{color:#de350b}#7 0x00007f33181b2b53 in clone () from /lib64/libc.so.6{color}
{color:#de350b}Thread 53 (Thread 0x7f3245ffb640 (LWP 36498)):{color}
{color:#de350b}#0 0x00007f33186a7ea0 in __lll_lock_wait () from
/lib64/libpthread.so.0{color}
{color:#de350b}#1 0x00007f33186a08f5 in pthread_mutex_lock () from
/lib64/libpthread.so.0{color}
{color:#de350b}#2 0x00007f33186dfc5f in lock (m=0x10edc90) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326{color}
{color:#de350b}#3 pni_raw_connection_done (rc=0x10ed3b8) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll_raw_connection.c:423{color}
{color:#de350b}#4 pn_proactor_done (batch=0x10ed970, p=0x10ed970) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2696{color}
{color:#de350b}#5 pn_proactor_done (p=0x10ed970,
batch=batch@entry=0x7f32a80994e8) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2676{color}
{color:#de350b}#6 0x00007f33187c1a11 in thread_run (arg=0x10f6e40) at
/home/mick/latest/qpid-dispatch/src/server.c:1140{color}
{color:#de350b}#7 0x00007f331869e3f9 in start_thread () from
/lib64/libpthread.so.0{color}
{color:#de350b}#8 0x00007f33181b2b53 in clone () from /lib64/libc.so.6{color}
{color:#de350b}Thread 52 (Thread 0x7f322e7fc640 (LWP 36504)):{color}
{color:#de350b}#0 0x00007f33186a7ea0 in __lll_lock_wait () from
/lib64/libpthread.so.0{color}
{color:#de350b}#1 0x00007f33186a08f5 in pthread_mutex_lock () from
/lib64/libpthread.so.0{color}
{color:#de350b}#2 0x00007f33186e2b7e in lock (m=0x10edc90) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326{color}
{color:#de350b}#3 process (tsk=<optimized out>) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2248{color}
{color:#de350b}#4 next_event_batch (p=<optimized out>, can_block=true) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2423{color}
{color:#de350b}#5 0x00007f33187c192f in thread_run (arg=0x10f6e40) at
/home/mick/latest/qpid-dispatch/src/server.c:1107{color}
{color:#de350b}#6 0x00007f331869e3f9 in start_thread () from
/lib64/libpthread.so.0{color}
{color:#de350b}#7 0x00007f33181b2b53 in clone () from /lib64/libc.so.6{color}
{color:#de350b}Thread 51 (Thread 0x7f32467fc640 (LWP 36497)):{color}
{color:#de350b}#0 0x00007f33186a7ea0 in __lll_lock_wait () from
/lib64/libpthread.so.0{color}
{color:#de350b}#1 0x00007f33186a08f5 in pthread_mutex_lock () from
/lib64/libpthread.so.0{color}
{color:#de350b}#2 0x00007f33186dfc5f in lock (m=0x10edc90) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326{color}
{color:#de350b}#3 pni_raw_connection_done (rc=0x10ed3b8) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll_raw_connection.c:423{color}
{color:#de350b}#4 pn_proactor_done (batch=0x10ed970, p=0x10ed970) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2696{color}
{color:#de350b}#5 pn_proactor_done (p=0x10ed970,
batch=batch@entry=0x7f329001ce38) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2676{color}
{color:#de350b}#6 0x00007f33187c1a11 in thread_run (arg=0x10f6e40) at
/home/mick/latest/qpid-dispatch/src/server.c:1140{color}
{color:#de350b}#7 0x00007f331869e3f9 in start_thread () from
/lib64/libpthread.so.0{color}
{color:#de350b}#8 0x00007f33181b2b53 in clone () from /lib64/libc.so.6{color}
{color:#de350b}Thread 50 (Thread 0x7f32da7fc640 (LWP 36463)):{color}
{color:#de350b}#0 0x00007f33186a7ea0 in __lll_lock_wait () from
/lib64/libpthread.so.0{color}
{color:#de350b}#1 0x00007f33186a08f5 in pthread_mutex_lock () from
/lib64/libpthread.so.0{color}
{color:#de350b}#2 0x00007f33186e2509 in lock (m=0x10edc90) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326{color}
{color:#de350b}#3 next_event_batch (p=0x10ed970, can_block=true) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2411{color}
{color:#de350b}#4 0x00007f33187c192f in thread_run (arg=0x10f6e40) at
/home/mick/latest/qpid-dispatch/src/server.c:1107{color}
{color:#de350b}#5 0x00007f331869e3f9 in start_thread () from
/lib64/libpthread.so.0{color}
{color:#de350b}#6 0x00007f33181b2b53 in clone () from /lib64/libc.so.6{color}
{color:#de350b}Thread 49 (Thread 0x7f3247fff640 (LWP 36494)):{color}
{color:#de350b}#0 0x00007f33186a7ea0 in __lll_lock_wait () from
/lib64/libpthread.so.0{color}
{color:#de350b}#1 0x00007f33186a08f5 in pthread_mutex_lock () from
/lib64/libpthread.so.0{color}
{color:#de350b}#2 0x00007f33186e2509 in lock (m=0x10edc90) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326{color}
{color:#de350b}#3 next_event_batch (p=0x10ed970, can_block=true) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2411{color}
{color:#de350b}#4 0x00007f33187c192f in thread_run (arg=0x10f6e40) at
/home/mick/latest/qpid-dispatch/src/server.c:1107{color}
{color:#de350b}#5 0x00007f331869e3f9 in start_thread () from
/lib64/libpthread.so.0{color}
{color:#de350b}#6 0x00007f33181b2b53 in clone () from /lib64/libc.so.6{color}
{color:#de350b}Thread 48 (Thread 0x7f3274ff9640 (LWP 36493)):{color}
{color:#de350b}#0 0x00007f33186a7ea0 in __lll_lock_wait () from
/lib64/libpthread.so.0{color}
{color:#de350b}#1 0x00007f33186a08f5 in pthread_mutex_lock () from
/lib64/libpthread.so.0{color}
{color:#de350b}#2 0x00007f33186e2509 in lock (m=0x10edc90) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326{color}
{color:#de350b}#3 next_event_batch (p=0x10ed970, can_block=true) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2411{color}
{color:#de350b}#4 0x00007f33187c192f in thread_run (arg=0x10f6e40) at
/home/mick/latest/qpid-dispatch/src/server.c:1107{color}
{color:#de350b}#5 0x00007f331869e3f9 in start_thread () from
/lib64/libpthread.so.0{color}
{color:#de350b}#6 0x00007f33181b2b53 in clone () from /lib64/libc.so.6{color}
{color:#de350b}Thread 47 (Thread 0x7f3275ffb640 (LWP 36491)):{color}
{color:#de350b}#0 0x00007f33186a7ea0 in __lll_lock_wait () from
/lib64/libpthread.so.0{color}
{color:#de350b}#1 0x00007f33186a08f5 in pthread_mutex_lock () from
/lib64/libpthread.so.0{color}
{color:#de350b}#2 0x00007f33186e2509 in lock (m=0x10edc90) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326{color}
{color:#de350b}#3 next_event_batch (p=0x10ed970, can_block=true) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2411{color}
{color:#de350b}#4 0x00007f33187c192f in thread_run (arg=0x10f6e40) at
/home/mick/latest/qpid-dispatch/src/server.c:1107{color}
{color:#de350b}#5 0x00007f331869e3f9 in start_thread () from
/lib64/libpthread.so.0{color}
{color:#de350b}#6 0x00007f33181b2b53 in clone () from /lib64/libc.so.6{color}
{color:#de350b}Thread 46 (Thread 0x7f32767fc640 (LWP 36490)):{color}
{color:#de350b}#0 0x00007f33186a7ea0 in __lll_lock_wait () from
/lib64/libpthread.so.0{color}
{color:#de350b}#1 0x00007f33186a08f5 in pthread_mutex_lock () from
/lib64/libpthread.so.0{color}
{color:#de350b}#2 0x00007f33186e2b7e in lock (m=0x10edc90) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326{color}
{color:#de350b}#3 process (tsk=<optimized out>) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2248{color}
{color:#de350b}#4 next_event_batch (p=<optimized out>, can_block=true) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2423{color}
{color:#de350b}#5 0x00007f33187c192f in thread_run (arg=0x10f6e40) at
/home/mick/latest/qpid-dispatch/src/server.c:1107{color}
{color:#de350b}#6 0x00007f331869e3f9 in start_thread () from
/lib64/libpthread.so.0{color}
{color:#de350b}#7 0x00007f33181b2b53 in clone () from /lib64/libc.so.6{color}
{color:#de350b}Thread 45 (Thread 0x7f32777fe640 (LWP 36488)):{color}
{color:#de350b}#0 0x00007f33186a7ea0 in __lll_lock_wait () from
/lib64/libpthread.so.0{color}
{color:#de350b}#1 0x00007f33186a08f5 in pthread_mutex_lock () from
/lib64/libpthread.so.0{color}
{color:#de350b}#2 0x00007f33186dfc5f in lock (m=0x10edc90) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326{color}
{color:#de350b}#3 pni_raw_connection_done (rc=0x10ed3b8) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll_raw_connection.c:423{color}
{color:#de350b}#4 pn_proactor_done (batch=0x10ed970, p=0x10ed970) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2696{color}
{color:#de350b}#5 pn_proactor_done (p=0x10ed970,
batch=batch@entry=0x7f32e4080b78) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2676{color}
{color:#de350b}#6 0x00007f33187c1a11 in thread_run (arg=0x10f6e40) at
/home/mick/latest/qpid-dispatch/src/server.c:1140{color}
{color:#de350b}#7 0x00007f331869e3f9 in start_thread () from
/lib64/libpthread.so.0{color}
{color:#de350b}#8 0x00007f33181b2b53 in clone () from /lib64/libc.so.6{color}
{color:#de350b}Thread 44 (Thread 0x7f327cff9640 (LWP 36487)):{color}
{color:#de350b}#0 0x00007f33186a7ea0 in __lll_lock_wait () from
/lib64/libpthread.so.0{color}
{color:#de350b}#1 0x00007f33186a08f5 in pthread_mutex_lock () from
/lib64/libpthread.so.0{color}
{color:#de350b}#2 0x00007f33186e2b7e in lock (m=0x10edc90) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326{color}
{color:#de350b}#3 process (tsk=<optimized out>) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2248{color}
{color:#de350b}#4 next_event_batch (p=<optimized out>, can_block=true) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2423{color}
{color:#de350b}#5 0x00007f33187c192f in thread_run (arg=0x10f6e40) at
/home/mick/latest/qpid-dispatch/src/server.c:1107{color}
{color:#de350b}#6 0x00007f331869e3f9 in start_thread () from
/lib64/libpthread.so.0{color}
{color:#de350b}#7 0x00007f33181b2b53 in clone () from /lib64/libc.so.6{color}
{color:#de350b}Thread 43 (Thread 0x7f32a17fa640 (LWP 36478)):{color}
{color:#de350b}#0 0x00007f33186a7ea0 in __lll_lock_wait () from
/lib64/libpthread.so.0{color}
{color:#de350b}#1 0x00007f33186a08f5 in pthread_mutex_lock () from
/lib64/libpthread.so.0{color}
{color:#de350b}#2 0x00007f33186e2b7e in lock (m=0x10edc90) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326{color}
{color:#de350b}#3 process (tsk=<optimized out>) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2248{color}
{color:#de350b}#4 next_event_batch (p=<optimized out>, can_block=true) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2423{color}
{color:#de350b}#5 0x00007f33187c192f in thread_run (arg=0x10f6e40) at
/home/mick/latest/qpid-dispatch/src/server.c:1107{color}
{color:#de350b}#6 0x00007f331869e3f9 in start_thread () from
/lib64/libpthread.so.0{color}
{color:#de350b}#7 0x00007f33181b2b53 in clone () from /lib64/libc.so.6{color}
{color:#de350b}Thread 42 (Thread 0x7f327effd640 (LWP 36483)):{color}
{color:#de350b}#0 0x00007f33186a7ea0 in __lll_lock_wait () from
/lib64/libpthread.so.0{color}
{color:#de350b}#1 0x00007f33186a08f5 in pthread_mutex_lock () from
/lib64/libpthread.so.0{color}
{color:#de350b}#2 0x00007f33186e2b7e in lock (m=0x10edc90) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326{color}
{color:#de350b}#3 process (tsk=<optimized out>) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2248{color}
{color:#de350b}#4 next_event_batch (p=<optimized out>, can_block=true) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2423{color}
{color:#de350b}#5 0x00007f33187c192f in thread_run (arg=0x10f6e40) at
/home/mick/latest/qpid-dispatch/src/server.c:1107{color}
{color:#de350b}#6 0x00007f331869e3f9 in start_thread () from
/lib64/libpthread.so.0{color}
{color:#de350b}#7 0x00007f33181b2b53 in clone () from /lib64/libc.so.6{color}
{color:#de350b}Thread 41 (Thread 0x7f327f7fe640 (LWP 36482)):{color}
{color:#de350b}#0 0x00007f33186a7ea0 in __lll_lock_wait () from
/lib64/libpthread.so.0{color}
{color:#de350b}#1 0x00007f33186a08f5 in pthread_mutex_lock () from
/lib64/libpthread.so.0{color}
{color:#de350b}#2 0x00007f33186e2b7e in lock (m=0x10edc90) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326{color}
{color:#de350b}#3 process (tsk=<optimized out>) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2248{color}
{color:#de350b}#4 next_event_batch (p=<optimized out>, can_block=true) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2423{color}
{color:#de350b}#5 0x00007f33187c192f in thread_run (arg=0x10f6e40) at
/home/mick/latest/qpid-dispatch/src/server.c:1107{color}
{color:#de350b}#6 0x00007f331869e3f9 in start_thread () from
/lib64/libpthread.so.0{color}
{color:#de350b}#7 0x00007f33181b2b53 in clone () from /lib64/libc.so.6{color}
{color:#de350b}Thread 40 (Thread 0x7f32d37fe640 (LWP 36460)):{color}
{color:#de350b}#0 0x00007f33186a7ea0 in __lll_lock_wait () from
/lib64/libpthread.so.0{color}
{color:#de350b}#1 0x00007f33186a08f5 in pthread_mutex_lock () from
/lib64/libpthread.so.0{color}
{color:#de350b}#2 0x00007f33186e2509 in lock (m=0x10edc90) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326{color}
{color:#de350b}#3 next_event_batch (p=0x10ed970, can_block=true) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2411{color}
{color:#de350b}#4 0x00007f33187c192f in thread_run (arg=0x10f6e40) at
/home/mick/latest/qpid-dispatch/src/server.c:1107{color}
{color:#de350b}#5 0x00007f331869e3f9 in start_thread () from
/lib64/libpthread.so.0{color}
{color:#de350b}#6 0x00007f33181b2b53 in clone () from /lib64/libc.so.6{color}
{color:#de350b}Thread 39 (Thread 0x7f32daffd640 (LWP 36462)):{color}
{color:#de350b}#0 0x00007f33186a7ea0 in __lll_lock_wait () from
/lib64/libpthread.so.0{color}
{color:#de350b}#1 0x00007f33186a08f5 in pthread_mutex_lock () from
/lib64/libpthread.so.0{color}
{color:#de350b}#2 0x00007f33186e2509 in lock (m=0x10edc90) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326{color}
{color:#de350b}#3 next_event_batch (p=0x10ed970, can_block=true) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2411{color}
{color:#de350b}#4 0x00007f33187c192f in thread_run (arg=0x10f6e40) at
/home/mick/latest/qpid-dispatch/src/server.c:1107{color}
{color:#de350b}#5 0x00007f331869e3f9 in start_thread () from
/lib64/libpthread.so.0{color}
{color:#de350b}#6 0x00007f33181b2b53 in clone () from /lib64/libc.so.6{color}
{color:#de350b}Thread 38 (Thread 0x7f3277fff640 (LWP 36480)):{color}
{color:#de350b}#0 0x00007f33186a7ea0 in __lll_lock_wait () from
/lib64/libpthread.so.0{color}
{color:#de350b}#1 0x00007f33186a08f5 in pthread_mutex_lock () from
/lib64/libpthread.so.0{color}
{color:#de350b}#2 0x00007f33186e2b7e in lock (m=0x10edc90) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326{color}
{color:#de350b}#3 process (tsk=<optimized out>) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2248{color}
{color:#de350b}#4 next_event_batch (p=<optimized out>, can_block=true) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2423{color}
{color:#de350b}#5 0x00007f33187c192f in thread_run (arg=0x10f6e40) at
/home/mick/latest/qpid-dispatch/src/server.c:1107{color}
{color:#de350b}#6 0x00007f331869e3f9 in start_thread () from
/lib64/libpthread.so.0{color}
{color:#de350b}#7 0x00007f33181b2b53 in clone () from /lib64/libc.so.6{color}
{color:#de350b}Thread 37 (Thread 0x7f32a0ff9640 (LWP 36479)):{color}
{color:#de350b}#0 0x00007f33186a7ea0 in __lll_lock_wait () from
/lib64/libpthread.so.0{color}
{color:#de350b}#1 0x00007f33186a08f5 in pthread_mutex_lock () from
/lib64/libpthread.so.0{color}
{color:#de350b}#2 0x00007f33186dfc5f in lock (m=0x10edc90) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326{color}
{color:#de350b}#3 pni_raw_connection_done (rc=0x10ed3b8) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll_raw_connection.c:423{color}
{color:#de350b}#4 pn_proactor_done (batch=0x10ed970, p=0x10ed970) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2696{color}
{color:#de350b}#5 pn_proactor_done (p=0x10ed970,
batch=batch@entry=0x7f328407b3b8) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2676{color}
{color:#de350b}#6 0x00007f33187c1a11 in thread_run (arg=0x10f6e40) at
/home/mick/latest/qpid-dispatch/src/server.c:1140{color}
{color:#de350b}#7 0x00007f331869e3f9 in start_thread () from
/lib64/libpthread.so.0{color}
{color:#de350b}#8 0x00007f33181b2b53 in clone () from /lib64/libc.so.6{color}
{color:#de350b}Thread 36 (Thread 0x7f32d0ff9640 (LWP 36472)):{color}
{color:#de350b}#0 0x00007f33186a7ea0 in __lll_lock_wait () from
/lib64/libpthread.so.0{color}
{color:#de350b}#1 0x00007f33186a08f5 in pthread_mutex_lock () from
/lib64/libpthread.so.0{color}
{color:#de350b}#2 0x00007f33186e2b7e in lock (m=0x10edc90) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326{color}
{color:#de350b}#3 process (tsk=<optimized out>) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2248{color}
{color:#de350b}#4 next_event_batch (p=<optimized out>, can_block=true) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2423{color}
{color:#de350b}#5 0x00007f33187c192f in thread_run (arg=0x10f6e40) at
/home/mick/latest/qpid-dispatch/src/server.c:1107{color}
{color:#de350b}#6 0x00007f331869e3f9 in start_thread () from
/lib64/libpthread.so.0{color}
{color:#de350b}#7 0x00007f33181b2b53 in clone () from /lib64/libc.so.6{color}
{color:#de350b}Thread 35 (Thread 0x7f32a1ffb640 (LWP 36477)):{color}
{color:#de350b}#0 0x00007f33186a7ea0 in __lll_lock_wait () from
/lib64/libpthread.so.0{color}
{color:#de350b}#1 0x00007f33186a08f5 in pthread_mutex_lock () from
/lib64/libpthread.so.0{color}
{color:#de350b}#2 0x00007f33186dfc5f in lock (m=0x10edc90) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326{color}
{color:#de350b}#3 pni_raw_connection_done (rc=0x10ed3b8) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll_raw_connection.c:423{color}
{color:#de350b}#4 pn_proactor_done (batch=0x10ed970, p=0x10ed970) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2696{color}
{color:#de350b}#5 pn_proactor_done (p=0x10ed970,
batch=batch@entry=0x7f31fc1293f8) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2676{color}
{color:#de350b}#6 0x00007f33187c1a11 in thread_run (arg=0x10f6e40) at
/home/mick/latest/qpid-dispatch/src/server.c:1140{color}
{color:#de350b}#7 0x00007f331869e3f9 in start_thread () from
/lib64/libpthread.so.0{color}
{color:#de350b}#8 0x00007f33181b2b53 in clone () from /lib64/libc.so.6{color}
{color:#de350b}Thread 34 (Thread 0x7f32a2ffd640 (LWP 36475)):{color}
{color:#de350b}#0 0x00007f33186a7ea0 in __lll_lock_wait () from
/lib64/libpthread.so.0{color}
{color:#de350b}#1 0x00007f33186a08f5 in pthread_mutex_lock () from
/lib64/libpthread.so.0{color}
{color:#de350b}#2 0x00007f33186dfc5f in lock (m=0x10edc90) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326{color}
{color:#de350b}#3 pni_raw_connection_done (rc=0x10ed3b8) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll_raw_connection.c:423{color}
{color:#de350b}#4 pn_proactor_done (batch=0x10ed970, p=0x10ed970) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2696{color}
{color:#de350b}#5 pn_proactor_done (p=0x10ed970,
batch=batch@entry=0x7f32b404a638) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2676{color}
{color:#de350b}#6 0x00007f33187c1a11 in thread_run (arg=0x10f6e40) at
/home/mick/latest/qpid-dispatch/src/server.c:1140{color}
{color:#de350b}#7 0x00007f331869e3f9 in start_thread () from
/lib64/libpthread.so.0{color}
{color:#de350b}#8 0x00007f33181b2b53 in clone () from /lib64/libc.so.6{color}
{color:#de350b}Thread 33 (Thread 0x7f32a37fe640 (LWP 36474)):{color}
{color:#de350b}#0 0x00007f33186a7ea0 in __lll_lock_wait () from
/lib64/libpthread.so.0{color}
{color:#de350b}#1 0x00007f33186a08f5 in pthread_mutex_lock () from
/lib64/libpthread.so.0{color}
{color:#de350b}#2 0x00007f33186e2b7e in lock (m=0x10edc90) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326{color}
{color:#de350b}#3 process (tsk=<optimized out>) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2248{color}
{color:#de350b}#4 next_event_batch (p=<optimized out>, can_block=true) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2423{color}
{color:#de350b}#5 0x00007f33187c192f in thread_run (arg=0x10f6e40) at
/home/mick/latest/qpid-dispatch/src/server.c:1107{color}
{color:#de350b}#6 0x00007f331869e3f9 in start_thread () from
/lib64/libpthread.so.0{color}
{color:#de350b}#7 0x00007f33181b2b53 in clone () from /lib64/libc.so.6{color}
{color:#de350b}Thread 32 (Thread 0x7f32a27fc640 (LWP 36476)):{color}
{color:#de350b}#0 0x00007f33186a7ea0 in __lll_lock_wait () from
/lib64/libpthread.so.0{color}
{color:#de350b}#1 0x00007f33186a08f5 in pthread_mutex_lock () from
/lib64/libpthread.so.0{color}
{color:#de350b}#2 0x00007f33186e2b7e in lock (m=0x10edc90) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326{color}
{color:#de350b}#3 process (tsk=<optimized out>) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2248{color}
{color:#de350b}#4 next_event_batch (p=<optimized out>, can_block=true) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2423{color}
{color:#de350b}#5 0x00007f33187c192f in thread_run (arg=0x10f6e40) at
/home/mick/latest/qpid-dispatch/src/server.c:1107{color}
{color:#de350b}#6 0x00007f331869e3f9 in start_thread () from
/lib64/libpthread.so.0{color}
{color:#de350b}#7 0x00007f33181b2b53 in clone () from /lib64/libc.so.6{color}
{color:#de350b}Thread 31 (Thread 0x7f32a3fff640 (LWP 36473)):{color}
{color:#de350b}#0 0x00007f33186a7ea0 in __lll_lock_wait () from
/lib64/libpthread.so.0{color}
{color:#de350b}#1 0x00007f33186a08f5 in pthread_mutex_lock () from
/lib64/libpthread.so.0{color}
{color:#de350b}#2 0x00007f33186e2b7e in lock (m=0x10edc90) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326{color}
{color:#de350b}#3 process (tsk=<optimized out>) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2248{color}
{color:#de350b}#4 next_event_batch (p=<optimized out>, can_block=true) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2423{color}
{color:#de350b}#5 0x00007f33187c192f in thread_run (arg=0x10f6e40) at
/home/mick/latest/qpid-dispatch/src/server.c:1107{color}
{color:#de350b}#6 0x00007f331869e3f9 in start_thread () from
/lib64/libpthread.so.0{color}
{color:#de350b}#7 0x00007f33181b2b53 in clone () from /lib64/libc.so.6{color}
{color:#de350b}Thread 30 (Thread 0x7f32d27fc640 (LWP 36469)):{color}
{color:#de350b}#0 0x00007f33186a7ea0 in __lll_lock_wait () from
/lib64/libpthread.so.0{color}
{color:#de350b}#1 0x00007f33186a08f5 in pthread_mutex_lock () from
/lib64/libpthread.so.0{color}
{color:#de350b}#2 0x00007f33186dfc5f in lock (m=0x10edc90) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326{color}
{color:#de350b}#3 pni_raw_connection_done (rc=0x10ed3b8) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll_raw_connection.c:423{color}
{color:#de350b}#4 pn_proactor_done (batch=0x10ed970, p=0x10ed970) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2696{color}
{color:#de350b}#5 pn_proactor_done (p=0x10ed970,
batch=batch@entry=0x7f329c0828d8) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2676{color}
{color:#de350b}#6 0x00007f33187c1a11 in thread_run (arg=0x10f6e40) at
/home/mick/latest/qpid-dispatch/src/server.c:1140{color}
{color:#de350b}#7 0x00007f331869e3f9 in start_thread () from
/lib64/libpthread.so.0{color}
{color:#de350b}#8 0x00007f33181b2b53 in clone () from /lib64/libc.so.6{color}
{color:#de350b}Thread 29 (Thread 0x7f33017fa640 (LWP 36451)):{color}
{color:#de350b}#0 0x00007f33186a7ea0 in __lll_lock_wait () from
/lib64/libpthread.so.0{color}
{color:#de350b}#1 0x00007f33186a08f5 in pthread_mutex_lock () from
/lib64/libpthread.so.0{color}
{color:#de350b}#2 0x00007f33186e2b7e in lock (m=0x10edc90) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326{color}
{color:#de350b}#3 process (tsk=<optimized out>) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2248{color}
{color:#de350b}#4 next_event_batch (p=<optimized out>, can_block=true) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2423{color}
{color:#de350b}#5 0x00007f33187c192f in thread_run (arg=0x10f6e40) at
/home/mick/latest/qpid-dispatch/src/server.c:1107{color}
{color:#de350b}#6 0x00007f331869e3f9 in start_thread () from
/lib64/libpthread.so.0{color}
{color:#de350b}#7 0x00007f33181b2b53 in clone () from /lib64/libc.so.6{color}
{color:#de350b}Thread 28 (Thread 0x7f32d1ffb640 (LWP 36470)):{color}
{color:#de350b}#0 0x00007f33186a7ea0 in __lll_lock_wait () from
/lib64/libpthread.so.0{color}
{color:#de350b}#1 0x00007f33186a08f5 in pthread_mutex_lock () from
/lib64/libpthread.so.0{color}
{color:#de350b}#2 0x00007f33186e2b7e in lock (m=0x10edc90) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326{color}
{color:#de350b}#3 process (tsk=<optimized out>) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2248{color}
{color:#de350b}#4 next_event_batch (p=<optimized out>, can_block=true) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2423{color}
{color:#de350b}#5 0x00007f33187c192f in thread_run (arg=0x10f6e40) at
/home/mick/latest/qpid-dispatch/src/server.c:1107{color}
{color:#de350b}#6 0x00007f331869e3f9 in start_thread () from
/lib64/libpthread.so.0{color}
{color:#de350b}#7 0x00007f33181b2b53 in clone () from /lib64/libc.so.6{color}
{color:#de350b}Thread 27 (Thread 0x7f32d8ff9640 (LWP 36466)):{color}
{color:#de350b}#0 0x00007f33186a7ea0 in __lll_lock_wait () from
/lib64/libpthread.so.0{color}
{color:#de350b}#1 0x00007f33186a08f5 in pthread_mutex_lock () from
/lib64/libpthread.so.0{color}
{color:#de350b}#2 0x00007f33186e2b7e in lock (m=0x10edc90) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326{color}
{color:#de350b}#3 process (tsk=<optimized out>) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2248{color}
{color:#de350b}#4 next_event_batch (p=<optimized out>, can_block=true) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2423{color}
{color:#de350b}#5 0x00007f33187c192f in thread_run (arg=0x10f6e40) at
/home/mick/latest/qpid-dispatch/src/server.c:1107{color}
{color:#de350b}#6 0x00007f331869e3f9 in start_thread () from
/lib64/libpthread.so.0{color}
{color:#de350b}#7 0x00007f33181b2b53 in clone () from /lib64/libc.so.6{color}
{color:#de350b}Thread 26 (Thread 0x7f32d3fff640 (LWP 36467)):{color}
{color:#de350b}#0 0x00007f33186a7ea0 in __lll_lock_wait () from
/lib64/libpthread.so.0{color}
{color:#de350b}#1 0x00007f33186a08f5 in pthread_mutex_lock () from
/lib64/libpthread.so.0{color}
{color:#de350b}#2 0x00007f33186dfc5f in lock (m=0x10edc90) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326{color}
{color:#de350b}#3 pni_raw_connection_done (rc=0x10ed3b8) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll_raw_connection.c:423{color}
{color:#de350b}#4 pn_proactor_done (batch=0x10ed970, p=0x10ed970) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2696{color}
{color:#de350b}#5 pn_proactor_done (p=0x10ed970,
batch=batch@entry=0x7f3238096088) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2676{color}
{color:#de350b}#6 0x00007f33187c1a11 in thread_run (arg=0x10f6e40) at
/home/mick/latest/qpid-dispatch/src/server.c:1140{color}
{color:#de350b}#7 0x00007f331869e3f9 in start_thread () from
/lib64/libpthread.so.0{color}
{color:#de350b}#8 0x00007f33181b2b53 in clone () from /lib64/libc.so.6{color}
{color:#de350b}Thread 25 (Thread 0x7f32fbfff640 (LWP 36453)):{color}
{color:#de350b}#0 0x00007f33186a7ea0 in __lll_lock_wait () from
/lib64/libpthread.so.0{color}
{color:#de350b}#1 0x00007f33186a08f5 in pthread_mutex_lock () from
/lib64/libpthread.so.0{color}
{color:#de350b}#2 0x00007f33186e2b7e in lock (m=0x10edc90) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326{color}
{color:#de350b}#3 process (tsk=<optimized out>) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2248{color}
{color:#de350b}#4 next_event_batch (p=<optimized out>, can_block=true) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2423{color}
{color:#de350b}#5 0x00007f33187c192f in thread_run (arg=0x10f6e40) at
/home/mick/latest/qpid-dispatch/src/server.c:1107{color}
{color:#de350b}#6 0x00007f331869e3f9 in start_thread () from
/lib64/libpthread.so.0{color}
{color:#de350b}#7 0x00007f33181b2b53 in clone () from /lib64/libc.so.6{color}
{color:#de350b}Thread 24 (Thread 0x7f3246ffd640 (LWP 36496)):{color}
{color:#de350b}#0 0x00007f33186a7ea0 in __lll_lock_wait () from
/lib64/libpthread.so.0{color}
{color:#de350b}#1 0x00007f33186a08f5 in pthread_mutex_lock () from
/lib64/libpthread.so.0{color}
{color:#de350b}#2 0x00007f33186e2509 in lock (m=0x10edc90) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326{color}
{color:#de350b}#3 next_event_batch (p=0x10ed970, can_block=true) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2411{color}
{color:#de350b}#4 0x00007f33187c192f in thread_run (arg=0x10f6e40) at
/home/mick/latest/qpid-dispatch/src/server.c:1107{color}
{color:#de350b}#5 0x00007f331869e3f9 in start_thread () from
/lib64/libpthread.so.0{color}
{color:#de350b}#6 0x00007f33181b2b53 in clone () from /lib64/libc.so.6{color}
{color:#de350b}Thread 23 (Thread 0x7f32fb7fe640 (LWP 36454)):{color}
{color:#de350b}#0 0x00007f33186a7ea0 in __lll_lock_wait () from
/lib64/libpthread.so.0{color}
{color:#de350b}#1 0x00007f33186a08f5 in pthread_mutex_lock () from
/lib64/libpthread.so.0{color}
{color:#de350b}#2 0x00007f33186e2b7e in lock (m=0x10edc90) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326{color}
{color:#de350b}#3 process (tsk=<optimized out>) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2248{color}
{color:#de350b}#4 next_event_batch (p=<optimized out>, can_block=true) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2423{color}
{color:#de350b}#5 0x00007f33187c192f in thread_run (arg=0x10f6e40) at
/home/mick/latest/qpid-dispatch/src/server.c:1107{color}
{color:#de350b}#6 0x00007f331869e3f9 in start_thread () from
/lib64/libpthread.so.0{color}
{color:#de350b}#7 0x00007f33181b2b53 in clone () from /lib64/libc.so.6{color}
{color:#de350b}Thread 22 (Thread 0x7f322ffff640 (LWP 36501)):{color}
{color:#de350b}#0 0x00007f33186a7ea0 in __lll_lock_wait () from
/lib64/libpthread.so.0{color}
{color:#de350b}#1 0x00007f33186a08f5 in pthread_mutex_lock () from
/lib64/libpthread.so.0{color}
{color:#de350b}#2 0x00007f33186dfc5f in lock (m=0x10edc90) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326{color}
{color:#de350b}#3 pni_raw_connection_done (rc=0x10ed3b8) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll_raw_connection.c:423{color}
{color:#de350b}#4 pn_proactor_done (batch=0x10ed970, p=0x10ed970) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2696{color}
{color:#de350b}#5 pn_proactor_done (p=0x10ed970,
batch=batch@entry=0x7f32180ef358) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2676{color}
{color:#de350b}#6 0x00007f33187c1a11 in thread_run (arg=0x10f6e40) at
/home/mick/latest/qpid-dispatch/src/server.c:1140{color}
{color:#de350b}#7 0x00007f331869e3f9 in start_thread () from
/lib64/libpthread.so.0{color}
{color:#de350b}#8 0x00007f33181b2b53 in clone () from /lib64/libc.so.6{color}
{color:#de350b}Thread 21 (Thread 0x7f32d97fa640 (LWP 36465)):{color}
{color:#de350b}#0 0x00007f33186a7ea0 in __lll_lock_wait () from
/lib64/libpthread.so.0{color}
{color:#de350b}#1 0x00007f33186a08f5 in pthread_mutex_lock () from
/lib64/libpthread.so.0{color}
{color:#de350b}#2 0x00007f33186e2b7e in lock (m=0x10edc90) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326{color}
{color:#de350b}#3 process (tsk=<optimized out>) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2248{color}
{color:#de350b}#4 next_event_batch (p=<optimized out>, can_block=true) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2423{color}
{color:#de350b}#5 0x00007f33187c192f in thread_run (arg=0x10f6e40) at
/home/mick/latest/qpid-dispatch/src/server.c:1107{color}
{color:#de350b}#6 0x00007f331869e3f9 in start_thread () from
/lib64/libpthread.so.0{color}
{color:#de350b}#7 0x00007f33181b2b53 in clone () from /lib64/libc.so.6{color}
{color:#de350b}Thread 20 (Thread 0x7f33027fc640 (LWP 36449)):{color}
{color:#de350b}#0 0x00007f33186a7ea0 in __lll_lock_wait () from
/lib64/libpthread.so.0{color}
{color:#de350b}#1 0x00007f33186a08f5 in pthread_mutex_lock () from
/lib64/libpthread.so.0{color}
{color:#de350b}#2 0x00007f33186e2509 in lock (m=0x10edc90) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326{color}
{color:#de350b}#3 next_event_batch (p=0x10ed970, can_block=true) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2411{color}
{color:#de350b}#4 0x00007f33187c192f in thread_run (arg=0x10f6e40) at
/home/mick/latest/qpid-dispatch/src/server.c:1107{color}
{color:#de350b}#5 0x00007f331869e3f9 in start_thread () from
/lib64/libpthread.so.0{color}
{color:#de350b}#6 0x00007f33181b2b53 in clone () from /lib64/libc.so.6{color}
{color:#de350b}Thread 19 (Thread 0x7f33037fe640 (LWP 36446)):{color}
{color:#de350b}#0 0x00007f33186a7ea0 in __lll_lock_wait () from
/lib64/libpthread.so.0{color}
{color:#de350b}#1 0x00007f33186a08f5 in pthread_mutex_lock () from
/lib64/libpthread.so.0{color}
{color:#de350b}#2 0x00007f33186e2509 in lock (m=0x10edc90) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326{color}
{color:#de350b}#3 next_event_batch (p=0x10ed970, can_block=true) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2411{color}
{color:#de350b}#4 0x00007f33187c192f in thread_run (arg=0x10f6e40) at
/home/mick/latest/qpid-dispatch/src/server.c:1107{color}
{color:#de350b}#5 0x00007f331869e3f9 in start_thread () from
/lib64/libpthread.so.0{color}
{color:#de350b}#6 0x00007f33181b2b53 in clone () from /lib64/libc.so.6{color}
{color:#de350b}Thread 18 (Thread 0x7f32db7fe640 (LWP 36461)):{color}
{color:#de350b}#0 0x00007f33186a7ea0 in __lll_lock_wait () from
/lib64/libpthread.so.0{color}
{color:#de350b}#1 0x00007f33186a08f5 in pthread_mutex_lock () from
/lib64/libpthread.so.0{color}
{color:#de350b}#2 0x00007f33186e2b7e in lock (m=0x10edc90) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326{color}
{color:#de350b}#3 process (tsk=<optimized out>) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2248{color}
{color:#de350b}#4 next_event_batch (p=<optimized out>, can_block=true) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2423{color}
{color:#de350b}#5 0x00007f33187c192f in thread_run (arg=0x10f6e40) at
/home/mick/latest/qpid-dispatch/src/server.c:1107{color}
{color:#de350b}#6 0x00007f331869e3f9 in start_thread () from
/lib64/libpthread.so.0{color}
{color:#de350b}#7 0x00007f33181b2b53 in clone () from /lib64/libc.so.6{color}
{color:#de350b}Thread 17 (Thread 0x7f32d9ffb640 (LWP 36464)):{color}
{color:#de350b}#0 0x00007f33186a7ea0 in __lll_lock_wait () from
/lib64/libpthread.so.0{color}
{color:#de350b}#1 0x00007f33186a08f5 in pthread_mutex_lock () from
/lib64/libpthread.so.0{color}
{color:#de350b}#2 0x00007f33186e2d79 in lock (m=0x10edc90) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326{color}
{color:#de350b}#3 suspend (ts=<optimized out>, p=<optimized out>) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:397{color}
{color:#de350b}#4 next_event_batch (p=0x10ed970, can_block=true) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2459{color}
{color:#de350b}#5 0x00007f33187c192f in thread_run (arg=0x10f6e40) at
/home/mick/latest/qpid-dispatch/src/server.c:1107{color}
{color:#de350b}#6 0x00007f331869e3f9 in start_thread () from
/lib64/libpthread.so.0{color}
{color:#de350b}#7 0x00007f33181b2b53 in clone () from /lib64/libc.so.6{color}
{color:#de350b}Thread 16 (Thread 0x7f3309369640 (LWP 36443)):{color}
{color:#de350b}#0 0x00007f33186a0687 in pthread_mutex_lock () from
/lib64/libpthread.so.0{color}
{color:#de350b}#1 0x00007f331878f219 in sys_mutex_lock (mutex=<optimized out>)
at /home/mick/latest/qpid-dispatch/src/posix/threading.c:57{color}
{color:#de350b}#2 0x00007f33187a0ec2 in qdr_delivery_continue_peers_CT
(core=core@entry=0x118e8b0, in_dlv=in_dlv@entry=0x7f325805ea48,
more=more@entry=true) at
/home/mick/latest/qpid-dispatch/src/router_core/delivery.c:1062{color}
{color:#de350b}#3 0x00007f33187a1bb7 in qdr_delivery_continue_CT
(action=<optimized out>, action=<optimized out>, discard=false, core=0x118e8b0)
at /home/mick/latest/qpid-dispatch/src/router_core/delivery.c:1126{color}
{color:#de350b}#4 qdr_delivery_continue_CT (core=0x118e8b0, action=<optimized
out>, discard=<optimized out>) at
/home/mick/latest/qpid-dispatch/src/router_core/delivery.c:1103{color}
{color:#de350b}#5 0x00007f33187ac2ed in router_core_thread (arg=0x118e8b0) at
/home/mick/latest/qpid-dispatch/src/router_core/router_core_thread.c:239{color}
{color:#de350b}#6 0x00007f331869e3f9 in start_thread () from
/lib64/libpthread.so.0{color}
{color:#de350b}#7 0x00007f33181b2b53 in clone () from /lib64/libc.so.6{color}
{color:#de350b}Thread 15 (Thread 0x7f32dbfff640 (LWP 36459)):{color}
{color:#de350b}#0 0x00007f33186a7ea0 in __lll_lock_wait () from
/lib64/libpthread.so.0{color}
{color:#de350b}#1 0x00007f33186a08f5 in pthread_mutex_lock () from
/lib64/libpthread.so.0{color}
{color:#de350b}#2 0x00007f33186dd6d0 in lock (m=0x10edc90) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326{color}
{color:#de350b}#3 poller_done (p=p@entry=0x10ed970, ts=ts@entry=0x7f32b0000b60)
at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2649{color}
{color:#de350b}#4 0x00007f33186e327a in next_event_batch (p=0x10ed970,
can_block=true) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2452{color}
{color:#de350b}#5 0x00007f33187c192f in thread_run (arg=0x10f6e40) at
/home/mick/latest/qpid-dispatch/src/server.c:1107{color}
{color:#de350b}#6 0x00007f331869e3f9 in start_thread () from
/lib64/libpthread.so.0{color}
{color:#de350b}#7 0x00007f33181b2b53 in clone () from /lib64/libc.so.6{color}
{color:#de350b}Thread 14 (Thread 0x7f32f8ff9640 (LWP 36458)):{color}
{color:#de350b}#0 0x00007f33186a7ea0 in __lll_lock_wait () from
/lib64/libpthread.so.0{color}
{color:#de350b}#1 0x00007f33186a08f5 in pthread_mutex_lock () from
/lib64/libpthread.so.0{color}
{color:#de350b}#2 0x00007f33186e2b7e in lock (m=0x10edc90) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326{color}
{color:#de350b}#3 process (tsk=<optimized out>) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2248{color}
{color:#de350b}#4 next_event_batch (p=<optimized out>, can_block=true) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2423{color}
{color:#de350b}#5 0x00007f33187c192f in thread_run (arg=0x10f6e40) at
/home/mick/latest/qpid-dispatch/src/server.c:1107{color}
{color:#de350b}#6 0x00007f331869e3f9 in start_thread () from
/lib64/libpthread.so.0{color}
{color:#de350b}#7 0x00007f33181b2b53 in clone () from /lib64/libc.so.6{color}
{color:#de350b}Thread 13 (Thread 0x7f32fa7fc640 (LWP 36455)):{color}
{color:#de350b}#0 0x00007f33186a7ea0 in __lll_lock_wait () from
/lib64/libpthread.so.0{color}
{color:#de350b}#1 0x00007f33186a08f5 in pthread_mutex_lock () from
/lib64/libpthread.so.0{color}
{color:#de350b}#2 0x00007f33186e2b7e in lock (m=0x10edc90) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326{color}
{color:#de350b}#3 process (tsk=<optimized out>) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2248{color}
{color:#de350b}#4 next_event_batch (p=<optimized out>, can_block=true) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2423{color}
{color:#de350b}#5 0x00007f33187c192f in thread_run (arg=0x10f6e40) at
/home/mick/latest/qpid-dispatch/src/server.c:1107{color}
{color:#de350b}#6 0x00007f331869e3f9 in start_thread () from
/lib64/libpthread.so.0{color}
{color:#de350b}#7 0x00007f33181b2b53 in clone () from /lib64/libc.so.6{color}
{color:#de350b}Thread 12 (Thread 0x7f32f97fa640 (LWP 36457)):{color}
{color:#de350b}#0 0x00007f33186a7ea0 in __lll_lock_wait () from
/lib64/libpthread.so.0{color}
{color:#de350b}#1 0x00007f33186a08f5 in pthread_mutex_lock () from
/lib64/libpthread.so.0{color}
{color:#de350b}#2 0x00007f33186e2b7e in lock (m=0x10edc90) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326{color}
{color:#de350b}#3 process (tsk=<optimized out>) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2248{color}
{color:#de350b}#4 next_event_batch (p=<optimized out>, can_block=true) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2423{color}
{color:#de350b}#5 0x00007f33187c192f in thread_run (arg=0x10f6e40) at
/home/mick/latest/qpid-dispatch/src/server.c:1107{color}
{color:#de350b}#6 0x00007f331869e3f9 in start_thread () from
/lib64/libpthread.so.0{color}
{color:#de350b}#7 0x00007f33181b2b53 in clone () from /lib64/libc.so.6{color}
{color:#de350b}Thread 11 (Thread 0x7f32f9ffb640 (LWP 36456)):{color}
{color:#de350b}#0 0x00007f33186a7ea0 in __lll_lock_wait () from
/lib64/libpthread.so.0{color}
{color:#de350b}#1 0x00007f33186a08f5 in pthread_mutex_lock () from
/lib64/libpthread.so.0{color}
{color:#de350b}#2 0x00007f33186dfc5f in lock (m=0x10edc90) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326{color}
{color:#de350b}#3 pni_raw_connection_done (rc=0x10ed3b8) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll_raw_connection.c:423{color}
{color:#de350b}#4 pn_proactor_done (batch=0x10ed970, p=0x10ed970) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2696{color}
{color:#de350b}#5 pn_proactor_done (p=0x10ed970,
batch=batch@entry=0x7f32180eed48) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2676{color}
{color:#de350b}#6 0x00007f33187c1a11 in thread_run (arg=0x10f6e40) at
/home/mick/latest/qpid-dispatch/src/server.c:1140{color}
{color:#de350b}#7 0x00007f331869e3f9 in start_thread () from
/lib64/libpthread.so.0{color}
{color:#de350b}#8 0x00007f33181b2b53 in clone () from /lib64/libc.so.6{color}
{color:#de350b}Thread 10 (Thread 0x7f3301ffb640 (LWP 36450)):{color}
{color:#de350b}#0 0x00007f33186a7ea0 in __lll_lock_wait () from
/lib64/libpthread.so.0{color}
{color:#de350b}#1 0x00007f33186a08f5 in pthread_mutex_lock () from
/lib64/libpthread.so.0{color}
{color:#de350b}#2 0x00007f33186e2509 in lock (m=0x10edc90) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326{color}
{color:#de350b}#3 next_event_batch (p=0x10ed970, can_block=true) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2411{color}
{color:#de350b}#4 0x00007f33187c192f in thread_run (arg=0x10f6e40) at
/home/mick/latest/qpid-dispatch/src/server.c:1107{color}
{color:#de350b}#5 0x00007f331869e3f9 in start_thread () from
/lib64/libpthread.so.0{color}
{color:#de350b}#6 0x00007f33181b2b53 in clone () from /lib64/libc.so.6{color}
{color:#de350b}Thread 9 (Thread 0x7f3300ff9640 (LWP 36452)):{color}
{color:#de350b}#0 0x00007f33186a7ea0 in __lll_lock_wait () from
/lib64/libpthread.so.0{color}
{color:#de350b}#1 0x00007f33186a08f5 in pthread_mutex_lock () from
/lib64/libpthread.so.0{color}
{color:#de350b}#2 0x00007f33186dfc5f in lock (m=0x10edc90) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326{color}
{color:#de350b}#3 pni_raw_connection_done (rc=0x10ed3b8) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll_raw_connection.c:423{color}
{color:#de350b}#4 pn_proactor_done (batch=0x10ed970, p=0x10ed970) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2696{color}
{color:#de350b}#5 pn_proactor_done (p=0x10ed970,
batch=batch@entry=0x7f3204155058) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2676{color}
{color:#de350b}#6 0x00007f33187c1a11 in thread_run (arg=0x10f6e40) at
/home/mick/latest/qpid-dispatch/src/server.c:1140{color}
{color:#de350b}#7 0x00007f331869e3f9 in start_thread () from
/lib64/libpthread.so.0{color}
{color:#de350b}#8 0x00007f33181b2b53 in clone () from /lib64/libc.so.6{color}
{color:#de350b}Thread 8 (Thread 0x7f3302ffd640 (LWP 36448)):{color}
{color:#de350b}#0 0x00007f33186a7ea0 in __lll_lock_wait () from
/lib64/libpthread.so.0{color}
{color:#de350b}#1 0x00007f33186a08f5 in pthread_mutex_lock () from
/lib64/libpthread.so.0{color}
{color:#de350b}#2 0x00007f33186dfc5f in lock (m=0x10edc90) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326{color}
{color:#de350b}#3 pni_raw_connection_done (rc=0x10ed3b8) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll_raw_connection.c:423{color}
{color:#de350b}#4 pn_proactor_done (batch=0x10ed970, p=0x10ed970) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2696{color}
{color:#de350b}#5 pn_proactor_done (p=0x10ed970,
batch=batch@entry=0x7f32600fd028) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2676{color}
{color:#de350b}#6 0x00007f33187c1a11 in thread_run (arg=0x10f6e40) at
/home/mick/latest/qpid-dispatch/src/server.c:1140{color}
{color:#de350b}#7 0x00007f331869e3f9 in start_thread () from
/lib64/libpthread.so.0{color}
{color:#de350b}#8 0x00007f33181b2b53 in clone () from /lib64/libc.so.6{color}
{color:#de350b}Thread 7 (Thread 0x7f327dffb640 (LWP 36485)):{color}
{color:#de350b}#0 0x00007f33186a7ea0 in __lll_lock_wait () from
/lib64/libpthread.so.0{color}
{color:#de350b}#1 0x00007f33186a08f5 in pthread_mutex_lock () from
/lib64/libpthread.so.0{color}
{color:#de350b}#2 0x00007f33186e2d79 in lock (m=0x10edc90) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326{color}
{color:#de350b}#3 suspend (ts=<optimized out>, p=<optimized out>) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:397{color}
{color:#de350b}#4 next_event_batch (p=0x10ed970, can_block=true) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2459{color}
{color:#de350b}#5 0x00007f33187c192f in thread_run (arg=0x10f6e40) at
/home/mick/latest/qpid-dispatch/src/server.c:1107{color}
{color:#de350b}#6 0x00007f331869e3f9 in start_thread () from
/lib64/libpthread.so.0{color}
{color:#de350b}#7 0x00007f33181b2b53 in clone () from /lib64/libc.so.6{color}
{color:#de350b}Thread 6 (Thread 0x7f32faffd640 (LWP 36447)):{color}
{color:#de350b}#0 0x00007f33186a7ea0 in __lll_lock_wait () from
/lib64/libpthread.so.0{color}
{color:#de350b}#1 0x00007f33186a08f5 in pthread_mutex_lock () from
/lib64/libpthread.so.0{color}
{color:#de350b}#2 0x00007f33186e2509 in lock (m=0x10edc90) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326{color}
{color:#de350b}#3 next_event_batch (p=0x10ed970, can_block=true) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2411{color}
{color:#de350b}#4 0x00007f33187c192f in thread_run (arg=0x10f6e40) at
/home/mick/latest/qpid-dispatch/src/server.c:1107{color}
{color:#de350b}#5 0x00007f331869e3f9 in start_thread () from
/lib64/libpthread.so.0{color}
{color:#de350b}#6 0x00007f33181b2b53 in clone () from /lib64/libc.so.6{color}
{color:#de350b}Thread 5 (Thread 0x7f327d7fa640 (LWP 36486)):{color}
{color:#de350b}#0 0x00007f33186a7ea0 in __lll_lock_wait () from
/lib64/libpthread.so.0{color}
{color:#de350b}#1 0x00007f33186a08f5 in pthread_mutex_lock () from
/lib64/libpthread.so.0{color}
{color:#de350b}#2 0x00007f33186dfc5f in lock (m=0x10edc90) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326{color}
{color:#de350b}#3 pni_raw_connection_done (rc=0x10ed3b8) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll_raw_connection.c:423{color}
{color:#de350b}#4 pn_proactor_done (batch=0x10ed970, p=0x10ed970) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2696{color}
{color:#de350b}#5 pn_proactor_done (p=0x10ed970,
batch=batch@entry=0x7f32dc076678) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2676{color}
{color:#de350b}#6 0x00007f33187c1a11 in thread_run (arg=0x10f6e40) at
/home/mick/latest/qpid-dispatch/src/server.c:1140{color}
{color:#de350b}#7 0x00007f331869e3f9 in start_thread () from
/lib64/libpthread.so.0{color}
{color:#de350b}#8 0x00007f33181b2b53 in clone () from /lib64/libc.so.6{color}
{color:#de350b}Thread 4 (Thread 0x7f3308b68640 (LWP 36444)):{color}
{color:#de350b}#0 0x00007f33186a7ea0 in __lll_lock_wait () from
/lib64/libpthread.so.0{color}
{color:#de350b}#1 0x00007f33186a08f5 in pthread_mutex_lock () from
/lib64/libpthread.so.0{color}
{color:#de350b}#2 0x00007f33186dfc5f in lock (m=0x10edc90) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326{color}
{color:#de350b}#3 pni_raw_connection_done (rc=0x10ed3b8) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll_raw_connection.c:423{color}
{color:#de350b}#4 pn_proactor_done (batch=0x10ed970, p=0x10ed970) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2696{color}
{color:#de350b}#5 pn_proactor_done (p=0x10ed970,
batch=batch@entry=0x7f32180dff68) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2676{color}
{color:#de350b}#6 0x00007f33187c1a11 in thread_run (arg=0x10f6e40) at
/home/mick/latest/qpid-dispatch/src/server.c:1140{color}
{color:#de350b}#7 0x00007f331869e3f9 in start_thread () from
/lib64/libpthread.so.0{color}
{color:#de350b}#8 0x00007f33181b2b53 in clone () from /lib64/libc.so.6{color}
{color:#de350b}Thread 3 (Thread 0x7f3303fff640 (LWP 36445)):{color}
{color:#de350b}#0 0x00007f33186a7ea0 in __lll_lock_wait () from
/lib64/libpthread.so.0{color}
{color:#de350b}#1 0x00007f33186a08f5 in pthread_mutex_lock () from
/lib64/libpthread.so.0{color}
{color:#de350b}#2 0x00007f33186e2b7e in lock (m=0x10edc90) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326{color}
{color:#de350b}#3 process (tsk=<optimized out>) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2248{color}
{color:#de350b}#4 next_event_batch (p=<optimized out>, can_block=true) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2423{color}
{color:#de350b}#5 0x00007f33187c192f in thread_run (arg=0x10f6e40) at
/home/mick/latest/qpid-dispatch/src/server.c:1107{color}
{color:#de350b}#6 0x00007f331869e3f9 in start_thread () from
/lib64/libpthread.so.0{color}
{color:#de350b}#7 0x00007f33181b2b53 in clone () from /lib64/libc.so.6{color}
{color:#de350b}Thread 2 (Thread 0x7f33179222c0 (LWP 36442)):{color}
{color:#de350b}#0 0x00007f33186a7ea0 in __lll_lock_wait () from
/lib64/libpthread.so.0{color}
{color:#de350b}#1 0x00007f33186a08f5 in pthread_mutex_lock () from
/lib64/libpthread.so.0{color}
{color:#de350b}#2 0x00007f33186dfc5f in lock (m=0x10edc90) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326{color}
{color:#de350b}#3 pni_raw_connection_done (rc=0x10ed3b8) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll_raw_connection.c:423{color}
{color:#de350b}#4 pn_proactor_done (batch=0x10ed970, p=0x10ed970) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2696{color}
{color:#de350b}#5 pn_proactor_done (p=0x10ed970,
batch=batch@entry=0x7f32a403fec8) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2676{color}
{color:#de350b}#6 0x00007f33187c1a11 in thread_run (arg=arg@entry=0x10f6e40) at
/home/mick/latest/qpid-dispatch/src/server.c:1140{color}
{color:#de350b}#7 0x00007f33187c20b0 in qd_server_run (qd=<optimized out>) at
/home/mick/latest/qpid-dispatch/src/server.c:1484{color}
{color:#de350b}#8 0x00000000004026a8 in main_process
(config_path=0x7ffede736db3 "./1_router.conf", python_pkgdir=<optimized out>,
test_hooks=<optimized out>, fd=2) at
/home/mick/latest/qpid-dispatch/router/src/main.c:113{color}
{color:#de350b}#9 0x000000000040240b in main (argc=3, argv=0x7ffede735188) at
/home/mick/latest/qpid-dispatch/router/src/main.c:367{color}
{color:#de350b}Thread 1 (Thread 0x7f32457fa640 (LWP 36499)):{color}
{color:#de350b}#0 0x00007f33186a0684 in pthread_mutex_lock () from
/lib64/libpthread.so.0{color}
{color:#de350b}#1 0x00007f33186e2848 in lock (m=<optimized out>) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:326{color}
{color:#de350b}#2 process (tsk=<optimized out>) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2248{color}
{color:#de350b}#3 next_event_batch (p=0x10ed970, can_block=true) at
/home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2423{color}
{color:#de350b}#4 0x00007f33187c192f in thread_run (arg=0x10f6e40) at
/home/mick/latest/qpid-dispatch/src/server.c:1107{color}
{color:#de350b}#5 0x00007f331869e3f9 in start_thread () from
/lib64/libpthread.so.0{color}
{color:#de350b}#6 0x00007f33181b2b53 in clone () from /lib64/libc.so.6{color}
{color:#de350b}(gdb){color}
{color:#de350b}}{color}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org
For additional commands, e-mail: dev-h...@qpid.apache.org
