Hi, Thanks for the response!
On Wednesday, February 10, 2021 at 3:44:56 AM UTC-6 Fotis Xenakis wrote: > On Monday, February 8, 2021 at 6:00:29 PM UTC+2 [email protected] wrote: > >> Hi, >> >> On Wednesday, February 3, 2021 at 11:47:13 AM UTC-5 Matthew Kenigsberg >> wrote: >> >>> >>> Hi, >>> >>> I'm trying to benchmark nginx with weighttp, but it appears to crash >>> after too many requests are sent. As soon as requests start failing, it >>> seems all following requests will fail. Any ideas why? >>> >> Not really but I think I may have seen similar behavior with nginx. >> > I have recently tried a very similar scenario: nginx on OSv, stressed with > vegeta on the host. While I didn't face the same issue, there was a problem > when using NFS for the underlying files (the benchmark target was the fs, > so I was using some static files, not the default nginx response page). I > didn't get to the root of it at the time, but is seemed like a deadlock, in > OSv. > > Out of curiosity: > - Have you tried to e.g. use lighttpd instead of nginx? > - What file system are you running (might be irrelevant)? > The default which I believe is zfs > - Does OSv seem to "hang"? Is it accepting connections or not? > Doesn't seem to hang, but maybe not accepting connections? A simple curl localhost gives curl: (52) Empty reply from server > >>> >> >>> $ curl ip >>> <!DOCTYPE html> >>> <html> >>> <head> >>> <title>Welcome to nginx!</title> >>> ... >>> $ weighttp -t 1 -n 1000 -c 1 http://ip >>> ... >>> requests: 1000 total, 1000 started, 1000 done, 1000 succeeded, 0 failed, >>> 0 errored >>> ... >>> >>> $ weighttp -t 1 -n 1000 -c 1 http://ip >>> ... >>> requests: 1000 total, 1000 started, 1000 done, 22 succeeded, 978 failed, >>> 0 errored >>> ... >>> >>> $ curl 129.114.109.194 >>> curl: (52) Empty reply from server >>> $ sleep 60 >>> $ curl 129.114.109.194 >>> curl: (52) Empty reply from server >>> >>> Not sure if this is related to either of: >>> https://groups.google.com/u/1/g/osv-dev/c/jclM9NhO7eg/m/BXV20hJUAQAJ >>> https://github.com/cloudius-systems/osv/issues/889 >>> >> Maybe. >> It would be best if you could connect with gdb to OSv when it >> crashes/hangs and send us a thread list and ideally stack traces of >> interesting threads (most likely those part of TCP stack). I suggest you >> build and run ROFS image as it will have fewer threads. See >> https://github.com/cloudius-systems/osv/wiki/Debugging-OSv. >> > Tried to use rofs but it's not working immediately. It was still trying to open error log files in read mode even after I changed nginx.conf. If that's important I could work on that more. +1 > The ability to debug with GDB has proved invaluable! > >> >> Does this crash happen with single cpu or with many? >> > I'm running on a server with many CPUs, I think up to 64 threads. Here's some debugging info from immediately before weighttp gets back 1000 failed requests (weighttp gets responses when I finish virtio::net::receiver). Debugging this kind of stuff is definitely outside of my comfort zone, so if there's more specific things I should be doing let me know! (gdb) bt #0 0x000000004036d64e in virtio::net::receiver (this=0xffff900000ee9000) at drivers/virtio-net.cc:539 #1 0x000000004036d8ac in virtio::net::<lambda()>::operator() (__closure=<optimized out>) at drivers/virtio-net.cc:242 #2 std::_Function_handler<void(), virtio::net::net(virtio::virtio_device&)::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /nix/store/wmzmnnrj780268ybilmcfyd3grn5qzi6-gcc-9.3.0/include/c++/9.3.0/bits/std_function.h:300 #3 0x00000000403f9e18 in sched::thread_main_c (t=0xffff80007ff54040) at arch/x64/arch-switch.hh:325 #4 0x000000004038cf13 in thread_main () at arch/x64/entry.S:113 (gdb) info threads Id Target Id Frame * 1 Thread 1.1 (CPU#0 [running]) 0x000000004036d64e in virtio::net::receiver (this=0xffff900000ee9000) at drivers/virtio-net.cc:539 2 Thread 1.2 (CPU#1 [running]) std::__atomic_base<unsigned long>::load (__m=std::memory_order_relaxed, this=0xffff8000000567e0) at /nix/store/wmzmnnrj780268ybilmcfyd3grn5qzi6-gcc-9.3.0/include/c++/9.3.0/bits/atomic_base.h:413 3 Thread 1.3 (CPU#2 [halted ]) sched::cpu::do_idle (this=0xffff800000088040) at core/sched.cc:405 4 Thread 1.4 (CPU#3 [running]) sched::cpu::do_idle (this=0xffff80000008c040) at include/boost/intrusive/bstree.hpp:945 (gdb) osv info threads 1 (0xffff800000017040) reclaimer cpu0 status::waiting condvar::wait(lockfree::mutex*, sched::timer*) at core/condvar.cc:43 vruntime 5.55762e-24 2 (0xffff800000091040) kvm_wall_clock_ cpu0 status::waiting std::_Function_handler<void(), kvmclock::kvmclock()::<lambda()> >::_M_invoke(const std::_Any_data &) at /nix/store/wmzmnnrj780268ybilmcfyd3grn5qzi6-gcc-9.3.0/include/c++/9.3.0/bits/std_function.h:300 vruntime 47.8555 3 (0xffff800000173040) page-access-sca cpu1 status::waiting pagecache::access_scanner::run() at core/pagecache.cc:695 vruntime 41259.8 4 (0xffff80000018c040) page_pool_l2 cpu0 status::waiting memory::page_pool::l2::fill_thread() at core/mempool.cc:1437 vruntime 3.18475e-05 5 (0xffff8000001a7040) itimer-real cpu? status::unstarted ?? at arch/x64/entry.S:112 vruntime 0 6 (0xffff8000001bf040) itimer-virt cpu? status::unstarted ?? at arch/x64/entry.S:112 vruntime 0 7 (0xffff800000a22040) balancer0 cpu0 status::waiting ?? at arch/x64/entry.S:113 vruntime 260.086 8 (0xffff800000a39040) rcu0 cpu0 status::waiting osv::rcu::cpu_quiescent_state_thread::do_work() at core/rcu.cc:178 vruntime 1.0958 9 (0xffff800000a50040) page_pool_l1_0 cpu0 status::waiting memory::page_pool::l1::fill_thread() at core/mempool.cc:1349 vruntime 0.0484772 10 (0xffff800000a67040) percpu0 cpu0 status::waiting workman::call_of_duty() at core/percpu-worker.cc:92 vruntime 5.55762e-24 11 (0xffff800000a7e040) async_worker0 cpu0 status::waiting async::async_worker::run() at core/async.cc:158 vruntime 280.215 12 (0xffff800000a95040) idle0 cpu0 status::queued std::_Function_handler<void(), sched::cpu::init_idle_thread()::<lambda()> >::_M_invoke(const std::_Any_data &) at /nix/store/wmzmnnrj780268ybilmcfyd3grn5qzi6-gcc-9.3.0/include/c++/9.3.0/bits/std_function.h:300 vruntime inf 13 (0xffff800000aac040) idle1 cpu1 status::running std::_Function_handler<void(), sched::cpu::init_idle_thread()::<lambda()> >::_M_invoke(const std::_Any_data &) at /nix/store/wmzmnnrj780268ybilmcfyd3grn5qzi6-gcc-9.3.0/include/c++/9.3.0/bits/std_function.h:300 vruntime inf 14 (0xffff800000adf040) idle2 cpu2 status::running std::_Function_handler<void(), sched::cpu::init_idle_thread()::<lambda()> >::_M_invoke(const std::_Any_data &) at /nix/store/wmzmnnrj780268ybilmcfyd3grn5qzi6-gcc-9.3.0/include/c++/9.3.0/bits/std_function.h:300 vruntime inf 15 (0xffff800000b12040) idle3 cpu3 status::running sched::cpu::do_idle() at include/boost/intrusive/bstree.hpp:945 vruntime inf 16 (0xffff800000b46040) rcu2 cpu2 status::waiting osv::rcu::cpu_quiescent_state_thread::do_work() at core/rcu.cc:178 vruntime 0.0608673 17 (0xffff800000b5f040) page_pool_l1_2 cpu2 status::waiting memory::page_pool::l1::fill_thread() at core/mempool.cc:1346 vruntime 1.11152e-23 18 (0xffff800000b77040) percpu2 cpu2 status::waiting workman::call_of_duty() at core/percpu-worker.cc:92 vruntime 1.11023e-23 19 (0xffff800000b8f040) async_worker2 cpu2 status::waiting async::async_worker::run() at core/async.cc:158 vruntime 1.11023e-23 20 (0xffff800000ba7040) rcu1 cpu1 status::waiting osv::rcu::cpu_quiescent_state_thread::do_work() at core/rcu.cc:178 vruntime 0.0379105 21 (0xffff800000bc1040) page_pool_l1_1 cpu1 status::waiting memory::page_pool::l1::fill_thread() at core/mempool.cc:1349 vruntime 1.19201e-10 22 (0xffff800000bd9040) percpu1 cpu1 status::waiting workman::call_of_duty() at core/percpu-worker.cc:92 vruntime 1.11152e-23 23 (0xffff800000bf1040) async_worker1 cpu1 status::waiting async::async_worker::run() at core/async.cc:158 vruntime 3.38554e-05 24 (0xffff800000c09040) >init cpu0 status::waiting std::_Function_handler<void (), osv::acpi_interrupt::acpi_interrupt(unsigned int, unsigned int (*)(void*), void*)::{lambda()#1}>::_M_invoke(std::_Any_data const&) at /nix/store/wmzmnnrj780268ybilmcfyd3grn5qzi6-gcc-9.3.0/include/c++/9.3.0/bits/std_function.h:300 vruntime 5.86781e-24 25 (0xffff800000c21040) rcu3 cpu3 status::waiting osv::rcu::cpu_quiescent_state_thread::do_work() at core/rcu.cc:178 vruntime 0.0317421 26 (0xffff800000c3b040) page_pool_l1_3 cpu3 status::waiting memory::page_pool::l1::fill_thread() at core/mempool.cc:1349 vruntime 2.42855e-22 --Type <RET> for more, q to quit, c to continue without paging--c 27 (0xffff800000cf3040) percpu3 cpu3 status::waiting workman::call_of_duty() at core/percpu-worker.cc:92 vruntime 6.44942e-24 28 (0xffff800000e4a040) async_worker3 cpu3 status::waiting async::async_worker::run() at core/async.cc:158 vruntime 5.89366e-24 29 (0xffff800001201040) >init cpu0 status::waiting std::_Function_handler<void(), sched::thread::reaper::reaper()::<lambda()> >::_M_invoke(const std::_Any_data &) at /nix/store/wmzmnnrj780268ybilmcfyd3grn5qzi6-gcc-9.3.0/include/c++/9.3.0/bits/std_function.h:300 vruntime 6.15991e-23 30 (0xffff800001218040) thread taskq cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 1.04819e-23 31 (0xffff800001223040) callout cpu1 status::waiting _callout_thread() at bsd/porting/callout.cc:138 vruntime 196.727 32 (0xffff800001242040) system_taskq_0 cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 4.63867e-23 33 (0xffff80000124d040) system_taskq_1 cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 2.57589e-23 34 (0xffff800001258040) system_taskq_2 cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 2.5578e-23 35 (0xffff800001263040) system_taskq_3 cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 2.57848e-23 36 (0xffff80000126e040) system_taskq_4 cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 2.56555e-23 37 (0xffff800001279040) system_taskq_5 cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 5.80319e-24 38 (0xffff800001284040) system_taskq_6 cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 5.82904e-24 39 (0xffff80000128f040) system_taskq_7 cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 5.80319e-24 40 (0xffff80007fe41040) solthread-0x402 cpu0 status::waiting condvar::wait(lockfree::mutex*, sched::timer*) at core/condvar.cc:43 vruntime 52.043 41 (0xffff80007fe4c040) solthread-0x402 cpu0 status::waiting condvar::wait(lockfree::mutex*, sched::timer*) at core/condvar.cc:43 vruntime 18.793 42 (0xffff80007fe9d040) netisr cpu0 status::waiting netisr_osv_thread_wrapper(void (*)(void*), void*) at bsd/sys/net/netisr1.cc:26 vruntime 1.75647e-23 43 (0xffff80007ff3a040) >init cpu0 status::waiting sched::thread::join() at core/sched.cc:1313 vruntime 3.59201e-20 joining on <optimized out> 44 (0xffff80007ff54040) virtio-net-rx cpu0 status::running virtio::net::receiver() at drivers/virtio-net.cc:539 vruntime 1.72028 45 (0xffff80007ff7c040) virtio-tx-0 cpu0 status::waiting osv::xmitter<virtio::net::txq, 4096u, std::function<bool ()>, boost::iterators::function_output_iterator<osv::xmitter_functor<virtio::net::txq> > >::poll_until() at include/osv/percpu_xmit.hh:395 vruntime 8.71137e-29 46 (0xffff80007ffa4040) virtio-tx-1 cpu1 status::waiting osv::xmitter<virtio::net::txq, 4096u, std::function<bool ()>, boost::iterators::function_output_iterator<osv::xmitter_functor<virtio::net::txq> > >::poll_until() at include/osv/percpu_xmit.hh:395 vruntime 2.22299e-29 47 (0xffff80007ffcc040) virtio-tx-2 cpu2 status::waiting osv::xmitter<virtio::net::txq, 4096u, std::function<bool ()>, boost::iterators::function_output_iterator<osv::xmitter_functor<virtio::net::txq> > >::poll_until() at include/osv/percpu_xmit.hh:395 vruntime 1.46063e-29 48 (0xffff80007ffd2040) virtio-tx-3 cpu3 status::waiting osv::xmitter<virtio::net::txq, 4096u, std::function<bool ()>, boost::iterators::function_output_iterator<osv::xmitter_functor<virtio::net::txq> > >::poll_until() at include/osv/percpu_xmit.hh:395 vruntime 1.43474e-29 49 (0xffff80000154e040) virtio-blk cpu0 status::waiting virtio::virtio_driver::wait_for_queue(virtio::vring*, bool (virtio::vring::*)() const) at drivers/virtio.cc:158 vruntime 1.34348e-06 50 (0xffff800001565040) virtio-rng cpu1 status::waiting condvar::wait(lockfree::mutex*, sched::timer*) at core/condvar.cc:43 vruntime 1446.79 51 (0xffff80000157f040) isa-serial-inpu cpu0 status::waiting console::LineDiscipline::read_poll(console::console_driver*) at drivers/line-discipline.cc:86 vruntime 1.67375e-23 52 (0xffff800001596040) kbd-input cpu0 status::waiting console::LineDiscipline::read_poll(console::console_driver*) at drivers/line-discipline.cc:86 vruntime 1.47859e-23 53 (0xffff8000015d6040) rand_harvestq cpu1 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 5295.02 136 (0xffff8000015e4040) zio_null_issue cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 8.72184e-07 137 (0xffff8000015ef040) zio_null_intr cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 8.70787e-08 138 (0xffff8000015fa040) zio_read_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 7.63833e-14 139 (0xffff800001605040) zio_read_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 7.4607e-14 140 (0xffff800001610040) zio_read_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 7.28306e-14 141 (0xffff80000161b040) zio_read_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 7.54952e-14 142 (0xffff800001626040) zio_read_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 1.20792e-13 143 (0xffff800001631040) zio_read_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 5.96046e-08 144 (0xffff80000163c040) zio_read_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 7.81597e-14 145 (0xffff800001647040) zio_read_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 0.00646973 146 (0xffff800001652040) zio_read_intr_0 cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 5.21541e-08 147 (0xffff80000165d040) zio_read_intr_1 cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 4.98549e-08 148 (0xffff800001668040) zio_read_intr_2 cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 5.72982e-08 149 (0xffff800001673040) zio_read_intr_3 cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 6.33299e-08 150 (0xffff80000167e040) zio_write_issue cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 3.17907e-06 151 (0xffff800001689040) zio_write_issue cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 3.19583e-06 152 (0xffff800001694040) zio_write_issue cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 1.03512e-05 153 (0xffff80000169f040) zio_write_issue cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 2.04239e-06 154 (0xffff8000016aa040) zio_write_issue cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 8.84049e-24 155 (0xffff8000016b5040) zio_write_issue cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 8.85342e-24 156 (0xffff8000016c0040) zio_write_issue cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 8.85342e-24 157 (0xffff8000016cb040) zio_write_issue cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 8.84049e-24 158 (0xffff8000016d6040) zio_write_issue cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 8.84049e-24 159 (0xffff8000016e1040) zio_write_intr_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 1.3276e-06 160 (0xffff8000016ec040) zio_write_intr_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 3.87896e-07 161 (0xffff8000016f7040) zio_write_intr_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 2.16532e-07 162 (0xffff800001702040) zio_write_intr_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 2.92901e-07 163 (0xffff80000170d040) zio_write_intr_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 2.26311e-07 164 (0xffff800001718040) zio_write_intr_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 1.29454e-07 165 (0xffff800001723040) zio_write_intr_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 9.82545e-08 166 (0xffff80000172e040) zio_write_intr_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 5.86268e-07 167 (0xffff800001739040) zio_write_intr_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 4.74975e-08 168 (0xffff800001744040) zio_write_intr_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 6.46804e-07 169 (0xffff80000174f040) zio_write_intr_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 7.54371e-08 170 (0xffff80000175a040) zio_write_intr_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 1.01794e-06 171 (0xffff800001765040) zio_write_intr_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 2.3609e-07 172 (0xffff800001770040) zio_write_intr_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 3.17581e-07 173 (0xffff80000177b040) zio_write_intr_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 3.16184e-07 174 (0xffff800001786040) zio_write_intr_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 5.9139e-08 175 (0xffff800001791040) zio_write_intr_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 8.90512e-24 176 (0xffff80000179c040) zio_write_intr_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 8.89219e-24 177 (0xffff8000017a7040) zio_write_intr_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 8.87927e-24 178 (0xffff8000017b2040) zio_write_intr_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 8.87927e-24 179 (0xffff8000017bd040) zio_write_intr_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 8.84049e-24 180 (0xffff8000017c8040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 8.19564e-08 181 (0xffff8000017d3040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 2.09548e-08 182 (0xffff8000017de040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 1.72295e-08 183 (0xffff8000017e9040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 1.90921e-08 184 (0xffff8000017f4040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 9.27993e-24 185 (0xffff8000017ff040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 6.37956e-08 186 (0xffff80000180a040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 2.00234e-08 187 (0xffff800001815040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 7.54952e-14 188 (0xffff800001820040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 1.76951e-08 189 (0xffff80000182b040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 9.68575e-08 190 (0xffff800001836040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 0.0022583 191 (0xffff800001841040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 4.98258e-08 192 (0xffff80000184c040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 0.00189209 193 (0xffff800001857040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 4.28408e-08 194 (0xffff800001862040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 0.00189209 195 (0xffff80000186d040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 1.72295e-08 196 (0xffff800001878040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 1.72295e-08 197 (0xffff800001883040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 2.18861e-08 198 (0xffff80000188e040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 1.21072e-07 199 (0xffff800001899040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 6.28643e-08 200 (0xffff8000018a4040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 1.45661e-13 201 (0xffff8000018af040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 3.19744e-14 202 (0xffff8000018ba040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 3.10862e-14 203 (0xffff8000018c5040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 0.00738525 204 (0xffff8000018d0040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 8.40984e-07 205 (0xffff8000018db040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 3.77186e-08 206 (0xffff8000018e6040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 0.0148926 207 (0xffff8000018f1040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 0.00628662 208 (0xffff8000018fc040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 0.00830078 209 (0xffff800001907040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 4.70318e-08 210 (0xffff800001912040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 0.0640259 211 (0xffff80000191d040) zio_free_issue_ cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 0.0196533 212 (0xffff800001928040) zio_free_intr cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 8.90512e-24 213 (0xffff800001933040) zio_claim_issue cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 8.91804e-24 214 (0xffff80000193e040) zio_claim_intr cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 8.95682e-24 215 (0xffff800001949040) zio_ioctl_issue cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 8.91804e-24 216 (0xffff800001954040) zio_ioctl_intr cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 8.14907e-08 217 (0xffff800001be9040) zfs_vn_rele_tas cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 9.79692e-24 218 (0xffff800001c83040) solthread-0x403 cpu0 status::waiting condvar::wait(lockfree::mutex*, sched::timer*) at core/condvar.cc:43 vruntime 3.93018e-07 219 (0xffff800001c8e040) solthread-0x403 cpu0 status::waiting condvar::wait(lockfree::mutex*, sched::timer*) at core/condvar.cc:43 vruntime 9.81754e-06 220 (0xffff800001cc6040) zil_clean cpu0 status::waiting synch_port::_msleep(void*, mtx*, int, char const*, int) at bsd/porting/synch.cc:101 vruntime 1.50479 221 (0xffff800001cda040) dhcp cpu0 status::waiting dhcp::dhcp_worker::dhcp_worker_fn() at core/dhcp.cc:828 vruntime 5.60518e-22 222 (0xffff800001d5c040) /nginx.so cpu0 status::waiting epoll_file::wait(epoll_event*, int, int) at core/epoll.cc:142 vruntime 93.4503 Number of threads: 140 > >> >>> >>> >>> Thanks, >>> Matthew >>> >> -- You received this message because you are subscribed to the Google Groups "OSv Development" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/osv-dev/061e339a-d8ed-48f3-99c2-2c1ec5e2f356n%40googlegroups.com.
