Hi,
Thanks for the response!
On Wednesday, February 10, 2021 at 3:44:56 AM UTC-6 Fotis Xenakis wrote:
> On Monday, February 8, 2021 at 6:00:29 PM UTC+2 jwkoz...@gmail.com wrote:
>
>> Hi,
>>
>> On Wednesday, February 3, 2021 at 11:47:13 AM UTC-5 Matthew Kenigsberg
>> wrote:
>>
>>>
>>> Hi,
>>>
>>> I'm trying to benchmark nginx with weighttp, but it appears to crash
>>> after too many requests are sent. As soon as requests start failing, it
>>> seems all following requests will fail. Any ideas why?
>>>
>> Not really but I think I may have seen similar behavior with nginx.
>>
> I have recently tried a very similar scenario: nginx on OSv, stressed with
> vegeta on the host. While I didn't face the same issue, there was a problem
> when using NFS for the underlying files (the benchmark target was the fs,
> so I was using some static files, not the default nginx response page). I
> didn't get to the root of it at the time, but is seemed like a deadlock, in
> OSv.
>
> Out of curiosity:
> - Have you tried to e.g. use lighttpd instead of nginx?
>
- What file system are you running (might be irrelevant)?
>
The default which I believe is zfs
> - Does OSv seem to "hang"? Is it accepting connections or not?
>
Doesn't seem to hang, but maybe not accepting connections? A simple curl
localhost gives
curl: (52) Empty reply from server
>
>>>
>>
>>> $ curl ip
>>>
>>>
>>>
>>> Welcome to nginx!
>>> ...
>>> $ weighttp -t 1 -n 1000 -c 1 http://ip
>>> ...
>>> requests: 1000 total, 1000 started, 1000 done, 1000 succeeded, 0 failed,
>>> 0 errored
>>> ...
>>>
>>> $ weighttp -t 1 -n 1000 -c 1 http://ip
>>> ...
>>> requests: 1000 total, 1000 started, 1000 done, 22 succeeded, 978 failed,
>>> 0 errored
>>> ...
>>>
>>> $ curl 129.114.109.194
>>> curl: (52) Empty reply from server
>>> $ sleep 60
>>> $ curl 129.114.109.194
>>> curl: (52) Empty reply from server
>>>
>>> Not sure if this is related to either of:
>>> https://groups.google.com/u/1/g/osv-dev/c/jclM9NhO7eg/m/BXV20hJUAQAJ
>>> https://github.com/cloudius-systems/osv/issues/889
>>>
>> Maybe.
>> It would be best if you could connect with gdb to OSv when it
>> crashes/hangs and send us a thread list and ideally stack traces of
>> interesting threads (most likely those part of TCP stack). I suggest you
>> build and run ROFS image as it will have fewer threads. See
>> https://github.com/cloudius-systems/osv/wiki/Debugging-OSv.
>>
>
Tried to use rofs but it's not working immediately. It was still trying to
open error log files in read mode even after I changed nginx.conf. If
that's important I could work on that more.
+1
> The ability to debug with GDB has proved invaluable!
>
>>
>> Does this crash happen with single cpu or with many?
>>
> I'm running on a server with many CPUs, I think up to 64 threads.
Here's some debugging info from immediately before weighttp gets back 1000
failed requests (weighttp gets responses when I finish
virtio::net::receiver). Debugging this kind of stuff is definitely outside
of my comfort zone, so if there's more specific things I should be doing
let me know!
(gdb) bt
#0 0x4036d64e in virtio::net::receiver (this=0x90ee9000)
at drivers/virtio-net.cc:539
#1 0x4036d8ac in virtio::netoperator()
(__closure=)
at drivers/virtio-net.cc:242
#2 std::_Function_handler >::_M_invoke(const
std::_Any_data &) (__functor=...)
at
/nix/store/wmzmnnrj780268ybilmcfyd3grn5qzi6-gcc-9.3.0/include/c++/9.3.0/bits/std_function.h:300
#3 0x403f9e18 in sched::thread_main_c (t=0x80007ff54040) at
arch/x64/arch-switch.hh:325
#4 0x4038cf13 in thread_main () at arch/x64/entry.S:113
(gdb) info threads
Id Target IdFrame
* 1Thread 1.1 (CPU#0 [running]) 0x4036d64e in
virtio::net::receiver (this=0x90ee9000)
at drivers/virtio-net.cc:539
2Thread 1.2 (CPU#1 [running]) std::__atomic_base::load
(__m=std::memory_order_relaxed,
this=0x800567e0)
at
/nix/store/wmzmnnrj780268ybilmcfyd3grn5qzi6-gcc-9.3.0/include/c++/9.3.0/bits/atomic_base.h:413
3Thread 1.3 (CPU#2 [halted ]) sched::cpu::do_idle
(this=0x80088040) at core/sched.cc:405
4Thread 1.4 (CPU#3 [running]) sched::cpu::do_idle
(this=0x8008c040)
at include/boost/intrusive/bstree.hpp:945
(gdb) osv info threads
1 (0x80017040) reclaimer cpu0 status::waiting
condvar::wait(lockfree::mutex*, sched::timer*) at core/condvar.cc:43
vruntime 5.55762e-24
2 (0x80091040) kvm_wall_clock_ cpu0 status::waiting
std::_Function_handler
>::_M_invoke(const std::_Any_data &) at
/nix/store/wmzmnnrj780268ybilmcfyd3grn5qzi6-gcc-9.3.0/include/c++/9.3.0/bits/std_function.h:300
vruntime 47.8555
3 (0x80173040) page-access-sca cpu1 status::waiting
pagecache::access_scanner::run() at core/pagecache.cc:695 vruntime
41259.8
4 (0x8018c040) page_pool_l2cpu0 status::waiting
