> Hello Roberto, > > well, as my program code became more complex with more async switches and > with more io wait/reads, I got in problem - now sometimes, async requests > can never END. it's more often happened under heavy load, but also > can happen - when almost no load. > > I also found, that than more processes I'am running, the more chances > to get these very long running or never finished async requests. is > there some harakiri for so requests? so I don't want to kill > process, I just want to kill asyn request if it's running more than > 4-5 seconds e.g. > > I tried to use nginx timeouts with buffering off, with hope that they > will terminate running request on uwsgi side - but no luck. > > so right now I have very often so situations: > Time taken for tests: 12.938 seconds > Complete requests: 20000 > Requests per second: 1545.80 [#/sec] (mean) > > Percentage of the requests served within a certain time (ms) > 50% 79 > 66% 104 > 75% 129 > 80% 155 > 90% 307 > 95% 804 > 98% 3236 > 99% 7709 > 100% 12928 (longest request) > > as you see longest requests are almost like full benchmark running > time. > > and more worse are so situations, when so requests never ends: > Completed 2000 requests > Completed 4000 requests > Completed 6000 requests > Completed 8000 requests > Completed 10000 requests > Completed 12000 requests > Completed 14000 requests > Completed 16000 requests > Completed 18000 requests > apr_poll: The timeout specified has expired (70007) > Total of 19986 requests completed > > so any ideas Roberto how to kill them? > > will repeat that it's more often with 4-8 uwsgi processes running (on 8 > core machine) and with high concurrences (200-500 threads), but still > exists when processes is fewer than 4 and > with small concurrences (like 50 simulations threads). > > P.S.: also uwsgi from the trunk works 2-3 times slower > >
During my tests i discovered that epoll+uWSGI (so only on Linux) is "vulnerable" to the "thundering herd" problem if run in preforking mode. (this is mitigated in 0.9.6 by the simpler code base but "explodes" in 0.9.7) Now that the master process has switched to epoll() i have discovered/solved it, but i need to fix all the linux-async stack. This will take a couple of days, then i think i can release an official beta1 version of uWSGI 0.9.7. Riccardo Magliocchetti discovered another bug in the master in the 0.9.7 that caused lot of cpu usage (syscall bombing), i have fixed it yesterday. Regarding "blocked" async requests, there is no way to kill only one of them as pratically if one request is blocked all the others will never get cpu (remember we are in cooperative mode). By the way harakiri should do its work, as it is reset after every accept() so if a request is stuck, the process will be reloaded safely. Another feature that i want/need to add to the async stack, is stopping giving cpu to new request if the stack is overloaded (this is easy to measure in async-mode). So if your system is overloaded new requests will be accepted but they will not receive cpu (so python/lua/whatever_you_use is not called until the stack returns healthy) Finally, i have to remove the leave_open functionality from the WSGI plugin (it allow to maintains connection opened as HTTP Keepalive) as it hurts performance a lot (i have to parse all the app generated headers to see if they export Content-Length) and it is overcomplex. Probably you hit this one and the master one to have worst performance. -- Roberto De Ioris http://unbit.it _______________________________________________ uWSGI mailing list [email protected] http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
