Hi, About tests with a standard server with the same hardware and without any kernel customizations, I've the same proportion between tests, but with less performance for all daemons. The only notable thing is that with API-Hour, I'd around 50 timeouts by round instead of 0 with my customized server.
About Nginx as frontend and process/threads comparison, I must finish to implement unix sockets and threads worker in API-Hour before to test. Regards. -- Ludovic Gasc On Fri, Feb 27, 2015 at 1:12 AM, Ludovic Gasc <[email protected]> wrote: > Hi all, > > I'm trying to address all remarks before to publish that on my blog. > Please open your chakras, informations I've added right now in the e-mail > are work in progress, I'm not an expert of unix socket, meinheld nor uwsgi, > certainly to change a config, it should change values: > > 1. *Change kernel parameters is a cheat*: No, it isn't a cheat, most > production applications recommend to do that, not only for benchmarks. > Example with: > 1. PostgreSQL: > https://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server > shared_buffers config (BTW, I've forgotten to push my kernel settings > for > postgresql, it's now in the repository) > 2. Nginx: http://wiki.nginx.org/SSL-Offloader section preparation > > 2. *Debug mode on daemons*: All daemons was impacted, I've disabled > that. I've relaunched localhost benchmark on /agents endpoint, I've almost > the same values, certainly because I've already disabled logging globally > in my benchmarks. If somebody could confirm my diagnostic ? > > 3. *wrk/wrk2 aren't good tool to benchmark HTTP, its can hit too much*: > It's the goal of a benchmark to hit highest as possible, isn't it ? FYI, > almost serious benchmark reports on the Web use wrk. > > 4. *Keep-alive*: I've installed a nginx on the server and I use UNIX > socket as suggested. I use meinheld (apparently the best async worker for > gunicorn) with this command line: > > gunicorn -w 16 -b 0.0.0.0:8000 -k meinheld.gmeinheld.MeinheldWorker -b > unix:/tmp/flask.sock --backlog=10240 application:app > > I don't relaunch everything, only some agents tests locally and with > distant. It's more catastrophic than without nginx when I hit with the 4000 > req/s. In fact, I've a lot of 404 errors about socket file BUT, it isn't a > config file error, because, at the beginning of HTTP keep-alive connection, > I received my JSON document. I can send a wireshark capture if you don't > believe me. > If you have a clue, I'm interested in. I'll try tomorrow to test on a > server without any system modifications, I want to find if I've an issue in > my system setup, or it's the reality. I'll also test more intermediary > values to try to find a better scenario for Django/Flask. > > But for your information, with agents webservices on the network, with > only 50 connections opened in the same time, I've no more error with Flask. > This is the values: > > For Nginx + gunicorn + meinheld (with keep alive): > > axelle@GMLUDO-XPS:~$ wrk -t12 -c50 -d30s > http://192.168.2.100:18000/agents > Running 30s test @ http://192.168.2.100:18000/agents > 12 threads and 50 connections > Thread Stats Avg Stdev Max +/- Stdev > Latency *74.93ms* 9.73ms 153.97ms 79.89% > Req/Sec 52.98 5.56 73.00 68.97% > 19234 requests in 30.00s, 133.39MB read > Requests/sec: *641.08* > Transfer/sec: 4.45MB > > For API-Hour: > > axelle@GMLUDO-XPS:~$ wrk -t12 -c50 -d30s > http://192.168.2.100:8008/agents > Running 30s test @ http://192.168.2.100:8008/agents > 12 threads and 50 connections > Thread Stats Avg Stdev Max +/- Stdev > Latency *12.82ms* 8.81ms 95.62ms 69.17% > Req/Sec 357.40 139.34 0.92k 71.68% > 118928 requests in 30.00s, 745.16MB read > Requests/sec: *3964.59* > Transfer/sec: 24.84MB > > Moreover, (not for this week) I'll ask at my company if I can open > source a more complex daemon example to confirm theses values with another > Flask daemon I've ported to API-Hour where I'd also some performance > issues. > Config files for Nginx are in: > https://github.com/Eyepea/API-Hour/tree/master/benchmarks/etc > > 5. With the test with 10 queries by seconds, I've almost same value > for latency with Gunicorn directly: > > $ wrk2 -t10 -c10 -d30s -R 10 http://192.168.2.100:18000/agents > Running 30s test @ http://192.168.2.100:18000/agents > 10 threads and 10 connections > Thread calibration: mean lat.: 22.676ms, rate sampling interval: 51ms > Thread calibration: mean lat.: 22.336ms, rate sampling interval: 52ms > Thread calibration: mean lat.: 22.409ms, rate sampling interval: 50ms > Thread calibration: mean lat.: 21.755ms, rate sampling interval: 48ms > Thread calibration: mean lat.: 21.366ms, rate sampling interval: 49ms > Thread calibration: mean lat.: 21.304ms, rate sampling interval: 48ms > Thread calibration: mean lat.: 21.398ms, rate sampling interval: 44ms > Thread calibration: mean lat.: 20.078ms, rate sampling interval: 43ms > Thread calibration: mean lat.: 18.530ms, rate sampling interval: 41ms > Thread calibration: mean lat.: 19.120ms, rate sampling interval: 41ms > Thread Stats Avg Stdev Max +/- Stdev > Latency 20.09ms 3.45ms 28.16ms 64.00% > Req/Sec 0.99 4.51 25.00 95.35% > 300 requests in 30.01s, 2.08MB read > Requests/sec: 10.00 > Transfer/sec: 71.00KB > 6. *uWSGI*: I continue to use Nginx for keepalive with Flask and > Django. It works when I call directly via HTTP, but not via unix sockets, > if somebody has a working config file, I'm interested in. > 7. *Use threads instead of workers*: I'll test tomorrow because > I really need to sleep. > > Don't worry, I'll relaunch full-monty benchmarks this week-end with all > details. > -- > Ludovic Gasc > > On Thu, Feb 26, 2015 at 6:49 PM, Guido van Rossum <[email protected]> > wrote: > >> On Thu, Feb 26, 2015 at 1:41 AM, Ludovic Gasc <[email protected]> wrote: >> >>> [...] To my knowledge, in my benchmark, I don't use threads, only >>> processes. >>> >> >> Maybe that's the reason why the benchmark seems so imbalanced. Processes >> cause way more overhead. Try comparing async I/O to a thread-based solution >> (which is what most people use). >> >> -- >> --Guido van Rossum (python.org/~guido) >> > >
