Hi all,
I'm trying to address all remarks before to publish that on my blog.
Please open your chakras, informations I've added right now in the e-mail
are work in progress, I'm not an expert of unix socket, meinheld nor uwsgi,
certainly to change a config, it should change values:
1. *Change kernel parameters is a cheat*: No, it isn't a cheat, most
production applications recommend to do that, not only for benchmarks.
Example with:
1. PostgreSQL:
https://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server
shared_buffers config (BTW, I've forgotten to push my kernel settings for
postgresql, it's now in the repository)
2. Nginx: http://wiki.nginx.org/SSL-Offloader section preparation
2. *Debug mode on daemons*: All daemons was impacted, I've disabled
that. I've relaunched localhost benchmark on /agents endpoint, I've almost
the same values, certainly because I've already disabled logging globally
in my benchmarks. If somebody could confirm my diagnostic ?
3. *wrk/wrk2 aren't good tool to benchmark HTTP, its can hit too much*:
It's the goal of a benchmark to hit highest as possible, isn't it ? FYI,
almost serious benchmark reports on the Web use wrk.
4. *Keep-alive*: I've installed a nginx on the server and I use UNIX
socket as suggested. I use meinheld (apparently the best async worker for
gunicorn) with this command line:
gunicorn -w 16 -b 0.0.0.0:8000 -k meinheld.gmeinheld.MeinheldWorker -b
unix:/tmp/flask.sock --backlog=10240 application:app
I don't relaunch everything, only some agents tests locally and with
distant. It's more catastrophic than without nginx when I hit with the 4000
req/s. In fact, I've a lot of 404 errors about socket file BUT, it isn't a
config file error, because, at the beginning of HTTP keep-alive connection,
I received my JSON document. I can send a wireshark capture if you don't
believe me.
If you have a clue, I'm interested in. I'll try tomorrow to test on a
server without any system modifications, I want to find if I've an issue in
my system setup, or it's the reality. I'll also test more intermediary
values to try to find a better scenario for Django/Flask.
But for your information, with agents webservices on the network, with
only 50 connections opened in the same time, I've no more error with Flask.
This is the values:
For Nginx + gunicorn + meinheld (with keep alive):
axelle@GMLUDO-XPS:~$ wrk -t12 -c50 -d30s
http://192.168.2.100:18000/agents
Running 30s test @ http://192.168.2.100:18000/agents
12 threads and 50 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency *74.93ms* 9.73ms 153.97ms 79.89%
Req/Sec 52.98 5.56 73.00 68.97%
19234 requests in 30.00s, 133.39MB read
Requests/sec: *641.08*
Transfer/sec: 4.45MB
For API-Hour:
axelle@GMLUDO-XPS:~$ wrk -t12 -c50 -d30s http://192.168.2.100:8008/agents
Running 30s test @ http://192.168.2.100:8008/agents
12 threads and 50 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency *12.82ms* 8.81ms 95.62ms 69.17%
Req/Sec 357.40 139.34 0.92k 71.68%
118928 requests in 30.00s, 745.16MB read
Requests/sec: *3964.59*
Transfer/sec: 24.84MB
Moreover, (not for this week) I'll ask at my company if I can open
source a more complex daemon example to confirm theses values with another
Flask daemon I've ported to API-Hour where I'd also some performance issues.
Config files for Nginx are in:
https://github.com/Eyepea/API-Hour/tree/master/benchmarks/etc
5. With the test with 10 queries by seconds, I've almost same value for
latency with Gunicorn directly:
$ wrk2 -t10 -c10 -d30s -R 10 http://192.168.2.100:18000/agents
Running 30s test @ http://192.168.2.100:18000/agents
10 threads and 10 connections
Thread calibration: mean lat.: 22.676ms, rate sampling interval: 51ms
Thread calibration: mean lat.: 22.336ms, rate sampling interval: 52ms
Thread calibration: mean lat.: 22.409ms, rate sampling interval: 50ms
Thread calibration: mean lat.: 21.755ms, rate sampling interval: 48ms
Thread calibration: mean lat.: 21.366ms, rate sampling interval: 49ms
Thread calibration: mean lat.: 21.304ms, rate sampling interval: 48ms
Thread calibration: mean lat.: 21.398ms, rate sampling interval: 44ms
Thread calibration: mean lat.: 20.078ms, rate sampling interval: 43ms
Thread calibration: mean lat.: 18.530ms, rate sampling interval: 41ms
Thread calibration: mean lat.: 19.120ms, rate sampling interval: 41ms
Thread Stats Avg Stdev Max +/- Stdev
Latency 20.09ms 3.45ms 28.16ms 64.00%
Req/Sec 0.99 4.51 25.00 95.35%
300 requests in 30.01s, 2.08MB read
Requests/sec: 10.00
Transfer/sec: 71.00KB
6. *uWSGI*: I continue to use Nginx for keepalive with Flask and Django.
It works when I call directly via HTTP, but not via unix sockets, if
somebody has a working config file, I'm interested in.
7. *Use threads instead of workers*: I'll test tomorrow because
I really need to sleep.
Don't worry, I'll relaunch full-monty benchmarks this week-end with all
details.
--
Ludovic Gasc
On Thu, Feb 26, 2015 at 6:49 PM, Guido van Rossum <[email protected]> wrote:
> On Thu, Feb 26, 2015 at 1:41 AM, Ludovic Gasc <[email protected]> wrote:
>
>> [...] To my knowledge, in my benchmark, I don't use threads, only
>> processes.
>>
>
> Maybe that's the reason why the benchmark seems so imbalanced. Processes
> cause way more overhead. Try comparing async I/O to a thread-based solution
> (which is what most people use).
>
> --
> --Guido van Rossum (python.org/~guido)
>
