Hi Aleks,
On Tue, Jan 17, 2017 at 8:55 AM, Aleksandar Lazic <[email protected]>
wrote:
> Hi.
>
> Am 17-01-2017 05:46, schrieb Atha Kouroussis:
>
> Hi all,
>>
>> I seem to hitting some kind of bottleneck at about 4k req/s and I'm not
>> able to find the cause.
>>
>> I have HAproxy 1.7.2 installed on Ubuntu 16.04.1, VM with 8 cores, 2GB
>> RAM, 1 Gbps networking. Testing with ab cannot get past ~4K req/s. Hitting
>> the backend directly can yield 8-10K without issues. Requests are POST, 1K
>> data. Requests should be very short lived, 10-50ms average, but if going
>> through HAproxy it seems to more than double to 150ms range. Roundtrip
>> between haproxy and backend is sub-1ms. Attaching haproxy and OS config
>> below.
>>
>> Any help/pointers on what might be wrong is greatly appreciated. Thanks
>> in advance!
>>
>
> What shows the status page of haproxy?
>
The status page of haproxy shows the session rate oscilates around
4500-5000 during the load test. Actual number of sessions remains very low,
doesn't go beyond 1K.
>
> Do you run ab & haproxy on the same machine?
No, ab is run on dedicated machines, generate load from 3 clients with ab.
>
Can you post the output of ab for the both differnt backends (haproxy &
> backend)
>
Output from ab against haproxy:
Document Path: /
Document Length: Variable
Concurrency Level: 200
Time taken for tests: 24.993 seconds
Complete requests: 100000
Failed requests: 0
Non-2xx responses: 23
Total transferred: 16396187 bytes
Total body sent: 113000000
HTML transferred: 2484 bytes
Requests per second: 4001.11 [#/sec] (mean)
Time per request: 49.986 [ms] (mean)
Time per request: 0.250 [ms] (mean, across all concurrent requests)
Transfer rate: 640.65 [Kbytes/sec] received
4415.28 kb/s sent
5055.94 kb/s total
Output from ab against backend:
Document Path: /
Document Length: Variable
Concurrency Level: 200
Time taken for tests: 16.734 seconds
Complete requests: 100000
Failed requests: 68
(Connect: 0, Receive: 34, Length: 0, Exceptions: 34)
Total transferred: 14495070 bytes
Total body sent: 113600000
HTML transferred: 0 bytes
Requests per second: 5975.80 [#/sec] (mean)
Time per request: 33.468 [ms] (mean)
Time per request: 0.167 [ms] (mean, across all concurrent requests)
Transfer rate: 845.89 [Kbytes/sec] received
6629.40 kb/s sent
7475.29 kb/s totall
That is with 1 ab client and haproxy with 1 backend.
> But all over ~4k/s isn't that bad ;-)
>
What is driving me nuts is that it doesn't matter if I use 1 or 4 server
backends in haproxy, or if I generate the load with 1 or 3 clients, the
constant is the bottleneck in haproxy at about 4k/s. Resources are plenty
in the VM, should be able to go much higher.
Best,
Atha
> BR Aleks
>
>
> Best,
>> Atha
>>
>> ##########
>> haproxy config
>> ##########
>>
>> global
>> log /dev/log local0
>> log /dev/log local1 notice
>> chroot /var/lib/haproxy
>> stats socket /run/haproxy/admin.sock mode 660 level admin
>> stats timeout 30s
>> stats bind-process 1
>> user haproxy
>> group haproxy
>> daemon
>> # Default SSL material locations
>> ca-base /etc/ssl/certs
>> crt-base /etc/ssl/private
>> ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES
>> 256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
>> ssl-default-bind-options no-sslv3
>>
>> maxconn 200000
>> nbproc 8
>> cpu-map 1 0
>> cpu-map 2 1
>> cpu-map 3 2
>> cpu-map 4 3
>> cpu-map 5 4
>> cpu-map 6 5
>> cpu-map 7 6
>> cpu-map 8 7
>>
>> defaults
>> mode http
>> timeout connect 5s
>> timeout client 50s
>> timeout server 50s
>> errorfile 400 /etc/haproxy/errors/400.http
>> errorfile 403 /etc/haproxy/errors/403.http
>> errorfile 408 /etc/haproxy/errors/408.http
>> errorfile 500 /etc/haproxy/errors/500.http
>> errorfile 502 /etc/haproxy/errors/502.http
>> errorfile 503 /etc/haproxy/errors/503.http
>> errorfile 504 /etc/haproxy/errors/504.http
>>
>> listen bidders
>> bind *:80
>> maxconn 200000
>> server srv1 xx.xx.xx.xx:yy check
>>
>> ##########
>> sysctl settings
>> ##########
>> net.ipv4.tcp_mem = 786432 1697152 1945728
>> net.ipv4.tcp_rmem = 4096 4096 16777216
>> net.ipv4.tcp_wmem = 4096 4096 16777216
>> net.ipv4.tcp_tw_reuse = 1
>> net.ipv4.ip_local_port_range = 1024 65023
>> net.ipv4.tcp_max_syn_backlog = 60000
>> net.ipv4.tcp_fin_timeout = 30
>> net.ipv4.tcp_synack_retries = 3
>> net.ipv4.ip_nonlocal_bind = 1
>> net.core.somaxconn = 60000
>> net.core.netdev_max_backlog = 10000
>> fs.file-max = 10000000
>> fs.nr_open = 10000000
>>
>> #######
>> Security limits conf
>> #######
>> haproxy soft nofile 999999
>> haproxy hard nofile 999999
>>
>
