Hi all,
I seem to hitting some kind of bottleneck at about 4k req/s and I’m not
able to find the cause.
I have HAproxy 1.7.2 installed on Ubuntu 16.04.1, VM with 8 cores, 2GB RAM,
1 Gbps networking. Testing with ab cannot get past ~4K req/s. Hitting the
backend directly can yield 8-10K without issues. Requests are POST, 1K
data. Requests should be very short lived, 10-50ms average, but if going
through HAproxy it seems to more than double to 150ms range. Roundtrip
between haproxy and backend is sub-1ms. Attaching haproxy and OS config
below.
Any help/pointers on what might be wrong is greatly appreciated. Thanks in
advance!
Best,
Atha
##########
haproxy config
##########
global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin
stats timeout 30s
stats bind-process 1
user haproxy
group haproxy
daemon
# Default SSL material locations
ca-base /etc/ssl/certs
crt-base /etc/ssl/private
ssl-default-bind-ciphers
ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
ssl-default-bind-options no-sslv3
maxconn 200000
nbproc 8
cpu-map 1 0
cpu-map 2 1
cpu-map 3 2
cpu-map 4 3
cpu-map 5 4
cpu-map 6 5
cpu-map 7 6
cpu-map 8 7
defaults
mode http
timeout connect 5s
timeout client 50s
timeout server 50s
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http
listen bidders
bind *:80
maxconn 200000
server srv1 xx.xx.xx.xx:yy check
##########
sysctl settings
##########
net.ipv4.tcp_mem = 786432 1697152 1945728
net.ipv4.tcp_rmem = 4096 4096 16777216
net.ipv4.tcp_wmem = 4096 4096 16777216
net.ipv4.tcp_tw_reuse = 1
net.ipv4.ip_local_port_range = 1024 65023
net.ipv4.tcp_max_syn_backlog = 60000
net.ipv4.tcp_fin_timeout = 30
net.ipv4.tcp_synack_retries = 3
net.ipv4.ip_nonlocal_bind = 1
net.core.somaxconn = 60000
net.core.netdev_max_backlog = 10000
fs.file-max = 10000000
fs.nr_open = 10000000
#######
Security limits conf
#######
haproxy soft nofile 999999
haproxy hard nofile 999999
