Hello,
I am trying to configure an HAProxy And seem to be running into a problem
where the HAProxy will spike. I have a high throughput server cluster,
which will need to handle about 10-20k QPS. I believe that HAProxy should
be able to handle that quite easily from what I have read. I am running
HAProxy on Ubuntu 12.04LTS. I am running an a Xeon 1270, with 8GB of
RAM(Which i believe is all overkill). I am doing a simple load balance
between two or three servers(I have two server clusters, one with two
servers, one with three). I am running 1.4.24.
The reason I believe something is wrong with my haproxy is because I can
refresh my webpage a few times, and every 4-5 times I will be get a spike
around 800ms-3sec to resolve. This also happens when not hitting my
backend server, and instead hitting the HAProxy stats page. None of my
servers are anywhere close to using up their system resources(all below 20%
in both memory and CPU). I don't see much in the syslog that seems wrong
other than this message:
Oct 10 16:48:28 haproxy rsyslogd-2177: imuxsock begins to drop message 3912
due to rate-limiting
I believe that has to do with the logging not being able to keep up, rather
than haproxy dropping packets.
I was wondering if there were any suggestions for me to look into to try
and fix my problem, or suggest how to configure a HAProxy correctly to be
able to handle the load I need(preferably higher than the load I need, as
this may increase over time). I am open to changing operating systems if
need be as well.
Here are my two config files:
global
log /dev/log local0 info
log /dev/log local0 notice
maxconn 50000
user y
group y
#debug
defaults
log global
mode http
option httplog
option dontlognull
option forwardfor
retries 3
option redispatch
option http-server-close
maxconn 30000
contimeout 10000
clitimeout 50000
srvtimeout 50000
balance leastconn
listen c_cluster 255.255.255.146:80
maxconn 30000
server c1 10.101.13.74:80 maxconn 10000
server c2 10.101.13.78:80 maxconn 10000
server c3 10.101.13.82:80 maxconn 10000
listen stats :8888
mode http
stats enable
stats hide-version
stats uri /
and
# this config needs haproxy-1.1.28 or haproxy-1.2.1
global
log /dev/log local0 info
log /dev/log local0 notice
maxconn 20000
user y
group y
#debug
defaults
log global
mode http
option httplog
option dontlognull
option forwardfor
retries 3
option redispatch
option http-server-close
maxconn 20000
contimeout 10000
clitimeout 50000
srvtimeout 50000
balance leastconn
listen a_cluster 255.255.255.151:80
maxconn 20000
server a1 10.101.13.68:80 maxconn 10000
server a2 10.101.13.66:80 maxconn 10000
listen stats :8888
mode http
stats enable
stats hide-version
stats uri /
My sysctl.conf has the following changes:
fs.file-max = 1000000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 1
Thank you,
Andy
