Hi Andy, Can't see anything wrong with your configuration. How about the server running haproxy? anything about that regarding cpu etc?
I wouldn't be using net.ipv4.tcp_tw_recycle = 1 tw_recycle is a bit dodgy and can give some unwanted side-effects. tw_reuse = 1 should be sufficient. How does a tcpdump look? Since you get this issue on the stats get a dump on that. What setting are there on the box regarding SYN backlog? net.core.somaxconn & net.ipv4.tcp_max_syn_backlog Best regards, pelle On Thu, Oct 10, 2013 at 7:16 PM, Andy M. <[email protected]> wrote: > Hello, > > I am trying to configure an HAProxy And seem to be running into a problem > where the HAProxy will spike. I have a high throughput server cluster, > which will need to handle about 10-20k QPS. I believe that HAProxy should > be able to handle that quite easily from what I have read. I am running > HAProxy on Ubuntu 12.04LTS. I am running an a Xeon 1270, with 8GB of > RAM(Which i believe is all overkill). I am doing a simple load balance > between two or three servers(I have two server clusters, one with two > servers, one with three). I am running 1.4.24. > > The reason I believe something is wrong with my haproxy is because I can > refresh my webpage a few times, and every 4-5 times I will be get a spike > around 800ms-3sec to resolve. This also happens when not hitting my > backend server, and instead hitting the HAProxy stats page. None of my > servers are anywhere close to using up their system resources(all below 20% > in both memory and CPU). I don't see much in the syslog that seems wrong > other than this message: > > Oct 10 16:48:28 haproxy rsyslogd-2177: imuxsock begins to drop message > 3912 due to rate-limiting > > I believe that has to do with the logging not being able to keep up, > rather than haproxy dropping packets. > > I was wondering if there were any suggestions for me to look into to try > and fix my problem, or suggest how to configure a HAProxy correctly to be > able to handle the load I need(preferably higher than the load I need, as > this may increase over time). I am open to changing operating systems if > need be as well. > > Here are my two config files: > > global > log /dev/log local0 info > log /dev/log local0 notice > maxconn 50000 > user y > group y > #debug > > defaults > log global > mode http > option httplog > option dontlognull > option forwardfor > retries 3 > option redispatch > option http-server-close > maxconn 30000 > contimeout 10000 > clitimeout 50000 > srvtimeout 50000 > balance leastconn > > listen c_cluster 255.255.255.146:80 > maxconn 30000 > server c1 10.101.13.74:80 maxconn 10000 > server c2 10.101.13.78:80 maxconn 10000 > server c3 10.101.13.82:80 maxconn 10000 > listen stats :8888 > mode http > stats enable > stats hide-version > stats uri / > > > and > > # this config needs haproxy-1.1.28 or haproxy-1.2.1 > global > log /dev/log local0 info > log /dev/log local0 notice > maxconn 20000 > user y > group y > #debug > > defaults > log global > mode http > option httplog > option dontlognull > option forwardfor > retries 3 > option redispatch > option http-server-close > maxconn 20000 > contimeout 10000 > clitimeout 50000 > srvtimeout 50000 > balance leastconn > > listen a_cluster 255.255.255.151:80 > maxconn 20000 > server a1 10.101.13.68:80 maxconn 10000 > server a2 10.101.13.66:80 maxconn 10000 > listen stats :8888 > mode http > stats enable > stats hide-version > stats uri / > > > My sysctl.conf has the following changes: > > fs.file-max = 1000000 > net.ipv4.tcp_tw_reuse = 1 > net.ipv4.tcp_tw_recycle = 1 > > Thank you, > > Andy >
