Thanks for the response pelle, The haproxy box is running at about 10-15% cpu.
Looking at the TCP Dump, it seems the client is doing 1 or more TCP_Retransmissions of the incoming request when there is a delay. Anyone know why this would happen? Here is the requested information: root@haproxy:~# sysctl net.core.somaxconn net.core.somaxconn = 128 root@haproxy:~# sysctl net.ipv4.tcp_max_syn_backlog net.ipv4.tcp_max_syn_backlog = 2048 I removed the tcp_tw_recycle, but this did not change anything Thank you, Andy On Thu, Oct 10, 2013 at 3:44 PM, Andy M. <[email protected]> wrote: > Thanks for the response pelle, > > The haproxy box is running at about 10-15% cpu. > > Looking at the TCP Dump, it seems the client is doing 1 or more > TCP_Retransmissions of the incoming request when there is a delay. Anyone > know why this would happen? > > Here is the requested information: > root@haproxy:~# sysctl net.core.somaxconn > net.core.somaxconn = 128 > root@haproxy:~# sysctl net.ipv4.tcp_max_syn_backlog > net.ipv4.tcp_max_syn_backlog = 2048 > > I removed the tcp_tw_recycle, but this did not change anything > > Thank you, > > Andy > > > > On Thu, Oct 10, 2013 at 1:58 PM, Pär Åslund <[email protected]> wrote: > >> Hi Andy, >> >> Can't see anything wrong with your configuration. >> How about the server running haproxy? anything about that regarding cpu >> etc? >> >> I wouldn't be using net.ipv4.tcp_tw_recycle = 1 >> tw_recycle is a bit dodgy and can give some unwanted side-effects. >> tw_reuse = 1 should be sufficient. >> >> How does a tcpdump look? Since you get this issue on the stats get a dump >> on that. >> >> What setting are there on the box regarding SYN backlog? >> net.core.somaxconn & net.ipv4.tcp_max_syn_backlog >> >> Best regards, >> pelle >> >> >> On Thu, Oct 10, 2013 at 7:16 PM, Andy M. <[email protected]> wrote: >> >>> Hello, >>> >>> I am trying to configure an HAProxy And seem to be running into a >>> problem where the HAProxy will spike. I have a high throughput server >>> cluster, which will need to handle about 10-20k QPS. I believe that >>> HAProxy should be able to handle that quite easily from what I have read. >>> I am running HAProxy on Ubuntu 12.04LTS. I am running an a Xeon 1270, >>> with 8GB of RAM(Which i believe is all overkill). I am doing a simple load >>> balance between two or three servers(I have two server clusters, one with >>> two servers, one with three). I am running 1.4.24. >>> >>> The reason I believe something is wrong with my haproxy is because I can >>> refresh my webpage a few times, and every 4-5 times I will be get a spike >>> around 800ms-3sec to resolve. This also happens when not hitting my >>> backend server, and instead hitting the HAProxy stats page. None of my >>> servers are anywhere close to using up their system resources(all below 20% >>> in both memory and CPU). I don't see much in the syslog that seems wrong >>> other than this message: >>> >>> Oct 10 16:48:28 haproxy rsyslogd-2177: imuxsock begins to drop message >>> 3912 due to rate-limiting >>> >>> I believe that has to do with the logging not being able to keep up, >>> rather than haproxy dropping packets. >>> >>> I was wondering if there were any suggestions for me to look into to >>> try and fix my problem, or suggest how to configure a HAProxy correctly to >>> be able to handle the load I need(preferably higher than the load I need, >>> as this may increase over time). I am open to changing operating systems >>> if need be as well. >>> >>> Here are my two config files: >>> >>> global >>> log /dev/log local0 info >>> log /dev/log local0 notice >>> maxconn 50000 >>> user y >>> group y >>> #debug >>> >>> defaults >>> log global >>> mode http >>> option httplog >>> option dontlognull >>> option forwardfor >>> retries 3 >>> option redispatch >>> option http-server-close >>> maxconn 30000 >>> contimeout 10000 >>> clitimeout 50000 >>> srvtimeout 50000 >>> balance leastconn >>> >>> listen c_cluster 255.255.255.146:80 >>> maxconn 30000 >>> server c1 10.101.13.74:80 maxconn 10000 >>> server c2 10.101.13.78:80 maxconn 10000 >>> server c3 10.101.13.82:80 maxconn 10000 >>> listen stats :8888 >>> mode http >>> stats enable >>> stats hide-version >>> stats uri / >>> >>> >>> and >>> >>> # this config needs haproxy-1.1.28 or haproxy-1.2.1 >>> global >>> log /dev/log local0 info >>> log /dev/log local0 notice >>> maxconn 20000 >>> user y >>> group y >>> #debug >>> >>> defaults >>> log global >>> mode http >>> option httplog >>> option dontlognull >>> option forwardfor >>> retries 3 >>> option redispatch >>> option http-server-close >>> maxconn 20000 >>> contimeout 10000 >>> clitimeout 50000 >>> srvtimeout 50000 >>> balance leastconn >>> >>> listen a_cluster 255.255.255.151:80 >>> maxconn 20000 >>> server a1 10.101.13.68:80 maxconn 10000 >>> server a2 10.101.13.66:80 maxconn 10000 >>> listen stats :8888 >>> mode http >>> stats enable >>> stats hide-version >>> stats uri / >>> >>> >>> My sysctl.conf has the following changes: >>> >>> fs.file-max = 1000000 >>> net.ipv4.tcp_tw_reuse = 1 >>> net.ipv4.tcp_tw_recycle = 1 >>> >>> Thank you, >>> >>> Andy >>> >> >> > >
