Hi Maxime,
On Tue, Oct 26, 2010 at 12:47:37PM -0400, Maxime Ducharme wrote:
>
> Hi guys
>
> I am new to haproxy & list, my experience with this software is very
> good yet.
>
> Got a question about sockets tuning. We have web site running on 10
> different httpd with 2 haproxy in front.
>
> We configured 3 IPs on each haproxy, we get about 2200 req/s each, peak
> time is 3500 req/s each.
>
> Current load is very low actually on haproxy boxes, but we have noticed
> some slow access to the website. Doing analysis we found out that
> sometime opening a TCP socket on haproxy box is slower than opening a
> socket directly on one of httpd behind.
>
> The actual configuration is quite simple, here is snippet :
>
> global
> maxconn 32768
> nbproc 8
>
> defaults
> log global
> retries 3
> maxconn 32768
> contimeout 5000
> clitimeout 50000
> srvtimeout 50000
>
> listen weblb1 1.1.1.1:80
> bind 1.1.1.2:80
> bind 1.1.1.3:80
>
> mode http
> balance roundrobin
>
> option forwardfor
> option httpchk HEAD / HTTP/1.0
> option httpclose
> stats enable
> server web1 1.1.2.1:80 weight 10 check port 80
> ..
> server web10 1.1.2.10:80 weight 10 check port 80
>
>
> We put nbprocs to the same amount of CPU cores we have.
>
> We noticed problem by tracing HTTP request with curl, ex:
>
> 15:14:13.684549 * About to connect() to www.website.com port 80 (#0)
> 15:14:13.685620 * Trying 1.1.1.1... connected
> --> 3 seconds here to open TCP connection
> 15:14:16.796281 * Connected to www.website.com (1.1.1.1) port 80 (#0)
> 15:14:16.797173 > GET / HTTP/1.1
> --> httpd replies here in less than 1 second
A 3 second delay is a typical SYN retransmit.
> This issue happens sometime, not always.
>
> My question, can someone point me a direction to look for for sockets
> optimization / debugging. I am currently unable to explain why it is
> slow, I know this is not hardware related since it is very powerful box.
> I believe some tuning will make a big difference. Maybe we have kernel
> tuning to do in here, if someone can enlighten me it would be very
> appreciated.
Two things to look for :
- if you have ip_conntrack / nf_conntrack loaded, either you have to
unload it, or to properly tune it for your usage (I'd recommend the
former, it's easier).
- check sys.net.core.somaxconn. If it's 128, then your TCP stack is not
tuned for a high connection rate, and you're surely dropping incoming
connections from time to time. Try to first increase that single
parameter to 10000, restart haproxy and check if it changes anything.
Note that you don't need 8 processes with that load, it will be harder to
debug, health checks will not be synced, and stats will only be per-process.
> Another question :
>
> can I enable stats on a particular IP ?
yes, simply put the "stats enable" statement in its own listen section.
Last, with version 1.4, you can also reduce the connection rate by using
"option http-server-close" instead of "option httpclose". It will enable
keep-alive on the client side. Do that only when you have fixed your
issues, because doing so can mask the problem without fixing it, and you'll
get it again later.
Regards,
Willy
