Hi,
On Thu, Jan 20, 2011 at 11:43:25PM +0100, [email protected] wrote:
> Hi All,
>
> I've got a working HAProxy setup however with a strange issue.
>
> Every now and then we get client responses that take anywhere from 15-60
> seconds for pages that are processed very fast on the server side. The page
> partially renders and is 'waiting' for the rest of the content which
> eventually filters in. I've witnessed the response on the server side and
> it's completed.
Indeed that's quite strange. I see that you have incomplete logs, could
you add "option httplog" to your defaults section and see what haproxy
reports in its logs for the faulty requests ? Since you have a stats page,
please take a look at it, especially in the "errors" and "warnings" columns.
The "conn", "retr" and "redis" will indicate whether there are connection
errors or not.
> It also only happens with then are 'multiple' servers configured, any clients
> with a cookie for the additional servers get the intermittent slow response.
> The slow response is also very intermittent. A client could actually use the
> site and never encounter it.
Could it be possible that only one of your servers has trouble and that
adding it to the farm makes the problem appear ? Also, what's your load
(bandwidth, session rate, concurrent sessions) ? Your description could
match three possible common low-level issues :
- at high session rates, it's possible that you fill a conntrack table
in a server. "dmesg" on that server will tell you if this happens.
- at high session rates, if the TCP settings are two low on your haproxy
box (source port range too short, tcp_tw_reuse not set, ...) and the
box fails to allocate a new port to connect to a server. The logs should
report "sC" or "SC" indicating connection failures to the server.
- at high data rates, it's possible that a faulty NIC regularly resets
itself and that the switch port at the other end blocks the port for
30-50 seconds to check if there's some spanning tree.
> I have analyzed our application and rebuilt servers in an attempt to fix this
> to no avail.
>
> I realize that this a pretty open ended post but am wondering if anyone has
> any ideas on why this would be occurring.
>
> We are using stunnel for SSL termination however that is for only 2 pages in
> the site.
Do you see the problem on both services or only on HTTP or only on SSL ?
> My config below:
>
> global
> daemon
> maxconn 20000
> nbproc 2
You should remove "nbproc 2" above. It makes the config hard to debug and
really provides no benefit at "normal" loads.
> log 127.0.0.1 syslog
>
> defaults
> log global
> option dontlognull
> option redispatch
> option http-server-close
> option http-pretend-keepalive
> retries 3
> timeout client 7000
> timeout server 60000
> timeout connect 5000
>
> listen web_http :80
> mode http
> balance roundrobin
> acl blocked src X.X.X.X X.X.X.X
> block if blocked
> cookie ODSERVER insert nocache indirect
> option httpchk GET /check.cfm HTTP/1.0
> option forwardfor except X.X.X.X
> reqadd X-Forwarded-Proto:\ http
> server server3 X.X.X.X:80 cookie server3 weight 10 maxconn 1000 check
> inter 5000
> #server server5 X.X.X.X:80 cookie server5 weight 5 maxconn 1000 check
> inter 5000
> server server6 X.X.X.X:80 cookie server6 weight 10 maxconn 1000 check
> inter 5000
> #server server7 X.X.X.X:80 cookie server7 weight 10 maxconn 1000 check
> inter 5000
> server server9 X.X.X.X:80 cookie server9 weight 10 maxconn 1000 check
> inter 5000
>
> listen web_https X.X.X.X:81
> mode http
> balance roundrobin
> cookie ODSERVER insert nocache indirect
> option httpchk GET /check.cfm HTTP/1.0
> option forwardfor except X.X.X.X
> reqadd X-Forwarded-Proto:\ https
> reqadd SSL-TERMINATION:\ ON
> server server3 X.X.X.X:81 cookie server3 weight 10 maxconn 1000 check
> port 80 inter 5000
> #server server5 X.X.X.X:81 cookie server5 weight 5 maxconn 1000 check
> port 80 inter 5000
> server server6 X.X.X.X:81 cookie server6 weight 10 maxconn 1000 check
> port 80 inter 5000
> #server server7 X.X.X.X:81 cookie server7 weight 10 maxconn 1000 check
> port 80 inter 5000
> server server9 X.X.X.X:81 cookie server9 weight 10 maxconn 1000 check
> port 80 inter 5000
>
> listen lb_stats X.X.X.X:8443
> mode http
> stats uri /monitor
> stats auth admin:blah
> #stats refresh 5s
Regards,
Willy
