Re: intermittent 503 errors

2009-09-30 Thread Duncan Hall

Willy,

Sorry for the delay in getting back to you.

I found the issue, it was caused by a misconfiguration in HA Heartbeat 
where the failover server kept trying to steal the IP address of the 
HAproxy server.


I should have known there was nothing wrong with HAproxy!

Thanks for your help and hard work.

regards,

Duncan

Willy Tarreau wrote:

Hi Duncan,

On Wed, Sep 16, 2009 at 10:15:55PM +1000, Duncan wrote:
  

Hi All,

Our load testing is bringing up intermittent 503 errors after about 15 
minutes of testing. It could be an issue with our application but I 
thought I'd post a copy of my config in the hope that somebody could run 
an eye over it as a bit of a sanity check. We are running on Linux 
CentOS 5.3 using the stock CentOS 2.6.18-164.el5 kernel and haproxy 1.3.20


Any suggestions? Thanks in advance.



Do you see the servers go down on the stats page ? If so, it could
mean that your health-checks are not tolerant enough and that you
should increase their interval (or timeout) to grant the servers
more time to respond.

If you don't see the servers go down, it is possible that from time
to time your machine is running out of source ports, or that it cannot
connect outside due to a conntrack table full (check dmesg).

Also, take a look at the logs which report a 503. The session termination
flags will be of great help. For instance, it's possible that a server's
queue is filling and that sessions are expired from it after the connect
timeout, eventhough it's not very likely in your case due to your already
large timeout. You should use timeout queue for the queue BTW.

Hoping this helps,
Willy

  





Re: intermittent 503 errors

2009-09-20 Thread Willy Tarreau
Hi Duncan,

On Wed, Sep 16, 2009 at 10:15:55PM +1000, Duncan wrote:
 Hi All,
 
 Our load testing is bringing up intermittent 503 errors after about 15 
 minutes of testing. It could be an issue with our application but I 
 thought I'd post a copy of my config in the hope that somebody could run 
 an eye over it as a bit of a sanity check. We are running on Linux 
 CentOS 5.3 using the stock CentOS 2.6.18-164.el5 kernel and haproxy 1.3.20
 
 Any suggestions? Thanks in advance.

Do you see the servers go down on the stats page ? If so, it could
mean that your health-checks are not tolerant enough and that you
should increase their interval (or timeout) to grant the servers
more time to respond.

If you don't see the servers go down, it is possible that from time
to time your machine is running out of source ports, or that it cannot
connect outside due to a conntrack table full (check dmesg).

Also, take a look at the logs which report a 503. The session termination
flags will be of great help. For instance, it's possible that a server's
queue is filling and that sessions are expired from it after the connect
timeout, eventhough it's not very likely in your case due to your already
large timeout. You should use timeout queue for the queue BTW.

Hoping this helps,
Willy