HAProxy - 504 Gateway Timeout error.
Hi,
We have recently migrated our game servers from Linux to FreeBSD. We have 8
web servers running in jails, with HAProxy as load balancer. We also have
CARP configured in case of network failover.
carp is running as master on the 1st server(webm01), and backup on the 2nd
server(webm02). haproxy on both servers are actively running, though only
one is working at a time, depending on which server with carp acting as
master. Both servers have pf running as well.
We are running FreeBSD 8.2-RELEASE, haproxy-1.4.15, apache-2.2.19 and the
game is php coded.
Our network architecture is as follows. There is a backend database running
as well on a jail in a different server, which I excluded from the diagram
(hope the ascii diagram will be displayed well in the mail):
+- wj01
|
(webm01) |-- wj02
user carp haproxy --+
| |-- wj03
| |
| +- wj04
|
| +- wj05
| |
| |- wj06
carp haproxy --+
(webm02) |- wj07
|
+- wj08
Our main problem at the moment is a lot of users (more than a hundred users)
have complained that they are getting a 504 Gateway Timeout error. This
normally happens at night (CEST), when most players start playing the game.
However, the load of our servers are consistently low at all time.
At the moment there is no obvious pattern as to when this error occurs.
Here is our haproxy.conf:
global
log /var/run/log local0 notice
maxconn 4096
daemon
chroot /var/run/haproxy
user haproxy
group haproxy
stats socket /var/run/haproxy/haproxy.sock uid 1005 gid 1005
defaults
logglobal
modehttp
optionhttpclose
optionforwardfor
optionhttplog
optiontcplog
optiondontlognull
optiontcpka
retries3
option redispatch
maxconn2000
timeout connect5000
timeout client 5
timeout server5
listenwebjailfarm 78.xx.xx.xx:80
mode http
cookieSERVERID insert nocache indirect
balanceroundrobin
option httpclose
option forwardfor
option httpchk HEAD / HTTP/1.0
stats uri /haproxy-status
stats enable
stats auth admin:password
serverwj01 192.168.30.10:80 http://192.168.30.10/ cookie A weight
10 check inter 2000 rise 2 fall 2
serverwj02 192.168.30.20:80 http://192.168.30.20/ cookie B weight
10 check inter 2000 rise 2 fall 2
serverwj03 192.168.30.30:80 http://192.168.30.30/ cookie C weight
10 check inter 2000 rise 2 fall 2
serverwj04 192.168.30.40:80 http://192.168.30.40/ cookie D weight
10 check inter 2000 rise 2 fall 2
serverwj05 192.168.30.50:80 http://192.168.30.50/ cookie E weight
10 check inter 2000 rise 2 fall 2
serverwj06 192.168.30.60:80 http://192.168.30.60/ cookie F weight
10 check inter 2000 rise 2 fall 2
serverwj07 192.168.30.70:80 http://192.168.30.70/ cookie G weight
10 check inter 2000 rise 2 fall 2
serverwj08 192.168.30.80:80 http://192.168.30.80/ cookie H weight
10 check inter 2000 rise 2 fall 2
##
And here is our pf.conf (the exact same pf is running on webm02, only the
IPs changed accordingly):
### macros
webm01 = 78.xx.xx.xx
db = 10.10.10.101
carp_dev = carp0
ext_if = igb0
jail_if = igb0:0
trusted = { 192.168.30.0/24, 10.10.10.0/24, 78.xx.xx.xx/xx, 85.xx.xx.xx/xx
}
tcp_services = { x, 4949 }
ssh_ports = { x, x, x, x }
icmp_types = { echoreq, unreach }
# jails
wj01 = 192.168.30.10
wj02 = 192.168.30.20
wj03 = 192.168.30.30
wj04 = 192.168.30.40
jails = { $wj01 $wj02 $wj03 $wj04 }
### normalization
scrub in all
### translation
nat on $ext_if from $jails to !10.10.10.0/24 - ($jail_if)
rdr pass on $ext_if inet proto tcp from any to $webm01 port x - $wj01
### ssh redirect
rdr pass on $ext_if inet proto tcp from any to $webm01 port x - $wj02
rdr pass on $ext_if inet proto tcp from any to $webm01 port x - $wj03
rdr pass on $ext_if inet proto tcp from any to $webm01 port x - $wj04
rdr pass on $ext_if inet proto tcp from any to ($carp_dev) port 80 -
$webm01
### filtering - drop incoming everything
block in all
block return
### keep state of outgoing connections
pass out keep state
### skip loopback interface
set skip on { lo0 }
### spoofing protection for all interfaces
block
Redirection based on source URL (using parameters)?
Dear All, I am trying to set up redirection rules that use part of the source URL on the destination URL. Example: /(..)/home.htm - /home/$1 Is it possible to do this in HAProxy? Would you please throw some pointers on how? Thank you so much. -- Kind Regards, RODRIGUEZ Unai Senior Administrator, Systems MUSIC Group Services SG (Pte.) Ltd IP Phone: 60651 ext 1825 Tel: +65 6845 1800 ext 1825 Email: infoservsys...@music-group.com Web: www.music-group.com | www.behringer.com | www.bugera-amps.com youtube.com/behringer twitter.com/behringer facebook.com/behringer myspace.com/behringer flickr.com/behringerrocks * Build Teamwork * Take Ownership * Don’t Waste Resources * Clean Workplace = Clean Mind * Respect Guidelines and Policies * Improve Yourself and Help Others * Don’t Forget to Smile and Say Thank You This email is intended exclusively for the addressee(s) named above and may contain privileged and confidential information. If you are not (among) the intended recipient(s), you may not copy, utilize or distribute any of the information contained herein. If you have received this email in error, please notify us immediately via return email and delete the original from your mailbox. Thank you.
Re: HAProxy - 504 Gateway Timeout error.
Try adding: optionhttplog under your listen, I am not sure what haproxy does if you say tcplog after saying httplog, so you want to make sure have httplog since those log entries provide more info. Run with option httplog on the listen during the busy time and post some examples of the full log entries for the 504s - obfuscated as needed. There are 6 or 8 fields that should give some clues to loading, timing, tcp connection disposition and other potential issues. If you switch to a frontend/backend config, I think the haproxy stats page provides slightly more info, but I don't use listen so I am not positive. If you have a heavy / page, even HEADs every 2 seconds might be some load (because AFAIK php has to spin the whole page to know if it has changed depending on the frameworks used), maybe not. Remember, load can be low on the machines/jails and they might still be near or at their limit for sockets, file descriptors, etc - so be sure to check those. Also you can obviously watch for the errors as they happen with something like: tail -f /var/run/log | fgrep 504 | more On 7/6/11 2:44 AM, Gi Dot wrote: Hi, We have recently migrated our game servers from Linux to FreeBSD. We have 8 web servers running in jails, with HAProxy as load balancer. We also have CARP configured in case of network failover. carp is running as master on the 1st server(webm01), and backup on the 2nd server(webm02). haproxy on both servers are actively running, though only one is working at a time, depending on which server with carp acting as master. Both servers have pf running as well. We are running FreeBSD 8.2-RELEASE, haproxy-1.4.15, apache-2.2.19 and the game is php coded. Our network architecture is as follows. There is a backend database running as well on a jail in a different server, which I excluded from the diagram (hope the ascii diagram will be displayed well in the mail): +- wj01 | (webm01) |-- wj02 user carp haproxy --+ | |-- wj03 | | | +- wj04 | | +- wj05 | | | |- wj06 carp haproxy --+ (webm02) |- wj07 | +- wj08 Our main problem at the moment is a lot of users (more than a hundred users) have complained that they are getting a 504 Gateway Timeout error. This normally happens at night (CEST), when most players start playing the game. However, the load of our servers are consistently low at all time. At the moment there is no obvious pattern as to when this error occurs. Here is our haproxy.conf: global log /var/run/log local0 notice maxconn 4096 daemon chroot /var/run/haproxy user haproxy group haproxy stats socket /var/run/haproxy/haproxy.sock uid 1005 gid 1005 defaults logglobal modehttp optionhttpclose optionforwardfor optionhttplog optiontcplog optiondontlognull optiontcpka retries3 option redispatch maxconn2000 timeout connect5000 timeout client 5 timeout server5 listenwebjailfarm 78.xx.xx.xx:80 mode http cookieSERVERID insert nocache indirect balanceroundrobin option httpclose option forwardfor option httpchk HEAD / HTTP/1.0 stats uri /haproxy-status stats enable stats auth admin:password serverwj01 192.168.30.10:80 http://192.168.30.10/ cookie A weight 10 check inter 2000 rise 2 fall 2 serverwj02 192.168.30.20:80 http://192.168.30.20/ cookie B weight 10 check inter 2000 rise 2 fall 2 serverwj03 192.168.30.30:80 http://192.168.30.30/ cookie C weight 10 check inter 2000 rise 2 fall 2 serverwj04 192.168.30.40:80 http://192.168.30.40/ cookie D weight 10 check inter 2000 rise 2 fall 2 serverwj05 192.168.30.50:80 http://192.168.30.50/ cookie E weight 10 check inter 2000 rise 2 fall 2 serverwj06 192.168.30.60:80 http://192.168.30.60/ cookie F weight 10 check inter 2000 rise 2 fall 2 serverwj07 192.168.30.70:80 http://192.168.30.70/ cookie G weight 10 check inter 2000 rise 2 fall 2 serverwj08 192.168.30.80:80 http://192.168.30.80/ cookie H weight 10 check inter 2000 rise 2 fall 2 ## And here is our pf.conf (the exact same
Re: HAProxy - 504 Gateway Timeout error.
hi, Your maxconn seems a bit low if you have a lot of clients... Maybe you should try increasing it or at lease increase the queue timeout. As hank said, turn on http log, it will provide you very interesting information about your issue. cheers
Redirection based on source URL (using parameters)?
Dear All, I am trying to set up redirection rules that use part of the source URL on the destination URL. Example: /(..)/home.htm - /home/$1 Is it possible to do this in HAProxy? Would you please throw some pointers on how? Thank you so much. -- Kind Regards, RODRIGUEZ Unai
