Chris,
Thanks for the swift reply.
I¹m running iptables on this server as well to provide basic firewalling.
Can I safely disable conntrack or any other netfilter modules?
I¹ve upgraded our staging site for some testing and will upgrade in
production tonight. I¹ve included my haproxy.cfg below. It¹s a bit lengthy,
but http_default is the important backend.
Michael Marano
------
global
log 127.0.0.1 local0 info # frontend logs -> connections
log 127.0.0.1 local1 notice notice # backend logs -> UP and DOWN
maxconn 100000
#chroot /usr/share/haproxy
user haproxy
group haproxy
spread-checks 5
daemon
#debug
#quiet
defaults http-in
log global
mode http
option httplog
option dontlognull
option forwardfor
option httpclose
retries 3
option redispatch
maxconn 100000
timeout connect 5000
timeout http-request 5000
timeout client 50000
timeout server 50000
stats enable
#stats scope .
stats uri /haproxy?stats
stats realm Haproxy\ Statistics
stats auth **********:*************
####################
#
# frontends
#
####################
frontend http_proxy
bind :80
log global
capture request header Host len 30
####################
# access control rules
####################
# static
acl static_dom hdr_dom(host) static
acl beta_static_dom hdr_beg(host) static.beta.gamesradar.com
acl base_static_dom hdr_beg(host) static.gamesradar.com
# api
acl api_dom hdr_dom(host) api
acl beg_api_dom hdr_beg(host) api
# user images
acl m1_dom hdr_dom(host) m1
acl m2_dom hdr_dom(host) m2
acl m3_dom hdr_dom(host) m3
# forums
acl forums_url url_beg /forums
acl forums_dom hdr_dom(host) forums
acl forum_dom hdr_dom(host) forum
# newsletterapi
acl newsletterapi_url url_beg /newsletterapi
####################
# backend mappings
####################
# static
use_backend static_http if static_dom
use_backend static_http if beta_static_dom
use_backend static_http if base_static_dom
# api
use_backend tomcat_http_default if api_dom
use_backend tomcat_http_default if beg_api_dom
# user images
use_backend m1_http if m1_dom
use_backend m2_http if m2_dom
use_backend m3_http if m3_dom
# forums
use_backend forums_http if forums_url
# newsletterapi
use_backend tomcat_http_default if newsletterapi_url
# default
default_backend http_default
frontend tomcat_http_proxy
bind :8080
log global
capture request header Host len 30
# default
default_backend tomcat_http_default
####################
#
# backends
#
####################
backend http_default
option httpchk OPTIONS /test.jsp HTTP/1.1\r\nHost:\
www.gamesradar.com
option redispatch
balance roundrobin
#fullconn 1000
server a1_w a1.gamesradar.com:80 check port 80 inter 4000 fall 3 rise
2 maxconn 350
server a2_w a2.gamesradar.com:80 check port 80 inter 4000 fall 3 rise
2 maxconn 350
server a3_w a3.gamesradar.com:80 check port 80 inter 4000 fall 3 rise
2 maxconn 350
backend tomcat_http_default
option httpchk OPTIONS /test.jsp HTTP/1.1\r\nHost:\
www.gamesradar.com
option redispatch
balance roundrobin
#fullconn 1000
server a1_t a1.gamesradar.com:8080 check port 8080 inter 4000 fall 3
rise 2 maxconn 195
server a2_t a2.gamesradar.com:8080 check port 8080 inter 4000 fall 3
rise 2 maxconn 195
server a3_t a3.gamesradar.com:8080 check port 8080 inter 4000 fall 3
rise 2 maxconn 195
backend forums_http
option httpchk HEAD /forums/help/help-text.html HTTP/1.1\r\nHost:\
www.gamesradar.com
option redispatch
balance roundrobin
#fullconn 1000
server f1_w f.gamesradar.com:80 check inter 2000 maxconn 295
backend static_http
option httpchk HEAD /test.html
option redispatch
balance roundrobin
server w1_static w1.gamesradar.com:81 check port 81 inter 2000
maxconn 1020
server w2_static w2.gamesradar.com:81 check port 81 inter 2000
maxconn 1020
# primary/failover setup for m1.gamesradar.com
backend m1_http
option httpchk OPTIONS / HTTP/1.1\r\nHost:\ m1.gamesradar.com
option redispatch
server m1_w a1.gamesradar.com:80 check inter 2000 maxconn 300
server m1_bk1 a2.gamesradar.com:80 check inter 2000 backup maxconn
300
server m1_bk2 a3.gamesradar.com:80 check inter 2000 backup maxconn
300
# primary/failover setup for m2.gamesradar.com
backend m2_http
option httpchk OPTIONS / HTTP/1.1\r\nHost:\ m2.gamesradar.com
option redispatch
server m2_w a2.gamesradar.com:80 check inter 2000 maxconn 300
server m2_bk1 a1.gamesradar.com:80 check inter 2000 backup maxconn
300
server m2_bk2 a3.gamesradar.com:80 check inter 2000 backup maxconn
300
# primary/failover setup for m3.gamesradar.com
backend m3_http
option httpchk OPTIONS / HTTP/1.1\r\nHost:\ m3.gamesradar.com
option redispatch
server m3_w a3.gamesradar.com:80 check inter 2000 maxconn 300
server m3_bk1 a1.gamesradar.com:80 check inter 2000 backup maxconn
300
server m3_bk2 a2.gamesradar.com:80 check inter 2000 backup maxconn
300
From: Chris Sarginson <[email protected]>
Date: Tue, 6 Oct 2009 18:56:19 +0100
To: Michael Marano <[email protected]>
Cc: "<[email protected]>" <[email protected]>
Subject: Re: Kernel tuning recommendations
The first piece of advice you will receive is to disable the nf_conntrack
module :)
That should give a performance improvement. I will send on my sysctl
parameters if possible later. I would also recommend you sending in your
haproxy config, and upgrading to the haproxy version in epel repos, if you
don't wish to compile from source.
Chris
Sent from my iPhone
On 6 Oct 2009, at 18:25, Michael Marano <[email protected]> wrote:
> We¹ve completed a move to Rackspace Cloud and are now using HAproxy as our
> load balancer. Haproxy is a Phenomenal piece of software.
>
> The primary issue I¹ve noticed from haproxy is that my backends are frequently
> going DOWN/UP, and we¹re having some long request times as well as serving
> occaisional 504¹s. I¹ve been doing my research and understand that I need to
> do some system tuning via sysctl to get things running properly. All
> references have different reccomendations on what parameters to tune, and I¹m
> a bit hesitant to copy/paste from multiple resources.
>
> Is there a baseline set of recommended tunings that I can apply as a first
> response before digging into the gritty details?
>
> I¹ve attached a bunch of details below. Thanks for any help you can provide.
>
> Michael
>
> -------
>
> [mmar...@w1 w1]$ cat /etc/redhat-release
> CentOS release 5.3 (Final)
> [mmar...@w1 w1]$ uname -a
> Linux w1.gamesradar.com <http://w1.gamesradar.com> 2.6.24-23-xen #1 SMP Mon
> Jan 26 03:09:12 UTC 2009 x86_64 x86_64 x86_64 GNU/Linux
> [mmar...@w1 w1]$ /usr/sbin/haproxy -v
> HA-Proxy version 1.3.18 2009/05/10
>
> Here¹s what I¹m finding in /var/log/messages:
>
> Oct 2 23:12:38 w1 kernel: [1556670.291082] printk: 482 messages suppressed.
> Oct 2 23:12:38 w1 kernel: [1556670.291102] nf_conntrack: table full, dropping
> packet.
> Oct 3 01:34:49 w1 kernel: [1566552.616316] possible SYN flooding on port 80.
> Sending cookies.
> Oct 3 03:19:52 w1 kernel: [1572838.886342] printk: 294 messages suppressed.
>
>
> [mmar...@w1 w1]$ sudo /sbin/sysctl -a | grep ^net
> net.ipv4.tcp_timestamps = 1
> net.ipv4.tcp_window_scaling = 1
> net.ipv4.tcp_sack = 1
> net.ipv4.tcp_retrans_collapse = 1
> net.ipv4.ip_forward = 0
> net.ipv4.ip_default_ttl = 64
> net.ipv4.ip_no_pmtu_disc = 0
> net.ipv4.ip_nonlocal_bind = 0
> net.ipv4.tcp_syn_retries = 5
> net.ipv4.tcp_synack_retries = 5
> net.ipv4.tcp_max_orphans = 65536
> net.ipv4.tcp_max_tw_buckets = 180000
> net.ipv4.ipfrag_high_thresh = 262144
> net.ipv4.ipfrag_low_thresh = 196608
> net.ipv4.ip_dynaddr = 0
> net.ipv4.ipfrag_time = 30
> net.ipv4.tcp_keepalive_time = 7200
> net.ipv4.tcp_keepalive_probes = 9
> net.ipv4.tcp_keepalive_intvl = 75
> net.ipv4.tcp_retries1 = 3
> net.ipv4.tcp_retries2 = 15
> net.ipv4.tcp_fin_timeout = 60
> net.ipv4.tcp_syncookies = 1
> net.ipv4.tcp_tw_recycle = 0
> net.ipv4.tcp_abort_on_overflow = 0
> net.ipv4.tcp_stdurg = 0
> net.ipv4.tcp_rfc1337 = 0
> net.ipv4.tcp_max_syn_backlog = 1024
> net.ipv4.ip_local_port_range = 32768 61000
> net.ipv4.icmp_echo_ignore_all = 0
> net.ipv4.icmp_echo_ignore_broadcasts = 1
> net.ipv4.icmp_ignore_bogus_error_responses = 1
> net.ipv4.icmp_errors_use_inbound_ifaddr = 0
> net.ipv4.route.min_delay = 2
> net.ipv4.route.max_delay = 10
> net.ipv4.route.gc_thresh = 262144
> net.ipv4.route.max_size = 4194304
> net.ipv4.route.gc_min_interval = 0
> net.ipv4.route.gc_min_interval_ms = 500
> net.ipv4.route.gc_timeout = 300
> net.ipv4.route.gc_interval = 60
> net.ipv4.route.redirect_load = 5
> net.ipv4.route.redirect_number = 9
> net.ipv4.route.redirect_silence = 5120
> net.ipv4.route.error_cost = 250
> net.ipv4.route.error_burst = 1250
> net.ipv4.route.gc_elasticity = 8
> net.ipv4.route.mtu_expires = 600
