We are running dual HAProxy machines as our load balancers for our web
application, with keepalived for failover. This is the 2nd time that both
HAProxy instances have died in production with no indication as to why.
The load on the servers at the time was extremely low. We routinely do
MUCH MUCH more traffic than we had at the time of the last crash. Our
environment will stay up for months at a time without a hickup, and then
boom.
Our environment
- Primary HAProxy version version 1.5-dev17
- Backup HAProxy version 1.5-dev19
- Primary uname -a: Linux lb-01 2.6.32-279.19.1.el6.x86_64 #1 SMP Wed Dec
19 07:05:20 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
- Backup uname -a: Linux lb-02 2.6.32-358.18.1.el6.x86_64 #1 SMP Wed Aug 28
17:19:38 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
- Dedicated machine with 8GB of RAM
Last successful HTTP request log entry:
Nov 10 19:02:42 localhost haproxy[10926]:
201.95.120.75:59875[10/Nov/2013:19:02:41.803] lb-01~ cluster-01/app-01
251/0/1/28/280 200 235
--VN 132/130/2/1/0 0/0 "POST /myappcontext/some/servlet/path HTTP/1.1"
HAProxy error log before crash:
Nov 10 19:02:32 localhost haproxy[10926]: backend chat-cluster-01 has
no server available!
Nov 10 19:02:43 lb-01 Keepalived_vrrp[21411]: VRRP_Script(chk_haproxy)
failed
HAProxy Config:
global
log 127.0.0.1 local0
log 127.0.0.1 local1 notice
#Adds a global syslog server. Up to two global servers can be
defined. They
#will receive logs for startups and exits, as well as all logs from
proxies
#configured with "log global". An optional level can be specified
to filter
#outgoing messages. By default, all messages are sent.
#An IPv4 address optionally followed by a colon and a UDP port. If
#no port is specified, 514 is used by default (the standard syslog
port).
maxconn 40000
#Sets the maximum per-process number of concurrent connections to
<number>. It
#is equivalent to the command-line argument "-n". Proxies will stop
accepting
#connections when this limit is reached. The "ulimit-n" parameter is
#automatically adjusted according to this value. See also "ulimit-n"
user haproxy
#Changes the process' user ID to <number>. It is recommended that
the user ID
#is dedicated to HAProxy or to a small set of similar daemons.
HAProxy must
#be started with superuser privileges in order to be able to switch
to another
#one. See also "gid" and "user".
group haproxy
#Changes the process' group ID to <number>. It is recommended that
the group
#ID is dedicated to HAProxy or to a small set of similar daemons.
HAProxy must
#be started with a user belonging to this group, or with superuser
privileges.
#See also "group" and "uid".
daemon
#Makes the process fork into background. This is the recommended
mode of
#operation. It is equivalent to the command line "-D" argument. It
can be
#disabled by the command line "-db" argument.
#debug
#NO NEED TO ENABLE - krams
#Enables debug mode which dumps to stdout all exchanges, and
disables forking
#into background. It is the equivalent of the command-line argument
"-d". It
#should never be used in a production configuration since it may
prevent full
#system startup.
#quiet
#NO NEED TO ENABLE - krams
#Do not display any message during startup. It is equivalent to the
command-
#line argument "-q".
stats socket /tmp/haproxy
defaults
log global
#Enable per-instance logging of events and traffic.
#global should be used when the instance's logging parameters are
the
#same as the global ones. This is the most common usage. "global"
#replaces <address>, <facility> and <level> with those of the log
#entries found in the "global" section. Only one "log global"
#statement may be used per instance, and this form takes no other
#parameter.
mode http
#Set the running mode or protocol of the instance
#The instance will work in HTTP mode. The client request will be
#analyzed in depth before connecting to any server. Any request
#which is not RFC-compliant will be rejected. Layer 7 filtering,
#processing and switching will be possible. This is the mode which
#brings HAProxy most of its value.
option httpclose
option httplog
#Enable logging of HTTP request, session state and timers
option dontlognull
#Enable or disable logging of null connections
retries 3
#Set the number of retries to perform on a server after a
connection failure
option redispatch
#Enable or disable session redistribution in case of connection
failure
maxconn 35500
#Fix the maximum number of concurrent connections on a frontend
#This value should not exceed the global maxconn
timeout connect 5s
#Set the maximum time to wait for a connection attempt to a server
to succeed.
timeout client 600s
#Set the maximum inactivity time on the client side.
#An unspecified timeout results in an infinite timeout, which
#is not recommended. Such a usage is accepted and works but reports
a warning
#during startup because it may results in accumulation of expired
sessions in
#the system if the system's timeouts are not configured either.
timeout server 600s
#Set the maximum inactivity time on the server side.
# Don't queue requests too long if saturated.
timeout queue 60s
#balance roundrobin
#NO NEED TO ENABLE. IT'S THE DEFAULT - krams
#The load balancing algorithm of a backend is set to roundrobin
when no other
#algorithm, mode nor option have been set
stats enable
stats hide-version
stats show-node
stats show-legends
stats realm LProxy\ LStatistics
stats uri /lproxy?stats
stats auth user:pass
option contstats
listen stats 10.10.10.2:9090
mode http
log global
frontend unsecured *:80
redirect location https://www.mysite.com/public/home
frontend lb-01
mode http
bind *:443 ssl ca-file /opt/haproxy-ssl/certs/gd_bundle.pem crt
/opt/haproxy-ssl/certs/www.mysite.com.pem
# other (self described) options are: [ciphers <suite>] [nosslv3]
[notlsv1]
acl chat url_sub /chat
acl chatsocket url_sub /chat_sock
use_backend chat-cluster-01 if chat
use_backend chat-cluster-01 if chatsocket
default_backend cluster-01
backend chat-cluster-01
mode http
option httpchk GET /public/service_health
http-check expect rstring Service OK
timeout check 5000s
errorfile 503 /opt/haproxy-ssl/errors/503.http
option forwardfor
#Enable insertion of the X-Forwarded-For header to requests sent to
servers
#Since HAProxy works in reverse-proxy mode, the servers see its IP
address as
#their client address. This is sometimes annoying when the client's
IP address
#is expected in server logs. To solve this problem, the well-known
HTTP header
#"X-Forwarded-For" may be added by HAProxy to all requests sent to
the server.
cookie JSESSIONID prefix
#Enable cookie-based persistence in a backend
#server <name> <address>[:port] [param*]
#Please refer to section 5 for more details.
server chat-01 10.10.10.31:8080 cookie chat-01 check inter 500ms
downinter 10s fall 1 rise 2
#Declare a server in a backend
#server <name> <address>[:port] [param*]
#<param*> is a list of parameters for this server. The "server"
keywords
#accepts an important number of options and has a complete section
#dedicated to it. Please refer to section 5 for more details.
backend cluster-01
mode http
option httpchk GET /public/service_health
http-check expect rstring Service OK
timeout check 5000s
errorfile 503 /opt/haproxy-ssl/errors/503.http
option forwardfor
#Enable insertion of the X-Forwarded-For header to requests sent to
servers
#Since HAProxy works in reverse-proxy mode, the servers see its IP
address as
#their client address. This is sometimes annoying when the client's
IP address
#is expected in server logs. To solve this problem, the well-known
HTTP header
#"X-Forwarded-For" may be added by HAProxy to all requests sent to
the server.
balance roundrobin
cookie JSESSIONID prefix
#Enable cookie-based persistence in a backend
#server <name> <address>[:port] [param*]
#Please refer to section 5 for more details.
server app-01 10.10.10.11:8080 cookie app-01 check inter 500ms
downinter 10s fall 1 rise 2
server app-02 10.10.10.12:8080 cookie app-02 check inter 500ms
downinter 10s fall 1 rise 2
#server batch-01 10.10.10.31:8080 cookie batch-01 check inter 500ms
downinter 10s fall 1 rise 2
#Declare a server in a backend
#server <name> <address>[:port] [param*]
#<param*> is a list of parameters for this server. The "server"
keywords
#accepts an important number of options and has a complete section
#dedicated to it. Please refer to section 5 for more details.
Any ideas why HAProxy SSL crashing and how I can prevent it?
Thanks so much!
