We are using a Layer 7 style health check against a tcp backend. It's
supposedly HTTP, but for whatever reason I couldn't get haproxy http check
to work. Anyway, our check "works", but only sometimes. Over the course
of day with no traffic to the backends at all (just our layer 7 health
check) we'll see many "downs"... but if I hammer the health check using
curl to do the soap request, it always returns.
The healthcheck takes milliseconds to respond... but when haproxy "fails"
the check it of course waits the full haproxy timeout period.
I temporarily worked around the spurious "Downs" by increasing the time
between checks (again, testing outside says we need almost no time between
checks) to 4s from the default 2s and by changing falls from the default
value to 5 times.
That doesn't prevent the layer 7 check problems, just keeps haproxy from
downing the backend (I figure the combination of delayed inter and number
of falls).
Is there a problem with health checks and haproxy? Again, using a machine
gun approach on the health check service, we see no problems, but for
whatever reason, occasionally (maybe 1 out 10, could be more), the haproxy
tcp expect fails. Using tcpdump, seems we're getting the right return
though.
We're using haproxy 1.15 without pcre expressions.
Also, both checks fail occasionally, the non-ssl one and the ssl one. We
see more check failures on the non-ssl one. But both are pretty high.
qsrv1 and qsrv2 are up. We just don't want things failing over to qsrv2
(backup) unless qsrv1 is really down... and right now it's saying qsrv1 is
down a lot, and our other tests say that it never went down and is
handling healtchecks without issue.
Here's the original config (using default inter, etc):
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
log 127.0.0.1 local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
# turn on stats unix socket
stats socket /var/lib/haproxy/stats
# This disables sslv3 (POODLE bug) and removes weak ciphers
# For the most secure
#ssl-default-bind-options no-sslv3 no-tls-tickets force-tlsv12
# For the most secure
#ssl-default-bind-ciphers AES128+EECDH:AES128+EDH
ssl-default-bind-options no-sslv3
ssl-default-bind-ciphers
EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH:EDH+aRSA:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS:!RC4
tune.ssl.default-dh-param 1024
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
mode http
log global
option dontlognull
option redispatch
retries 3
timeout http-request 10s
timeout connect 5000
timeout client 50000
timeout server 50000
maxconn 20000
#
# Web stats (temporary?)
#
listen stats :8080
mode http
stats enable
stats hide-version
stats uri /
#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
frontend http *:10080
mode tcp
default_backend srv-http
option tcplog
frontend https
mode tcp
bind :10443 ssl crt
/etc/haproxy-ssl/example-combined.pem
option tcplog
default_backend srv-https
backend srv-http
option tcp-check
tcp-check send POST\ /services/hcheck\ HTTP/1.1\r\nUser-Agent:\
curl/7.37.0\r\nHost:\ qsrv1.example.com:10080\r\nAccept:\
*/*\r\nContent-Length:\ 114\r\nContent-Type:\
text/xml;charset=UTF-8\r\n\r\n<soapenv:Envelope\
xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/";>\ \ \
<soapenv:Body\ /></soapenv:Envelope>
tcp-check expect rstring HTTP/1.1\ 200\ OK.*>OK<
server qsrv1 10.1.50.52:10080 check slowstart 5000
server qsrv2 10.1.50.150:10080 check slowstart 5000 backup
backend srv-https
option tcp-check
tcp-check send POST\ /services/hcheck\ HTTP/1.1\r\nUser-Agent:\
curl/7.37.0\r\nHost:\ qsrv1.example.com:10080\r\nAccept:\
*/*\r\nContent-Length:\ 114\r\nContent-Type:\
text/xml;charset=UTF-8\r\n\r\n<soapenv:Envelope\
xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/";>\ \ \
<soapenv:Body\ /></soapenv:Envelope>
tcp-check expect rstring HTTP/1.1\ 200\ OK.*>OK<
server qsrv1 10.1.50.52:10443 check slowstart 5000 ssl verify none
server qsrv2 10.1.50.150:10443 check slowstart 5000 ssl verify none
backup
