We have 5 Apache servers behind haproxy and we're trying to enable use the
"httpchk" option along with some performance monitoring. For some reason,
haproxy keeps thinking that 3/5 apache servers are "down" even though it's
obvious that haproxy is both asking the questions and the servers are
answering.
Is there a way to log httpchk failures? How can I ask haproxy why it seems to
think that several apache servers are down?
Our config:
CentOS 6.x recently updated, 64 bit.
Performing an agent-check manually seems to give good results. The below
result is immediate:
[root@xr1 ~]# telnet 10.1.1.12 9333
Trying 10.1.1.12...
Connected to 10.1.1.12.
Escape character is '^]'.
up 78%
Connection closed by foreign host.
I can see that xinetd on the logic server got the response:
Apr 13 18:45:02 curie xinetd[21890]: EXIT: calcload333 status=0 pid=25693
duration=0(sec)
Apr 13 18:45:06 curie xinetd[21890]: START: calcload333 pid=26590
from=::ffff:10.1.1.1
I can see that apache is serving happy replies to the load balancer:
[root@curie ~]# tail -f /var/log/httpd/access_log | grep -i "10.1.1.1 "
10.1.1.1 - - [13/Apr/2015:18:47:15 +0000] "OPTIONS / HTTP/1.0" 302 - "-" "-"
10.1.1.1 - - [13/Apr/2015:18:47:17 +0000] "OPTIONS / HTTP/1.0" 302 - "-" "-"
10.1.1.1 - - [13/Apr/2015:18:47:19 +0000] "OPTIONS / HTTP/1.0" 302 - "-" "-"
^C
Why is haproxy pulling this server out of the queue and setting status to
DRAIN? Here's the stats output for the server:
[pxname] => logic333
[svname] => server12
[qcur] => 0
[qmax] => 0
[scur] => 0
[smax] => 0
[slim] => 256
[stot] => 0
[bin] => 0
[bout] => 0
[dreq] =>
[dresp] => 0
[ereq] =>
[econ] => 0
[eresp] => 0
[wretr] => 0
[wredis] => 0
[status] => DRAIN
[weight] => 0
[act] => 1
[bck] => 0
[chkfail] => 0
[chkdown] => 0
[lastchg] => 997
[downtime] => 0
[qlimit] =>
[pid] => 1
[iid] => 5
[sid] => 3
[throttle] =>
[lbtot] => 0
[tracked] =>
[type] => 2
[rate] => 0
[rate_lim] =>
[rate_max] => 0
[check_status] => L7OK
[check_code] => 302
[check_duration] => 16
[hrsp_1xx] => 0
[hrsp_2xx] => 0
[hrsp_3xx] => 0
[hrsp_4xx] => 0
[hrsp_5xx] => 0
[hrsp_other] => 0
[hanafail] => 0
[req_rate] =>
[req_rate_max] =>
[req_tot] =>
[cli_abrt] => 0
[srv_abrt] => 0
[comp_in] =>
[comp_out] =>
[comp_byp] =>
[comp_rsp] =>
[lastsess] => -1
[last_chk] => Found
[last_agt] => via agent : up
[qtime] => 0
[ctime] => 0
[rtime] => 0
[ttime] => 0
[root@xr1 ~]# haproxy -v
HA-Proxy version 1.5.6 2014/10/18
Copyright 2000-2014 Willy Tarreau <[email protected]>
relevant parts of /etc/haproxy/haproxy.conf
-- SNIP --
frontend FrontProd333
mode http
option httplog
option dontlognull
option forwardfor
bind [MYIP]:443 ssl crt star.mydomain.com.pem no-sslv3
default_backend logic333
stats uri /haproxy?stats
stats realm StrictlyPrivate
stats auth [SNIP]
errorfile 503 /etc/haproxy/errorfile.http
backend logic333
mode http
option httpchk OPTIONS /
server server10 10.1.1.10:20333 maxconn 256 check agent-check agent-port
9333 agent-inter 4000
server server11 10.1.1.11:20333 maxconn 256 check agent-check agent-port
9333 agent-inter 4000
server server12 10.1.1.12:20333 maxconn 256 check agent-check agent-port
9333 agent-inter 4000
server server13 10.1.1.13:20333 maxconn 256 check agent-check agent-port
9333 agent-inter 4000
server server14 10.1.1.14:20333 maxconn 256 check agent-check agent-port
9333 agent-inter 4000
-- /SNIP --
Is there any other information I could provide to help resolve?
Thanks,
Ben Smith
