Hi.
We've tried migrating haproxy from 1.4.22 to 1.5.2. As a result we
experienced a serious performance impact. The only thing that was changed
was hapxory version. We tried 1.5.12 afterwards, but that didn't really
help. This is running on AWS Amazon Linux 64bit. 1.5.2 was AWS stock,
1.5.12 we built ourselves.
The manifestations were that at some point, haproxy would stop responding
(we have a probe that checks the status of haproxy) to admin interfaces,
and declare servers "down" due to layer 4 check failures, and return 503 to
the requests. This sort of happens in clusters, and only under high load
(over 1,500 requests/sec). All, or most of the servers are at this point
declared "down", even the ones that only get very little traffic. The check
failures logged like this:
May 18 15:06:16 ip-10-16-20-170 haproxy[2218]: Server node15/node15 is
DOWN, reason: Layer4 timeout, check duration: 2000ms. 0 active and 0 backup
servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
I wonder if there are some configuration values that need to be tinkered
with for 1.5 that weren't needed for 1.4. Any suggestions will be
appreciated as well. The traffic is http only, and mostly short messages
(~1k responses), though some (1/5) are ~200k. About 80% of the traffic
would go through the "nodeXX" backends, the rest is through "wl" backend.
haproxy.cfg:
global
log /dev/log local5 info
maxconn 16000
nbproc 2
stats socket /var/run/haproxy.sock user nrpe group nrpe mode user
listen stats-odd :1936
bind-process odd
mode http
acl AUTH src 10.16.0.165
stats enable
stats uri /
stats admin if AUTH
timeout client 300s
listen stats-even :1937
bind-process even
mode http
acl AUTH src 10.16.0.165
stats enable
stats uri /
stats admin if AUTH
timeout client 300s
defaults
log global
option allbackups
option httpclose
option redispatch
mode http
retries 3
timeout client 300s
timeout connect 3s
timeout server 300s
frontend ssldemark
bind *:8888
maxconn 6000
acl for_pe path_beg /pe
acl for_mis path_beg /mispot
acl for_n1 url_sub _gswnid=001
acl for_n2 url_sub _gswnid=002
acl for_n3 url_sub _gswnid=003
acl for_n4 url_sub _gswnid=004
acl for_n5 url_sub _gswnid=005
acl for_n6 url_sub _gswnid=006
acl for_n7 url_sub _gswnid=007
acl for_n8 url_sub _gswnid=008
acl for_n9 url_sub _gswnid=009
acl for_n11 url_sub _gswnid=011
acl for_n12 url_sub _gswnid=012
acl for_n13 url_sub _gswnid=013
acl for_n14 url_sub _gswnid=014
acl for_n15 url_sub _gswnid=015
acl for_n16 url_sub _gswnid=016
acl for_n17 url_sub _gswnid=017
acl for_n18 url_sub _gswnid=018
acl for_n19 url_sub _gswnid=019
acl for_n21 url_sub _gswnid=021
acl for_n22 url_sub _gswnid=022
acl for_n23 url_sub _gswnid=023
acl for_n24 url_sub _gswnid=024
acl for_n25 url_sub _gswnid=025
acl for_n26 url_sub _gswnid=026
acl for_n27 url_sub _gswnid=027
acl for_n28 url_sub _gswnid=028
acl for_n29 url_sub _gswnid=029
reqadd Proxy-SSL:\ true
reqadd X-gsm-secure:\ true
default_backend static
use_backend node1 if for_n1
use_backend node2 if for_n2
use_backend node3 if for_n3
use_backend node4 if for_n4
use_backend node5 if for_n5
use_backend node6 if for_n6
use_backend node7 if for_n7
use_backend node8 if for_n8
use_backend node9 if for_n9
use_backend node11 if for_n11
use_backend node12 if for_n12
use_backend node13 if for_n13
use_backend node14 if for_n14
use_backend node15 if for_n15
use_backend node16 if for_n16
use_backend node17 if for_n17
use_backend node18 if for_n18
use_backend node19 if for_n19
use_backend node21 if for_n21
use_backend node22 if for_n22
use_backend node23 if for_n23
use_backend node24 if for_n24
use_backend node25 if for_n25
use_backend node26 if for_n26
use_backend node27 if for_n27
use_backend node28 if for_n28
use_backend node29 if for_n29
use_backend wl if for_pe
use_backend mis if for_mis
frontend mispot
bind *:80
maxconn 16000
capture request header Referer len 64
capture request header X-Forwarded-For len 64
capture response header Status len 64
capture response header Server len 20
acl for_pe path_beg /pe
acl for_mis path_beg /mispot
acl for_n1 url_sub _gswnid=001
acl for_n2 url_sub _gswnid=002
acl for_n3 url_sub _gswnid=003
acl for_n4 url_sub _gswnid=004
acl for_n5 url_sub _gswnid=005
acl for_n6 url_sub _gswnid=006
acl for_n7 url_sub _gswnid=007
acl for_n8 url_sub _gswnid=008
acl for_n9 url_sub _gswnid=009
acl for_n11 url_sub _gswnid=011
acl for_n12 url_sub _gswnid=012
acl for_n13 url_sub _gswnid=013
acl for_n14 url_sub _gswnid=014
acl for_n15 url_sub _gswnid=015
acl for_n16 url_sub _gswnid=016
acl for_n17 url_sub _gswnid=017
acl for_n18 url_sub _gswnid=018
acl for_n19 url_sub _gswnid=019
acl for_n21 url_sub _gswnid=021
acl for_n22 url_sub _gswnid=022
acl for_n23 url_sub _gswnid=023
acl for_n24 url_sub _gswnid=024
acl for_n25 url_sub _gswnid=025
acl for_n26 url_sub _gswnid=026
acl for_n27 url_sub _gswnid=027
acl for_n28 url_sub _gswnid=028
acl for_n29 url_sub _gswnid=029
reqidel ^Proxy-Client-IP:.*
reqidel ^X-gsm-secure:.*
option forwardfor header Proxy-Client-IP
option httplog
log global
default_backend static
use_backend node1 if for_n1
use_backend node2 if for_n2
use_backend node3 if for_n3
use_backend node4 if for_n4
use_backend node5 if for_n5
use_backend node6 if for_n6
use_backend node7 if for_n7
use_backend node8 if for_n8
use_backend node9 if for_n9
use_backend node11 if for_n11
use_backend node12 if for_n12
use_backend node13 if for_n13
use_backend node14 if for_n14
use_backend node15 if for_n15
use_backend node16 if for_n16
use_backend node17 if for_n17
use_backend node18 if for_n18
use_backend node19 if for_n19
use_backend node21 if for_n21
use_backend node22 if for_n22
use_backend node23 if for_n23
use_backend node24 if for_n24
use_backend node25 if for_n25
use_backend node26 if for_n26
use_backend node27 if for_n27
use_backend node28 if for_n28
use_backend node29 if for_n29
use_backend wl if for_pe
use_backend mis if for_mis
backend wl
option httpchk /pe/oks.jsp
server web11 10.16.20.11:8080 weight 14 check error-limit 10 on-error
mark-down
server web12 10.16.20.12:8080 weight 14 check error-limit 10 on-error
mark-down
server web13 10.16.20.13:8080 weight 14 check error-limit 10 on-error
mark-down
server web17 10.16.20.17:8080 weight 14 check error-limit 10 on-error
mark-down
server web18 10.16.20.18:8080 weight 14 check error-limit 10 on-error
mark-down
server web21 10.16.30.11:8080 backup weight 14 check error-limit 5
on-error mark-down
server web22 10.16.30.12:8080 backup weight 14 check error-limit 5
on-error mark-down
server web23 10.16.30.13:8080 backup weight 14 check error-limit 5
on-error mark-down
server web24 10.16.30.14:8080 backup weight 14 check error-limit 5
on-error mark-down
server web25 10.16.30.15:8080 backup weight 14 check error-limit 5
on-error mark-down
server web26 10.16.30.16:8080 backup weight 14 check error-limit 5
on-error mark-down
backend static
option httpchk GET /index.html
server web15 10.16.20.15:88 check error-limit 5 on-error mark-down
server web11 10.16.20.11:88 backup check error-limit 5 on-error
mark-down
server web12 10.16.20.12:88 backup check error-limit 5 on-error
mark-down
server web13 10.16.20.13:88 backup check error-limit 5 on-error
mark-down
server web21 10.16.30.11:88 backup check error-limit 2 on-error
mark-down
server web22 10.16.30.12:88 backup check error-limit 2 on-error
mark-down
server web23 10.16.30.13:88 backup check error-limit 2 on-error
mark-down
backend mis
no option allbackups
option httpchk /mispot/login.htm
server jpweb15 10.16.20.15:8080 check error-limit 10 on-error mark-down
server jpweb04 10.16.43.14:8080 backup check error-limit 10 on-error
mark-down
backend node1
option httpchk /pe/oks.jsp
server node1 10.16.43.11:8080 check error-limit 20 on-error mark-down
backend node2
option httpchk /pe/oks.jsp
server node2 10.16.43.12:8080 check error-limit 20 on-error mark-down
backend node3
option httpchk /pe/oks.jsp
server node3 10.16.43.13:8080 check error-limit 20 on-error mark-down
backend node4
option httpchk /pe/oks.jsp
server node4 10.16.43.14:8080 check error-limit 20 on-error mark-down
backend node5
option httpchk /pe/oks.jsp
server node5 10.16.43.15:8080 check error-limit 20 on-error mark-down
backend node6
option httpchk /pe/oks.jsp
server node6 10.16.43.16:8080 check error-limit 20 on-error mark-down
backend node7
option httpchk /pe/oks.jsp
server node7 10.16.43.17:8080 check error-limit 20 on-error mark-down
backend node8
option httpchk /pe/oks.jsp
server node8 10.16.43.18:8080 check error-limit 20 on-error mark-down
backend node9
option httpchk /pe/oks.jsp
server node9 10.16.43.19:8080 check error-limit 20 on-error mark-down
backend node11
option httpchk /pe/oks.jsp
server node11 10.16.20.11:8080 check error-limit 20 on-error mark-down
backend node12
option httpchk /pe/oks.jsp
server node12 10.16.20.12:8080 check error-limit 20 on-error mark-down
backend node13
option httpchk /pe/oks.jsp
server node13 10.16.20.13:8080 check error-limit 20 on-error mark-down
backend node14
option httpchk /pe/oks.jsp
server node14 10.16.20.14:8080 check error-limit 20 on-error mark-down
backend node15
option httpchk /pe/oks.jsp
server node15 10.16.20.15:8080 check error-limit 20 on-error mark-down
backend node16
option httpchk /pe/oks.jsp
server node16 10.16.20.16:8080 check error-limit 20 on-error mark-down
backend node17
option httpchk /pe/oks.jsp
server node13 10.16.20.17:8080 check error-limit 20 on-error mark-down
backend node18
option httpchk /pe/oks.jsp
server node18 10.16.20.18:8080 check error-limit 20 on-error mark-down
backend node19
option httpchk /pe/oks.jsp
server node19 10.16.20.19:8080 check error-limit 20 on-error mark-down
backend node21
option httpchk /pe/oks.jsp
server node21 10.16.30.11:8080 check error-limit 20 on-error mark-down
backend node22
option httpchk /pe/oks.jsp
server node22 10.16.30.12:8080 check error-limit 20 on-error mark-down
backend node23
option httpchk /pe/oks.jsp
server node23 10.16.30.13:8080 check error-limit 20 on-error mark-down
backend node24
option httpchk /pe/oks.jsp
server node24 10.16.30.14:8080 check error-limit 20 on-error mark-down
backend node25
option httpchk /pe/oks.jsp
server node25 10.16.30.15:8080 check error-limit 20 on-error mark-down
backend node26
option httpchk /pe/oks.jsp
server node26 10.16.30.16:8080 check error-limit 20 on-error mark-down
backend node27
option httpchk /pe/oks.jsp
server node27 10.16.30.17:8080 check error-limit 20 on-error mark-down
backend node28
option httpchk /pe/oks.jsp
server node28 10.16.30.18:8080 check error-limit 20 on-error mark-down
backend node29
option httpchk /pe/oks.jsp
server node29 10.16.30.19:8080 check error-limit 20 on-error mark-down
