Hello. I'm having an issue with connections to haproxy hanging upon the
connect stage. My setup is 2 nginx/php-fpm backends with haproxy load
balancing them (3 servers total). When connecting directly to the app
servers everything is great, but through haproxy my browser hangs at
"connecting" for 1-30 seconds sometimes.. or just never connects
requiring a hard refresh.
Sysctl and haproxy settings on the load balancer are:
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_tw_buckets = 360000
net.ipv4.tcp_fin_timeout = 20
net.ipv4.ip_local_port_range = 2000 64000
net.ipv4.tcp_syncookies = 0
net.ipv4.tcp_max_orphans = 262144
net.ipv4.tcp_max_syn_backlog = 262144
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_syn_retries = 2
kernel.shmmax = 4294967296
fs.file-max = 1048576
net.core.netdev_max_backlog = 100000
net.core.somaxconn = 100000
net.core.rmem_max = 8388608
net.ipv4.tcp_rmem = 4096 1048576 8388608
net.core.wmem_max = 8388608
net.ipv4.tcp_wmem = 4096 1048576 8388608
net.ipv4.tcp_mem = 8388608 8388608 8388608
net.core.optmem_max = 40960
------------
global
log 127.0.0.1 local0
user haproxy
group haproxy
daemon
maxconn 100000
# stats socket /opt/haproxy/etc/sock.haproxy uid 0 gid 0 mode 700
level admin
defaults
log global
option dontlognull
balance leastconn
retries 3
option redispatch
timeout connect 20000ms
timeout server 30000ms
timeout client 20000ms
listen stats hidden:47880
mode http
stats enable
stats uri /stats
stats realm HAProxy\ Statistics
stats auth hidden
listen www A.B.C.D:80
mode http
option httpclose
option nolinger
timeout client 20000ms
option httpchk HEAD / HTTP/1.0
option forwardfor
cookie SERVERID insert
balance leastconn
maxconn 50000
server app1 10.240.0.2:80 cookie a1 maxconn 25000 check
server app2 10.240.0.3:80 cookie a2 maxconn 25000 check
#server app3 10.240.0.4:80 cookie a3 maxconn 15000 check backup
tcpdump output when this happens: http://pastebin.com/cXE8kWew (note the
8 second "hang")
20:46:11.923871 IP MY-CLIENT.55494 > MY-SERVER.www: Flags [P.], seq
706215489:706216213, ack 3030317681, win 4280, length 724
20:46:12.118792 IP MY-CLIENT.55494 > MY-SERVER.www: Flags [.], ack 2635,
win 4280, length 0
20:46:12.303468 IP MY-CLIENT.55494 > MY-SERVER.www: Flags [.], ack 5269,
win 4280, length 0
20:46:12.303483 IP MY-CLIENT.55494 > MY-SERVER.www: Flags [.], ack 7659,
win 4280, length 0
20:46:12.303492 IP MY-CLIENT.55494 > MY-SERVER.www: Flags [F.], seq 724,
ack 7659, win 4280, length 0
20:46:20.111011 IP MY-CLIENT.55493 > MY-SERVER.www: Flags [P.], seq
3064322029:3064322755, ack 1531281668, win 4280, length 726
20:46:20.140771 IP MY-CLIENT.55495 > MY-SERVER.www: Flags [P.], seq
2914190645:2914191371, ack 2823463340, win 4280, length 726
20:46:20.305019 IP MY-CLIENT.55493 > MY-SERVER.www: Flags [.], ack 320,
win 4200, length 0
20:46:20.305032 IP MY-CLIENT.55493 > MY-SERVER.www: Flags [F.], seq 726,
ack 320, win 4200, length 0
20:46:20.324906 IP MY-CLIENT.55495 > MY-SERVER.www: Flags [.], ack 709,
win 4103, length 0
20:46:20.334540 IP MY-CLIENT.55495 > MY-SERVER.www: Flags [F.], seq 726,
ack 709, win 4103, length 0
20:46:22.087034 IP MY-CLIENT.55496 > MY-SERVER.www: Flags [F.], seq
2675337998, ack 678907057, win 4280, length 0
20:46:22.277682 IP MY-CLIENT.55496 > MY-SERVER.www: Flags [R.], seq 1,
ack 188, win 0, length 0
Also of interest is the fact that there is about 3k SYN_RECV, 20k
TIME_WAIT and 200 ESTABLISHED connections in netstat. Under the "errors
- resp" column of the haproxy stats page for the backend, if I hover
over the numbers for backend it says "connection resets during transfer:
300k+ client, 0 server".
Any help is greatly appreciated, I am stumped.
Ivan
