Hi, You should setup net.ipv4.ip_local_port_range as well to increase allowed opened ports to servers. Your maxconns seems too high, but I doubt this is the source of your issue.
cheers On Thu, Feb 2, 2012 at 10:16 PM, Ivan Ator <ivanat...@gmail.com> wrote: > Hello. I'm having an issue with connections to haproxy hanging upon the > connect stage. My setup is 2 nginx/php-fpm backends with haproxy load > balancing them (3 servers total). When connecting directly to the app > servers everything is great, but through haproxy my browser hangs at > "connecting" for 1-30 seconds sometimes.. or just never connects requiring a > hard refresh. > > Sysctl and haproxy settings on the load balancer are: > > net.ipv4.tcp_tw_recycle = 1 > net.ipv4.tcp_tw_reuse = 1 > net.ipv4.tcp_max_tw_buckets = 360000 > net.ipv4.tcp_fin_timeout = 20 > net.ipv4.ip_local_port_range = 2000 64000 > net.ipv4.tcp_syncookies = 0 > net.ipv4.tcp_max_orphans = 262144 > net.ipv4.tcp_max_syn_backlog = 262144 > net.ipv4.tcp_synack_retries = 2 > net.ipv4.tcp_syn_retries = 2 > > kernel.shmmax = 4294967296 > fs.file-max = 1048576 > > net.core.netdev_max_backlog = 100000 > net.core.somaxconn = 100000 > > net.core.rmem_max = 8388608 > net.ipv4.tcp_rmem = 4096 1048576 8388608 > net.core.wmem_max = 8388608 > net.ipv4.tcp_wmem = 4096 1048576 8388608 > net.ipv4.tcp_mem = 8388608 8388608 8388608 > net.core.optmem_max = 40960 > > ------------ > > global > log 127.0.0.1 local0 > user haproxy > group haproxy > daemon > maxconn 100000 > # stats socket /opt/haproxy/etc/sock.haproxy uid 0 gid 0 mode 700 > level admin > > defaults > log global > option dontlognull > balance leastconn > retries 3 > option redispatch > timeout connect 20000ms > timeout server 30000ms > timeout client 20000ms > > listen stats hidden:47880 > mode http > stats enable > stats uri /stats > stats realm HAProxy\ Statistics > stats auth hidden > > listen www A.B.C.D:80 > mode http > option httpclose > option nolinger > timeout client 20000ms > > option httpchk HEAD / HTTP/1.0 > option forwardfor > > cookie SERVERID insert > > balance leastconn > maxconn 50000 > > server app1 10.240.0.2:80 cookie a1 maxconn 25000 check > server app2 10.240.0.3:80 cookie a2 maxconn 25000 check > #server app3 10.240.0.4:80 cookie a3 maxconn 15000 check backup > > tcpdump output when this happens: http://pastebin.com/cXE8kWew (note the 8 > second "hang") > > 20:46:11.923871 IP MY-CLIENT.55494 > MY-SERVER.www: Flags [P.], seq > 706215489:706216213, ack 3030317681, win 4280, length 724 > 20:46:12.118792 IP MY-CLIENT.55494 > MY-SERVER.www: Flags [.], ack 2635, win > 4280, length 0 > 20:46:12.303468 IP MY-CLIENT.55494 > MY-SERVER.www: Flags [.], ack 5269, win > 4280, length 0 > 20:46:12.303483 IP MY-CLIENT.55494 > MY-SERVER.www: Flags [.], ack 7659, win > 4280, length 0 > 20:46:12.303492 IP MY-CLIENT.55494 > MY-SERVER.www: Flags [F.], seq 724, ack > 7659, win 4280, length 0 > 20:46:20.111011 IP MY-CLIENT.55493 > MY-SERVER.www: Flags [P.], seq > 3064322029:3064322755, ack 1531281668, win 4280, length 726 > 20:46:20.140771 IP MY-CLIENT.55495 > MY-SERVER.www: Flags [P.], seq > 2914190645:2914191371, ack 2823463340, win 4280, length 726 > 20:46:20.305019 IP MY-CLIENT.55493 > MY-SERVER.www: Flags [.], ack 320, win > 4200, length 0 > 20:46:20.305032 IP MY-CLIENT.55493 > MY-SERVER.www: Flags [F.], seq 726, ack > 320, win 4200, length 0 > 20:46:20.324906 IP MY-CLIENT.55495 > MY-SERVER.www: Flags [.], ack 709, win > 4103, length 0 > 20:46:20.334540 IP MY-CLIENT.55495 > MY-SERVER.www: Flags [F.], seq 726, ack > 709, win 4103, length 0 > 20:46:22.087034 IP MY-CLIENT.55496 > MY-SERVER.www: Flags [F.], seq > 2675337998, ack 678907057, win 4280, length 0 > 20:46:22.277682 IP MY-CLIENT.55496 > MY-SERVER.www: Flags [R.], seq 1, ack > 188, win 0, length 0 > > Also of interest is the fact that there is about 3k SYN_RECV, 20k TIME_WAIT > and 200 ESTABLISHED connections in netstat. Under the "errors - resp" column > of the haproxy stats page for the backend, if I hover over the numbers for > backend it says "connection resets during transfer: 300k+ client, 0 server". > > Any help is greatly appreciated, I am stumped. > > Ivan >
