Ah I found, see below : On Thu, Oct 18, 2012 at 05:40:58PM +0800, Delta Yeh wrote: > inject31(192.168.6.66) --haproxy(192.168.6.1)---webserver(192.168.6.252) > > global > maxconn 1000000 > user root > group root > pidfile /var/run/haproxy.pid > daemon > nbproc 1 > tune.bufsize 16384 > log /dev/log local3 crit > stats socket /var/run/haproxy_stats_socket > defaults > option accept-invalid-http-response > option accept-invalid-http-request > mode http > timeout client 60s > timeout connect 60s > timeout server 120s > no option httpclose > option clitcpka > backlog 40960 > option dontlognull > no option abortonclose > no option allbackups > no option forceclose > no option nolinger > option redispatch > frontend web > #bind 0.0.0.0:8001 transparent > bind 192.168.6.1:8001 transparent > maxconn 1000000 > use_backend www if TRUE > backend www > source 0.0.0.0 usesrc clientip > server SERVER 192.168.6.252:8001 maxconn 200000 > option http-server-close > option http-pretend-keepalive > ----------------------------------------------- > > inject output if bind on 192.168.6.1 > > # ./inject -T 1000 -G 192.168.6.252:8001/ -o 900 -u 100
First, these arguments are wrong. 900*100 = 90k concurrent conns, but they are limited to 1000 by default in inject (needs -n and to set ulimit -n). Also, inject performs very poorly above a few hundreds parallel connections. > hits ^hits hits/s ^h/s bytes kB/s last errs tout htime sdht ptime > 3888 3888 3626 3626 567648 529 529 0 0 0.1 0.3 247.8 > 7222 3334 3611 3588 1054412 527 523 0 0 0.1 0.3 253.7 > 11106 3884 3702 3884 1621476 540 567 0 0 4.7 118.1 233.4 > 17827 6721 4456 6721 2602744 650 981 0 0 55.0 347.4 476.4 > 24703 6876 4940 6876 3606640 721 1003 0 0 14.0 2.3 194.3 > 28214 3511 4702 3511 4119246 686 512 0 0 14.0 2.4 203.8 > 28214 0 4030 0 4119246 588 0 0 0 0.0 0.0 0.0 Port roll-over, I'm sure you have the same values as these on the inject side : $ cat /proc/sys/net/ipv4/ip_local_port_range 32768 61000 If so this means that you probably have conntrack running somewhere and that inject it the first one to close, preventing it from reopening a connection to the same ip:port. If you add "option http-server-close" to haproxy, it might fix the issue. > inject output if haproxy bind on 0.0.0.0 is > > # ./inject -T 1000 -G 192.168.6.252:8001/ -o 900 -u 100 > > hits ^hits hits/s ^h/s bytes kB/s last errs tout htime sdht ptime > 3440 3440 4190 4190 502240 611 611 0 0 0.0 0.3 215.3 > 7778 4338 3889 3676 1135588 567 536 0 0 0.1 0.3 236.8 > 11263 3485 3754 3485 1644398 548 508 0 0 0.1 0.3 256.0 > 18101 6838 4525 6838 2642746 660 998 0 0 56.7 355.0 486.1 > 24920 6819 4984 6819 3638320 727 995 0 0 14.1 2.3 187.9 > 31675 6755 5279 6755 4624550 770 986 0 0 14.3 2.3 202.2 > 38446 6771 5492 6771 5613116 801 988 0 0 14.3 2.3 209.3 > 45194 6748 5649 6748 6598324 824 985 0 0 14.3 2.3 475.0 > 51961 6767 5773 6767 7586306 842 987 0 0 14.3 2.3 256.0 You know what's happening here ? Your tproxy rules are not matched (I already got hit by -m socket not matching sockets bound to 0.0.0.0), so the connection is not intercepted by haproxy and goes directly to the server. You can stop haproxy to confirm that it still works without it ! However I'd be interested in knowing where the connections are stuck, because if you restarted inject immediately, they should not have had the time to expire and should not have worked yet. Whn the connections don't establish anymore, it would be nice if you could run strace on the haproxy process to see what it's seeing and trying to do. I hope it's not failing to establish a connection by lack of source ports, because this would not necessarily be easy to work around. Willy
