-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On Feb 19, 2009, at 1:48 PM, Willy Tarreau wrote:
Thanks for the long answer.
You're correct, this is typically what happens in such a case.
The increase of CPU usage is generally caused by two factors :
- connection retries, which act as a traffic multiplier
- immediate closure which causes the load injection tool
to immediately send a new connection instead of having to
wait a few ms for the application.
Yep, that was my guess too.
# the following stops low-level messages on console
kernel.printk = 4 4 1 7
fs.inotify.max_user_watches = 524288
# some spoof protection
net.ipv4.conf.default.rp_filter=1
net.ipv4.conf.all.rp_filter=1
# General gigabit tuning:
net.core.rmem_max = 33554432
net.core.wmem_max = 33554432
net.ipv4.tcp_rmem = 4096 16384 33554432
net.ipv4.tcp_wmem = 4096 16384 33554432
net.ipv4.tcp_mem = 786432 1048576 26777216
This one above is wrong (though it's not our problem here). It says
that you'll use a minimum of 3 GB, a default of 4 GB and a max of
100 GB for the TCP stack. The unit here is pages, not bytes.
net.ipv4.tcp_max_tw_buckets = 360000
net.core.netdev_max_backlog = 2500
vm.min_free_kbytes = 65536
vm.swappiness = 0
net.ipv4.ip_local_port_range = 25000 65535
Could you check net.ipv4.tcp_tw_reuse, and set it to 1 if it's zero ?
It probably was set to 0... This fix and the change of tcp_mem to the
standard values (which are created dynamically depending on the
available memory) basically fixed the problem.
Now HaProxy is giving me much much better latency numbers than
nginx, and that's what I was looking for. Now 2 machines combined
run at a maximum of about 4700 req/sec more or less and average
latency for the full request is 0.311 while the fastest is 77ms.
And this is my modified sysctl.conf.
# the following stops low-level messages on console
kernel.printk = 4 4 1 7
fs.inotify.max_user_watches = 524288
# some spoof protection
net.ipv4.conf.default.rp_filter=1
net.ipv4.conf.all.rp_filter=1
# General gigabit tuning:
net.core.rmem_max = 8738000
net.core.wmem_max = 6553600
net.ipv4.tcp_rmem = 8192 873800 8738000
net.ipv4.tcp_wmem = 4096 655360 6553600
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_tw_buckets = 360000
net.core.netdev_max_backlog = 2500
vm.min_free_kbytes = 65536
vm.swappiness = 0
net.ipv4.ip_local_port_range = 30000 65535
thanks a lot again.
- --
Valentino Volonghi aka Dialtone
Now running MacOS X 10.5
Home Page: http://www.twisted.it
http://www.adroll.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)
iEYEARECAAYFAkmd8noACgkQ9Llz28widGVZHgCfXzpQ5h5m1kcFk8Xr2vZNSS+8
vMgAnRE0XvTNgS97KezpFCIE40Ml+9Ql
=CgJG
-----END PGP SIGNATURE-----
