Hi,
I have played around with HAproxy and I must say that it is a great product.
My tests were conducted on a Xen hypervisor with 3 Virtual Machines running
Ubuntu Server 8.04 64bit:
#1: HAPROXY 1.3.18 compiled from source 128 MB, 1 CPU
#2: Apache 256 MB, 1 CPU
#3 Lighttpd 512 MB, 2 CPU
This is the HAProxy configuration that I used:
dra...@haproxyubu:~$ cat /etc/haproxy/haproxy.cfg
global
log 127.0.0.1 local0
log 127.0.0.1 local1 notice
#log loghost local0 info
maxconn 131070
#debug
#quiet
user haproxy
group haproxy
defaults
log global
mode http
option httplog
option dontlognull
option redispatch
retries 3
maxconn 65535
contimeout 5000
clitimeout 50000
srvtimeout 50000
listen webfarm 192.168.1.79:80
mode http
stats enable
stats auth someuser:somepassword
balance roundrobin
cookie JSESSIONID prefix
option httpclose
option forwardfor
option httpchk HEAD /check.txt HTTP/1.0
server webA 192.168.1.83:80 cookie A check
server webB 192.168.1.84:80 cookie B check
Both Apache2-mpm-worker and lighttpd were serving a very simple index.html
<html><body>It works!</body></html>
I started some tests and I soon figured out that all 3 operating systems were
running out of resources:
dra...@apache1ubu:~$ sudo ab -n 10000 -c 100 -t 20
http://192.168.1.79/index.html
This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright 2006 The Apache Software Foundation, http://www.apache.org/
Benchmarking 192.168.1.79 (be patient)
Finished 1991 requests
Server Software: lighttpd/1.4.19
Server Hostname: 192.168.1.79
Server Port: 80
Document Path: /index.html
Document Length: 45 bytes
Concurrency Level: 100
Time taken for tests: 20.769130 seconds
Complete requests: 1991
Failed requests: 997
(Connect: 0, Length: 997, Exceptions: 0)
Write errors: 0
Non-2xx responses: 3
Total transferred: 572186 bytes
HTML transferred: 90775 bytes
Requests per second: 95.86 [#/sec] (mean)
Time per request: 1043.151 [ms] (mean)
Time per request: 10.432 [ms] (mean, across all concurrent requests)
Transfer rate: 26.87 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 112 532.6 21 8998
Processing: 1 178 594.4 83 13040
Waiting: 0 163 594.6 72 13039
Total: 1 291 876.6 113 14007
Percentage of the requests served within a certain time (ms)
50% 113
66% 135
75% 156
80% 181
90% 331
95% 532
98% 3113
99% 3409
100% 14007 (longest request)
It seems that the connections stack fills very fast and then the webservers
responds very slow or timesout.
I don't know if this relates to Xen's hypervisor own TCP/IP implementation or
the ubuntu server tcp/ip settings.
Based on this
My question is: what settings do you recommend for the Linux kernel/system in
order that
HAProxy and the webservers to be able to serve many connections at a time ?
I am looking for answers that relates to these 3 components:
* linux kernel TCP/IP configrations
* ulimit settings
* system wide settings
-------------------------------------------------------------------
What I already did:
sysctl.conf:
# These ensure that TIME_WAIT ports either get reused or closed fast.
net.ipv4.tcp_fin_timeout = 1
net.ipv4.tcp_tw_recycle = 1
# TCP memory
# net.core.rmem_max = 16777216
# net.core.rmem_default = 16777216
# net.core.netdev_max_backlog = 262144
# net.core.somaxconn = 262144
# net.ipv4.tcp_syncookies = 1
# net.ipv4.tcp_max_orphans = 262144
# net.ipv4.tcp_max_syn_backlog = 262144 // this is the connection queue ?
# net.ipv4.tcp_synack_retries = 2
# net.ipv4.tcp_syn_retries = 2
dra...@apache1ubu:~$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 1951
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 2000000
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 1951
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
I don't know how to corelate the sysctl settings with my desired simultaneous
connections/second.
I am sure that here lurks some benevolent Linux kernel experts that are willing
to help with this :)
So, to get to the point what Kernel, TCP/IP, ulimit settings do you
have/suggest for high traffic(10000+ simultaneous
connections/second) HAProxy/webservers.
Thank you
