Hi Willy, Pavlos,
Thank you once again for your advice.
> Requests per second: 19071.55 [#/sec] (mean)
> > Transfer rate: 9461.28 [Kbytes/sec] received
>
> These numbers are extremely low and very likely indicate an http
> close mode combined with an untuned nf_conntrack.
>
Yes, it was due to http close mode, and wrong irq pinning (nf_conntrack_max
was
set to 640K).
> > mpstat (first 4 processors only, rest are almost zero):
> > Average: CPU %usr %nice %sys %iowait %irq %soft %steal
> > %guest %gnice %idle
> > Average: 0 0.25 0.00 0.75 0.00 0.00 98.01 0.00
> > 0.00 0.00 1.00
>
> This CPU is spending its time in softirq, probably due to conntrack
> spending a lot of time looking for the session for each packet in too
> small a hash table.
>
I had not done irq pinning. Today I am getting much better results with irq
pinning
and keepalive.
Note, this is about 2 Gbps. How is your network configured ? You should
> normally see either 1 Gbps with a gig NIC or 10 Gbps with a 10G NIC,
> because retrieving a static file is very cheap. Would you happen to be
> using bonding in round-robin mode maybe ? If that's the case, it's a
> performance disaster due to out-of-order packets and could explain some
> of the high %softirq.
>
My setup is as follows (no bonding, etc, and Sys stands for baremetal
system, each with
48 core, 128GB mem, ixgbe single ethernet port card).
Sys1-with-ab <-eth0-> Sys1-with-Haproxy, which uses two nginx backend
systems
over the same eth0 card (that is the current restriction, no extra ethernet
interface for
separate frontend/backend, etc). Today I am getting a high of 7.7 Gbps with
your
suggestions. Is it possible to get higher than that (direct to server gets
8.6 Gbps)?
Please retry without http-server-close to maintain keep-alive to the
> servers, that will avoid the session setup/teardown. If that becomes
> better, there's definitely something to fix in the conntrack or maybe
> in iptables rules if you have some. But in any case don't put such a
There are a few iptables rules, which seem clean. The results now are:
ab -k -n 1000000 -c 500 http://<haproxy>:80/64 (I am getting some errors
though,
which is not present when running against the backend directly):
Document Length: 64 bytes
Concurrency Level: 500
Time taken for tests: 6.181 seconds
Complete requests: 1000000
Failed requests: 18991
(Connect: 0, Receive: 0, Length: 9675, Exceptions: 9316)
Write errors: 0
Keep-Alive requests: 990330
Total transferred: 296554848 bytes
HTML transferred: 63381120 bytes
Requests per second: 161783.42 [#/sec] (mean)
Time per request: 3.091 [ms] (mean)
Time per request: 0.006 [ms] (mean, across all concurrent requests)
Transfer rate: 46853.18 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.5 0 8
Processing: 0 3 6.2 3 1005
Waiting: 0 3 6.2 3 1005
Total: 0 3 6.3 3 1010
Percentage of the requests served within a certain time (ms)
50% 3
66% 3
75% 3
80% 3
90% 4
95% 5
98% 6
99% 8
100% 1010 (longest request)
pidstat (some system numbers are very high, 50%, maybe due to small packet
sizes?):
Average: UID PID %usr %system %guest %CPU CPU Command
Average: 110 52601 6.00 9.33 0.00 15.33 - haproxy
Average: 110 52602 6.33 11.83 0.00 18.17 - haproxy
Average: 110 52603 11.33 17.83 0.00 29.17 - haproxy
Average: 110 52604 17.50 30.33 0.00 47.83 - haproxy
Average: 110 52605 20.50 38.50 0.00 59.00 - haproxy
Average: 110 52606 24.50 51.33 0.00 75.83 - haproxy
Average: 110 52607 22.50 51.33 0.00 73.83 - haproxy
Average: 110 52608 23.67 47.17 0.00 70.83 - haproxy
mpstat (of interesting cpus only):
Average: CPU %usr %nice %sys %iowait %irq %soft %steal
%guest %gnice %idle
Average: all 2.58 0.00 4.36 0.00 0.00 0.89 0.00
0.00 0.00 92.17
Average: 0 6.84 0.00 11.46 0.00 0.00 2.03 0.00
0.00 0.00 79.67
Average: 1 11.15 0.00 19.85 0.00 0.00 5.29 0.00
0.00 0.00 63.71
Average: 2 8.32 0.00 12.20 0.00 0.00 2.22 0.00
0.00 0.00 77.26
Average: 3 7.92 0.00 11.97 0.00 0.00 2.39 0.00
0.00 0.00 77.72
Average: 4 8.81 0.00 13.76 0.00 0.00 2.39 0.00
0.00 0.00 75.05
Average: 5 6.96 0.00 12.27 0.00 0.00 2.38 0.00
0.00 0.00 78.39
Average: 6 9.21 0.00 12.52 0.00 0.00 3.31 0.00
0.00 0.00 74.95
Average: 7 7.56 0.00 13.65 0.00 0.00 2.95 0.00
0.00 0.00 75.83
Average: 8 6.33 0.00 10.80 0.00 0.00 1.86 0.00
0.00 0.00 81.01
Average: 9 7.27 0.00 11.82 0.00 0.00 2.36 0.00
0.00 0.00 78.55
Average: 10 6.80 0.00 12.68 0.00 0.00 2.94 0.00
0.00 0.00 77.57
Average: 11 6.97 0.00 10.64 0.00 0.00 2.20 0.00
0.00 0.00 80.18
Average: 12 6.33 0.00 12.12 0.18 0.00 2.71 0.00
0.00 0.00 78.66
Average: 13 5.83 0.00 11.84 0.00 0.00 2.37 0.00
0.00 0.00 79.96
Average: 14 5.73 0.00 12.01 0.00 0.00 1.97 0.00
0.00 0.00 80.29
Average: 15 6.01 0.00 11.11 0.00 0.00 2.00 0.00
0.00 0.00 80.87
Average: 16 3.24 0.00 4.26 0.00 0.00 0.85 0.00
0.00 0.00 91.65
Average: 17 4.17 0.00 7.64 0.00 0.00 1.39 0.00
0.00 0.00 86.81
Average: 18 1.18 0.00 1.85 0.00 0.00 0.17 0.00
0.00 0.00 96.81
Average: 19 1.35 0.00 2.03 0.00 0.00 0.17 0.00
0.00 0.00 96.45
Average: 20 0.68 0.00 0.84 0.00 0.00 0.17 0.00
0.00 0.00 98.31
Average: 21 0.34 0.00 0.50 0.00 0.00 0.17 0.00
0.00 0.00 98.99
Average: 22 1.01 0.00 1.34 0.00 0.00 0.34 0.00
0.00 0.00 97.32
Average: 25 1.00 0.00 1.67 0.00 0.00 0.67 0.00
0.00 0.00 96.66
Similarly for large IO:
ab -k -n 100000 -c 500 http://<haproxy>:80/512K
Document Length: 524288 bytes
Concurrency Level: 500
Time taken for tests: 52.190 seconds
Complete requests: 100000
Failed requests: 0
Write errors: 0
Keep-Alive requests: 100000
Total transferred: 52452889936 bytes
HTML transferred: 52428800000 bytes
Requests per second: 1916.08 [#/sec] (mean)
Time per request: 260.950 [ms] (mean)
Time per request: 0.522 [ms] (mean, across all concurrent requests)
Transfer rate: 981483.06 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.4 0 9
Processing: 1 257 244.9 217 6879
Waiting: 0 23 71.3 5 1476
Total: 1 257 244.9 217 6879
Percentage of the requests served within a certain time (ms)
50% 217
66% 273
75% 355
80% 441
90% 560
95% 739
98% 932
99% 1106
100% 6879 (longest request)
pidstat:
Average: UID PID %usr %system %guest %CPU CPU Command
Average: 110 52601 1.71 8.82 0.00 10.53 - haproxy
Average: 110 52602 2.08 11.67 0.00 13.75 - haproxy
Average: 110 52603 1.18 7.47 0.00 8.65 - haproxy
Average: 110 52604 1.73 9.51 0.00 11.24 - haproxy
Average: 110 52605 1.71 9.53 0.00 11.24 - haproxy
Average: 110 52606 1.86 10.45 0.00 12.31 - haproxy
Average: 110 52607 2.63 12.82 0.00 15.45 - haproxy
Average: 110 52608 1.31 7.35 0.00 8.67 - haproxy
mpstat:
Average: CPU %usr %nice %sys %iowait %irq %soft %steal
%guest %gnice %idle
Average: all 0.23 0.00 1.25 0.01 0.00 1.84 0.00
0.00 0.00 96.68
Average: 0 0.41 0.00 2.57 0.29 0.00 4.17 0.00
0.00 0.00 92.56
Average: 1 0.58 0.00 2.82 0.00 0.00 4.69 0.00
0.00 0.00 91.91
Average: 2 0.49 0.00 2.62 0.00 0.00 3.77 0.00
0.00 0.00 93.11
Average: 3 0.62 0.00 2.89 0.00 0.00 5.13 0.00
0.00 0.00 91.36
Average: 4 0.60 0.00 2.71 0.00 0.00 3.76 0.00
0.00 0.00 92.94
Average: 5 0.45 0.00 2.69 0.00 0.00 3.45 0.00
0.00 0.00 93.40
Average: 6 0.45 0.00 2.71 0.00 0.00 3.87 0.00
0.00 0.00 92.97
Average: 7 0.45 0.00 2.18 0.00 0.00 3.56 0.00
0.00 0.00 93.81
Average: 8 0.49 0.00 2.29 0.00 0.00 3.85 0.00
0.00 0.00 93.36
Average: 9 0.49 0.00 2.63 0.00 0.00 4.01 0.00
0.00 0.00 92.86
Average: 10 0.52 0.00 2.71 0.00 0.00 4.28 0.00
0.00 0.00 92.50
Average: 11 0.61 0.00 3.15 0.00 0.00 4.05 0.00
0.00 0.00 92.19
Average: 12 0.35 0.00 2.17 0.00 0.00 3.28 0.00
0.00 0.00 94.20
Average: 13 0.47 0.00 2.70 0.00 0.00 3.95 0.00
0.00 0.00 92.89
Average: 14 0.56 0.00 2.41 0.00 0.00 3.79 0.00
0.00 0.00 93.24
Average: 15 0.45 0.00 2.56 0.00 0.00 3.84 0.00
0.00 0.00 93.15
Average: 16 0.43 0.00 2.51 0.00 0.00 3.72 0.00
0.00 0.00 93.33
Average: 17 0.33 0.00 2.41 0.00 0.00 3.38 0.00
0.00 0.00 93.87
Average: 18 0.58 0.00 2.82 0.00 0.00 3.68 0.00
0.00 0.00 92.92
Average: 19 0.45 0.00 2.53 0.04 0.00 3.67 0.00
0.00 0.00 93.31
Average: 20 0.35 0.00 2.13 0.00 0.00 3.21 0.00
0.00 0.00 94.32
Average: 21 0.47 0.00 2.52 0.00 0.00 3.30 0.00
0.00 0.00 93.70
Average: 22 0.49 0.00 2.12 0.00 0.00 2.86 0.00
0.00 0.00 94.53
Average: 23 0.33 0.00 2.37 0.00 0.00 3.11 0.00
0.00 0.00 94.19
The config file is, I hope it is simpler and saner now:
global
daemon
maxconn 60000
quiet
nbproc 8
user haproxy
group haproxy
stats socket /var/run/haproxy.sock mode 600 level admin
stats timeout 2m
defaults
mode http
option forwardfor
retries 3
option redispatch
maxconn 60000
option splice-auto
option prefer-last-server
timeout connect 5000ms
timeout client 50000ms
timeout server 50000ms
frontend www-http
bind IP0:80
default_backend www-backend
backend www-backend
mode http
maxconn 60000
stats enable
stats uri /stats
balance roundrobin
option prefer-last-server
option forwardfor
option splice-auto
cookie FKSID prefix indirect nocache
server nginx-1 IP1:80 maxconn 20000 cookie S1 check
server nginx-2 IP2:80 maxconn 20000 cookie S2 check
server nginx-3 IP3:80 maxconn 20000 cookie S3 check
Also, during the test, the status of various backend's change often between
OK to DOWN,
and then gets back to OK almost immediately:
www-backend,nginx-3,0,0,0,10,30000,184,23843,96517588,,0,,27,0,0,180,DOWN
1/2,1,1,0,7,3,6,39,,7,3,1,,220,,2,0,,37,L4CON,,0,0,184,0,0,0,0,0,,,,0,0,,,,,6,Out
of local source ports on the system,,0,2,3,92,
Please let me know if this can be fixed, as it might help performance even
more.
In short, for small file sizes, haproxy results are *much* better than
running against a single
backend server directly (with some failures as shown above). For big files,
the numbers for
haproxy are slightly lower.
Thanks,
- Krishna Kumar
