On 05/05/2015 02:06 μμ, Krishna Kumar (Engineering) wrote: > Hi Willy, Pavlos, > > Thank you once again for your advice. > > > Requests per second: 19071.55 [#/sec] (mean) > > Transfer rate: 9461.28 [Kbytes/sec] received > > These numbers are extremely low and very likely indicate an http > close mode combined with an untuned nf_conntrack. > > > Yes, it was due to http close mode, and wrong irq pinning > (nf_conntrack_max was > set to 640K). > > > > mpstat (first 4 processors only, rest are almost zero): > > Average: CPU %usr %nice %sys %iowait %irq %soft %steal > > %guest %gnice %idle > > Average: 0 0.25 0.00 0.75 0.00 0.00 98.01 0.00 > > 0.00 0.00 1.00 > > This CPU is spending its time in softirq, probably due to conntrack > spending a lot of time looking for the session for each packet in too > small a hash table. > > > I had not done irq pinning. Today I am getting much better results with > irq pinning > and keepalive. > > Note, this is about 2 Gbps. How is your network configured ? You should > normally see either 1 Gbps with a gig NIC or 10 Gbps with a 10G NIC, > because retrieving a static file is very cheap. Would you happen to be > using bonding in round-robin mode maybe ? If that's the case, it's a > performance disaster due to out-of-order packets and could explain some > of the high %softirq. > > > My setup is as follows (no bonding, etc, and Sys stands for baremetal > system, each with > 48 core, 128GB mem, ixgbe single ethernet port card). > > Sys1-with-ab <-eth0-> Sys1-with-Haproxy, which uses two nginx > backend systems > over the same eth0 card (that is the current restriction, no extra > ethernet interface for > separate frontend/backend, etc). Today I am getting a high of 7.7 Gbps > with your > suggestions. Is it possible to get higher than that (direct to server > gets 8.6 Gbps)? > > Please retry without http-server-close to maintain keep-alive to the > servers, that will avoid the session setup/teardown. If that becomes > better, there's definitely something to fix in the conntrack or maybe > in iptables rules if you have some. But in any case don't put such a > > > There are a few iptables rules, which seem clean. The results now are: >
Shall I assume that you have run the same tests without iptables and got the same results? May I suggest to try also httpress and wrk tool? Have you compared 'sysctl -a' between haproxy and nginx server? > ab -k -n 1000000 -c 500 http://<haproxy>:80/64 (I am getting some > errors though, > which is not present when running against the backend directly): > > Document Length: 64 bytes > Concurrency Level: 500 > Time taken for tests: 6.181 seconds > Complete requests: 1000000 > Failed requests: 18991 > (Connect: 0, Receive: 0, Length: 9675, Exceptions: 9316) > Write errors: 0 > Keep-Alive requests: 990330 > Total transferred: 296554848 bytes > HTML transferred: 63381120 bytes > Requests per second: 161783.42 [#/sec] (mean) > Time per request: 3.091 [ms] (mean) > Time per request: 0.006 [ms] (mean, across all concurrent requests) > Transfer rate: 46853.18 [Kbytes/sec] received > > Connection Times (ms) > min mean[+/-sd] median max > Connect: 0 0 0.5 0 8 > Processing: 0 3 6.2 3 1005 > Waiting: 0 3 6.2 3 1005 > Total: 0 3 6.3 3 1010 > > Percentage of the requests served within a certain time (ms) > 50% 3 > 66% 3 > 75% 3 > 80% 3 > 90% 4 > 95% 5 > 98% 6 > 99% 8 > 100% 1010 (longest request) > > pidstat (some system numbers are very high, 50%, maybe due to small > packet sizes?): > Average: UID PID %usr %system %guest %CPU CPU Command > Average: 110 52601 6.00 9.33 0.00 15.33 - haproxy > Average: 110 52602 6.33 11.83 0.00 18.17 - haproxy > Average: 110 52603 11.33 17.83 0.00 29.17 - haproxy > Average: 110 52604 17.50 30.33 0.00 47.83 - haproxy > Average: 110 52605 20.50 38.50 0.00 59.00 - haproxy > Average: 110 52606 24.50 51.33 0.00 75.83 - haproxy > Average: 110 52607 22.50 51.33 0.00 73.83 - haproxy > Average: 110 52608 23.67 47.17 0.00 70.83 - haproxy > > mpstat (of interesting cpus only): > Average: CPU %usr %nice %sys %iowait %irq %soft > %steal %guest %gnice %idle > Average: all 2.58 0.00 4.36 0.00 0.00 0.89 > 0.00 0.00 0.00 92.17 > Average: 0 6.84 0.00 11.46 0.00 0.00 2.03 > 0.00 0.00 0.00 79.67 > Average: 1 11.15 0.00 19.85 0.00 0.00 5.29 > 0.00 0.00 0.00 63.71 > Average: 2 8.32 0.00 12.20 0.00 0.00 2.22 > 0.00 0.00 0.00 77.26 > Average: 3 7.92 0.00 11.97 0.00 0.00 2.39 > 0.00 0.00 0.00 77.72 > Average: 4 8.81 0.00 13.76 0.00 0.00 2.39 > 0.00 0.00 0.00 75.05 > Average: 5 6.96 0.00 12.27 0.00 0.00 2.38 > 0.00 0.00 0.00 78.39 > Average: 6 9.21 0.00 12.52 0.00 0.00 3.31 > 0.00 0.00 0.00 74.95 > Average: 7 7.56 0.00 13.65 0.00 0.00 2.95 > 0.00 0.00 0.00 75.83 > Average: 8 6.33 0.00 10.80 0.00 0.00 1.86 > 0.00 0.00 0.00 81.01 > Average: 9 7.27 0.00 11.82 0.00 0.00 2.36 > 0.00 0.00 0.00 78.55 > Average: 10 6.80 0.00 12.68 0.00 0.00 2.94 > 0.00 0.00 0.00 77.57 > Average: 11 6.97 0.00 10.64 0.00 0.00 2.20 > 0.00 0.00 0.00 80.18 > Average: 12 6.33 0.00 12.12 0.18 0.00 2.71 > 0.00 0.00 0.00 78.66 > Average: 13 5.83 0.00 11.84 0.00 0.00 2.37 > 0.00 0.00 0.00 79.96 > Average: 14 5.73 0.00 12.01 0.00 0.00 1.97 > 0.00 0.00 0.00 80.29 > Average: 15 6.01 0.00 11.11 0.00 0.00 2.00 > 0.00 0.00 0.00 80.87 > Average: 16 3.24 0.00 4.26 0.00 0.00 0.85 > 0.00 0.00 0.00 91.65 > Average: 17 4.17 0.00 7.64 0.00 0.00 1.39 > 0.00 0.00 0.00 86.81 > Average: 18 1.18 0.00 1.85 0.00 0.00 0.17 > 0.00 0.00 0.00 96.81 > Average: 19 1.35 0.00 2.03 0.00 0.00 0.17 > 0.00 0.00 0.00 96.45 > Average: 20 0.68 0.00 0.84 0.00 0.00 0.17 > 0.00 0.00 0.00 98.31 > Average: 21 0.34 0.00 0.50 0.00 0.00 0.17 > 0.00 0.00 0.00 98.99 > Average: 22 1.01 0.00 1.34 0.00 0.00 0.34 > 0.00 0.00 0.00 97.32 > Average: 25 1.00 0.00 1.67 0.00 0.00 0.67 > 0.00 0.00 0.00 96.66 > > Similarly for large IO: > ab -k -n 100000 -c 500 http://<haproxy>:80/512K > > Document Length: 524288 bytes > Concurrency Level: 500 > Time taken for tests: 52.190 seconds > Complete requests: 100000 > Failed requests: 0 > Write errors: 0 > Keep-Alive requests: 100000 > Total transferred: 52452889936 bytes > HTML transferred: 52428800000 bytes > Requests per second: 1916.08 [#/sec] (mean) > Time per request: 260.950 [ms] (mean) > Time per request: 0.522 [ms] (mean, across all concurrent requests) > Transfer rate: 981483.06 [Kbytes/sec] received > > Connection Times (ms) > min mean[+/-sd] median max > Connect: 0 0 0.4 0 9 > Processing: 1 257 244.9 217 6879 > Waiting: 0 23 71.3 5 1476 > Total: 1 257 244.9 217 6879 > > Percentage of the requests served within a certain time (ms) > 50% 217 > 66% 273 > 75% 355 > 80% 441 > 90% 560 > 95% 739 > 98% 932 > 99% 1106 > 100% 6879 (longest request) > > pidstat: > Average: UID PID %usr %system %guest %CPU CPU Command > Average: 110 52601 1.71 8.82 0.00 10.53 - haproxy > Average: 110 52602 2.08 11.67 0.00 13.75 - haproxy > Average: 110 52603 1.18 7.47 0.00 8.65 - haproxy > Average: 110 52604 1.73 9.51 0.00 11.24 - haproxy > Average: 110 52605 1.71 9.53 0.00 11.24 - haproxy > Average: 110 52606 1.86 10.45 0.00 12.31 - haproxy > Average: 110 52607 2.63 12.82 0.00 15.45 - haproxy > Average: 110 52608 1.31 7.35 0.00 8.67 - haproxy > HAProxy is sleeping and the system as well, which means something else kills your performance. > mpstat: > Average: CPU %usr %nice %sys %iowait %irq %soft > %steal %guest %gnice %idle > Average: all 0.23 0.00 1.25 0.01 0.00 1.84 > 0.00 0.00 0.00 96.68 > Average: 0 0.41 0.00 2.57 0.29 0.00 4.17 > 0.00 0.00 0.00 92.56 > Average: 1 0.58 0.00 2.82 0.00 0.00 4.69 > 0.00 0.00 0.00 91.91 > Average: 2 0.49 0.00 2.62 0.00 0.00 3.77 > 0.00 0.00 0.00 93.11 > Average: 3 0.62 0.00 2.89 0.00 0.00 5.13 > 0.00 0.00 0.00 91.36 > Average: 4 0.60 0.00 2.71 0.00 0.00 3.76 > 0.00 0.00 0.00 92.94 > Average: 5 0.45 0.00 2.69 0.00 0.00 3.45 > 0.00 0.00 0.00 93.40 > Average: 6 0.45 0.00 2.71 0.00 0.00 3.87 > 0.00 0.00 0.00 92.97 > Average: 7 0.45 0.00 2.18 0.00 0.00 3.56 > 0.00 0.00 0.00 93.81 > Average: 8 0.49 0.00 2.29 0.00 0.00 3.85 > 0.00 0.00 0.00 93.36 > Average: 9 0.49 0.00 2.63 0.00 0.00 4.01 > 0.00 0.00 0.00 92.86 > Average: 10 0.52 0.00 2.71 0.00 0.00 4.28 > 0.00 0.00 0.00 92.50 > Average: 11 0.61 0.00 3.15 0.00 0.00 4.05 > 0.00 0.00 0.00 92.19 > Average: 12 0.35 0.00 2.17 0.00 0.00 3.28 > 0.00 0.00 0.00 94.20 > Average: 13 0.47 0.00 2.70 0.00 0.00 3.95 > 0.00 0.00 0.00 92.89 > Average: 14 0.56 0.00 2.41 0.00 0.00 3.79 > 0.00 0.00 0.00 93.24 > Average: 15 0.45 0.00 2.56 0.00 0.00 3.84 > 0.00 0.00 0.00 93.15 > Average: 16 0.43 0.00 2.51 0.00 0.00 3.72 > 0.00 0.00 0.00 93.33 > Average: 17 0.33 0.00 2.41 0.00 0.00 3.38 > 0.00 0.00 0.00 93.87 > Average: 18 0.58 0.00 2.82 0.00 0.00 3.68 > 0.00 0.00 0.00 92.92 > Average: 19 0.45 0.00 2.53 0.04 0.00 3.67 > 0.00 0.00 0.00 93.31 > Average: 20 0.35 0.00 2.13 0.00 0.00 3.21 > 0.00 0.00 0.00 94.32 > Average: 21 0.47 0.00 2.52 0.00 0.00 3.30 > 0.00 0.00 0.00 93.70 > Average: 22 0.49 0.00 2.12 0.00 0.00 2.86 > 0.00 0.00 0.00 94.53 > Average: 23 0.33 0.00 2.37 0.00 0.00 3.11 > 0.00 0.00 0.00 94.19 > > The config file is, I hope it is simpler and saner now: > > global > daemon > maxconn 60000 > quiet > nbproc 8 > user haproxy > group haproxy > stats socket /var/run/haproxy.sock mode 600 level admin > stats timeout 2m > > defaults > mode http > option forwardfor > retries 3 > option redispatch > maxconn 60000 > option splice-auto > option prefer-last-server > timeout connect 5000ms > timeout client 50000ms > timeout server 50000ms > > frontend www-http > bind IP0:80 > default_backend www-backend > > backend www-backend > mode http > maxconn 60000 > stats enable > stats uri /stats > balance roundrobin > option prefer-last-server > option forwardfor > option splice-auto > cookie FKSID prefix indirect nocache > server nginx-1 IP1:80 maxconn 20000 cookie S1 check > server nginx-2 IP2:80 maxconn 20000 cookie S2 check > server nginx-3 IP3:80 maxconn 20000 cookie S3 check > > Also, during the test, the status of various backend's change often > between OK to DOWN, Have you checked if you got all backends reported down at the same time? How many workers do you use on your Nginx which acts as LB? > and then gets back to OK almost immediately: > This flapping could cause performance problems, you are strong as your weakest link they say:-) > www-backend,nginx-3,0,0,0,10,30000,184,23843,96517588,,0,,27,0,0,180,DOWN > 1/2,1,1,0,7,3,6,39,,7,3,1,,220,,2,0,,37,L4CON,,0,0,184,0,0,0,0,0,,,,0,0,,,,,6,Out > of local source ports on the system,,0,2,3,92, > Hold on a second, what is this 'Out of local source ports on the system' message? ab reports 'Concurrency Level: 500' and you said that HAProxy runs in keepalive mode(default on 1.5 releases) which means there will be only 500 TCP connections opened from HAProxy towards the backends, which it isn't that high and you shouldn't get that message unless net.ipv4.ip_local_port_range is very small( I don't think so). I have done the same stress tests against HAProxy(HAPEE) and Nginx+ and never got so bad results. But, my setup was a bit different as I used only physical hardware on HAProxy/Nginx and on backends(Nginx servers). I also had dual-link setup with north and south side. Look my results with 6 processes # wrk --timeout 3s --latency -c 1000 -d 5m -t 24 http://a.b.c.d Running 5m test @ http://a.b.c.d 24 threads and 1000 connections Thread Stats Avg Stdev Max +/- Stdev Latency 87.07ms 593.84ms 7.85s 95.63% Req/Sec 16.45k 7.43k 60.89k 74.25% Latency Distribution 50% 1.75ms 75% 2.40ms 90% 3.57ms 99% 3.27s 111452585 requests in 5.00m, 15.98GB read Socket errors: connect 0, read 0, write 0, timeout 33520 Requests/sec: 371504.85 Transfer/sec: 54.56MB The timeouts were coming from backend servers. > Please let me know if this can be fixed, as it might help performance > even more. > > In short, for small file sizes, haproxy results are *much* better than > running against a single > backend server directly (with some failures as shown above). For big > files, the numbers for > haproxy are slightly lower. > I got different results on my tests.
signature.asc
Description: OpenPGP digital signature
