I hope I am not saying something stupid (haven't met `ab` before), but wouldn't non-2xx responses mean HTTP errors? In that case you have the same amount of errors (actually, a bit more???) than requests sent. You should fix those first before running any tests.
Also, as Hank said, you should probably run ab for more than just a few seconds. Anze On Thursday 07 October 2010, Les Stroud wrote: > My goal in this testing was to produce a report for management showing how > haproxy with the four tomcats increased capacity and improved scale vs > having direct access to tomcat. I would think it would be pretty straight > forward to produce that graph. Unfortunately, the results that I received > did not line up. So, I figure that I have a misconfiguration of some > sort. Here are the details; > > So, I have 3 scenarios: > > Scenario 1: > /———>tomcat1 > /———> machine 3 — > / \———>tomcat2 > ab on machine1 ——> haproxy on machine 2 —— > \ /———->tomcat3 > \———> machine 4 — > \———>tomcat4 > > > Scenario 2: > /———>tomcat1 > /———> machine 3 — > / \———>tomcat2 > ab and haproxy on machine 2 —— > \ /———->tomcat3 > \———> machine 4 — > \———>tomcat4 > > > Scenario 3: > /———>tomcat1 > ab and haproxy on machine 3 —— > \———>tomcat2 > > > For all of the scenarios, each box is a vm with 4 procs, 12 Gb of RAM, and > fiber channel io. They are all using the latest drivers from vmware and > are patched up redhat EL 5.5 (latest). Here is the uname string: > 2.6.18-194.17.1.el5 #1 SMP Mon Sep 20 07:12:06 EDT 2010 x86_64 x86_64 > x86_64 GNU/Linux > > In all cases I tested by running the following test: > ab -n 10000 -c 100 http://tomcat1:8080/examples/jsp/ > ab -n 10000 -c 100 http://haproxy:81/examples/jsp/ > > I also ran these tests at various levels of concurrency (actually ran them > at powers of 2) and received the same results. > > Here is an example of the results that I receive: > > Scenario 1: > > ab -n 10000 -c 100 http://tomcat1:8080/examples/jsp/ > Concurrency Level: 100 > Time taken for tests: 1.602049 seconds > Complete requests: 10000 > Failed requests: 0 > Write errors: 0 > Non-2xx responses: 10001 > Total transferred: 1630163 bytes > HTML transferred: 0 bytes > Requests per second: 6242.01 [#/sec] (mean) > Time per request: 16.020 [ms] (mean) > Time per request: 0.160 [ms] (mean, across all concurrent requests) > Transfer rate: 993.10 [Kbytes/sec] received > > ab -n 10000 -c 100 http://haproxy:81/examples/jsp/ > Concurrency Level: 100 > Time taken for tests: 1.821838 seconds > Complete requests: 10000 > Failed requests: 0 > Write errors: 0 > Non-2xx responses: 10000 > Total transferred: 1930000 bytes > HTML transferred: 0 bytes > Requests per second: 5488.96 [#/sec] (mean) > Time per request: 18.218 [ms] (mean) > Time per request: 0.182 [ms] (mean, across all concurrent requests) > Transfer rate: 1034.12 [Kbytes/sec] received > > > Scenario 2: > > ab -n 10000 -c 100 http://tomcat1:8080/examples/jsp/ > Concurrency Level: 100 > Time taken for tests: 1.558019 seconds > Complete requests: 10000 > Failed requests: 0 > Write errors: 0 > Non-2xx responses: 10002 > Total transferred: 1630326 bytes > HTML transferred: 0 bytes > Requests per second: 6418.41 [#/sec] (mean) > Time per request: 15.580 [ms] (mean) > Time per request: 0.156 [ms] (mean, across all concurrent requests) > Transfer rate: 1021.81 [Kbytes/sec] received > > ab -n 10000 -c 100 http://haproxy:81/examples/jsp/ > Concurrency Level: 100 > Time taken for tests: 1.915394 seconds > Complete requests: 10000 > Failed requests: 0 > Write errors: 0 > Non-2xx responses: 10012 > Total transferred: 1932316 bytes > HTML transferred: 0 bytes > Requests per second: 5220.86 [#/sec] (mean) > Time per request: 19.154 [ms] (mean) > Time per request: 0.192 [ms] (mean, across all concurrent requests) > Transfer rate: 985.18 [Kbytes/sec] received > > Scenario 3: > > ab -n 10000 -c 100 http://tomcat1:8080/examples/jsp/ > Concurrency Level: 100 > Time taken for tests: 1.765041 seconds > Complete requests: 10000 > Failed requests: 0 > Write errors: 0 > Non-2xx responses: 10010 > Total transferred: 1631630 bytes > HTML transferred: 0 bytes > Requests per second: 5665.59 [#/sec] (mean) > Time per request: 17.650 [ms] (mean) > Time per request: 0.177 [ms] (mean, across all concurrent requests) > Transfer rate: 902.53 [Kbytes/sec] received > > ab -n 10000 -c 100 http://haproxy:81/examples/jsp/ > Concurrency Level: 100 > Time taken for tests: 2.978128 seconds > Complete requests: 10000 > Failed requests: 0 > Write errors: 0 > Non-2xx responses: 10023 > Total transferred: 1934439 bytes > HTML transferred: 0 bytes > Requests per second: 3357.81 [#/sec] (mean) > Time per request: 29.781 [ms] (mean) > Time per request: 0.298 [ms] (mean, across all concurrent requests) > Transfer rate: 634.29 [Kbytes/sec] received > > I appreciate any insights you can provide — or even just ideas :) > > LES > > On Oct 7, 2010, at 12:18 AM, Hank A. Paulson wrote: > > What did the haproxy stats web page show during the test? > > How long was each test run? many people seem to run ab for a few seconds. > > Was tomcat "doing" anything for the test urls, I am a bit shocked you got > > 3700 rps from tomcat. Most apps I have seen on it fail at much lower > > rps. > > > > Raise the max conn for each server and for the front end and see if you > > get better results. > > > > On 10/6/10 7:11 PM, Les Stroud wrote: > >> I did a little more digging and found several blogs that suggest that I > >> will take a performance hit on virtual platforms. In fact, this guy > >> (http://www.mail-archive.com/[email protected]/msg03119.html) seems > >> to have the same problem. The part that is concerning me is not the > >> overall performance, but that I am getting worse performance with 4 > >> servers than I am with 1 server. I realize there are a lot of > >> complications, but I have to be doing something very wrong to get a > >> decrease. > >> > >> I have even tried putting haproxy on the same server with 2 tomcat > >> servers and used 127.0.0.1 to take as much of the network out as > >> possible. I still get a lower number of requests per second when going > >> through haproxy to the 2 tomcats (as opposed to going directly to one > >> of the tomcats). This test is using ab locally on the same machine. > >> > >> I have tried all of the sysctl settings that I have found listed on the > >> board. Is there anything I am missing?? > >> > >> I appreciate the help, > >> Les Stroud > >> > >> On Oct 6, 2010, at 3:56 PM, Les Stroud wrote: > >>> I’ve figured I would find answers to this in the archive, but have been > >>> unable to. So, I appreciate the time. > >>> > >>> I am setting up an haproxy instance in front of some tomcat instances. > >>> As a test, I ran ab against one of the tomcat instances directly with > >>> an increasing number of concurrent connections. I then repeated the > >>> same test with haproxy fronting 4 tomcat servers. I was hoping to see > >>> that the haproxy setup would perform a higher number of requests per > >>> second and hold that higher number with increasingly high traffic. > >>> Unfortunately, it did not. > >>> > >>> Hitting the tomcat servers directly, I was able to get in excess of > >>> 3700 rqs/s. With haproxy in front of that tomcat instance and three > >>> others (using roundrobin), I never surpassed 2500. I also did not find > >>> that I was able to handle an increased amount of concurrency (both > >>> started giving errors around 20000). > >>> > >>> I have tuned the tcp params on the linux side per the suggestions I > >>> have seen on here. Are there any other places I can start to figure > >>> out what I have wrong in my configuration?? > >>> > >>> Thanx, > >>> LES > >>> > >>> > >>> ——— > >>> > >>> haproxy.cfg > >>> > >>> global > >>> #log loghost local0 info > >>> maxconn 500 > >>> nbproc 4 > >>> stats socket /tmp/haproxy.sock level admin > >>> defaults > >>> log global > >>> clitimeout 60000 > >>> srvtimeout 30000 > >>> contimeout 4000 > >>> retries 3 > >>> option redispatch > >>> option httpclose > >>> option abortonclose > >>> > >>> listen stats 192.168.60.158:8081 > >>> mode http > >>> stats uri /stat #Comment this if you need to specify diff stat path for > >>> viewing stat page > >>> stats enable > >>> listen erp_cluster_https 0.0.0.0:81 > >>> mode http > >>> balance roundrobin > >>> option forwardfor except 0.0.0.0 > >>> reqadd X-Forwarded-Proto:\ https > >>> cookie SERVERID insert indirect > >>> server tomcat01-instance1 192.168.60.156:8080 cookie A check > >>> server tomcat01-instance2 192.168.60.156:18080 cookie A check > >>> server tomcat02-instance1 192.168.60.157:8080 cookie A check > >>> server tomcat02-instance2 192.168.60.157:18080 cookie A check
