Martijn, would you mind to share the benchmark suite you have? I'd like to test a few changes.
*Bruno Borges* www.brunoborges.com.br +55 21 76727099 On Thu, Jun 16, 2011 at 1:00 PM, Igor Vaynberg <[email protected]>wrote: > lol. i was waiting for someone to say that! > > -igor > > On Thu, Jun 16, 2011 at 8:56 AM, James Carman > <[email protected]> wrote: > > Perhaps we should re-write it using wicket-velocity! ;) > > > > On Thu, Jun 16, 2011 at 11:53 AM, Igor Vaynberg <[email protected]> > wrote: > >> his test is hand-tailored to fail any true component oriented > >> framework. 5000 items per page? really? show me a single website that > >> does that. even facebook doesnt show that many comments on the wall > >> without making you page. but, after looking at other "i hate wicket" > >> posts on his blog i am not surprised with his testing approach :) > >> > >> if this situation came up in the real life any sane developer would > >> wrap the entire product list into a single component that would stream > >> html directly using string manipulation. and if you do that then > >> wicket will be on par - if not faster - then the other ones. because > >> at that point you are measuring the same thing - how fast the jvm > >> loops. > >> > >> -igor > >> > >> > >> On Thu, Jun 16, 2011 at 6:56 AM, Martijn Dashorst > >> <[email protected]> wrote: > >>> The application used in > >>> http://www.jtict.com/blog/rails-wicket-grails-play-lift-jsp/ has been > >>> controversial as it pits all popular frameworks against one another. > >>> I've tried to replicate the results on my box with AB, and did some > >>> modifications to the code: > >>> - removed the image from the markup: browsers will hit the > >>> non-existent image causing long load times > >>> - did the categories list in a similar way that the Tapestry > >>> implementation did: using a StringBuilder (saves ~300ms for a single > >>> user with 5000 items) > >>> - randomized the results for each query to the products() method such > >>> that caching the results is not an option > >>> > >>> I couldn't replicate the tapestry results with these settings (instead > >>> of being almost constant time, it now is linear but 1.5x faster than > >>> wicket). I'm not sure if the author actually looked at the results of > >>> the tapestry tests, but it seems to me that something went wrong. > >>> > >>> The code has been altered with the above changes, including changing > >>> the ListView into a RepeatingView, and rendering the categories in a > >>> StringBuilder. Here are the results of an ab test on my laptop for 200 > >>> products and 32 concurrent users: > >>> > >>> dashorst$ ab -c 32 -n 10240 > http://localhost:8080/wicketapp/products3/200 > >>> This is ApacheBench, Version 2.3 <$Revision: 655654 $> > >>> Copyright 1996 Adam Twiss, Zeus Technology Ltd, > http://www.zeustech.net/ > >>> Licensed to The Apache Software Foundation, http://www.apache.org/ > >>> > >>> Benchmarking localhost (be patient) > >>> Completed 1024 requests > >>> Completed 2048 requests > >>> Completed 3072 requests > >>> Completed 4096 requests > >>> Completed 5120 requests > >>> Completed 6144 requests > >>> Completed 7168 requests > >>> Completed 8192 requests > >>> Completed 9216 requests > >>> Completed 10240 requests > >>> Finished 10240 requests > >>> > >>> > >>> Server Software: Jetty(6.1.16) > >>> Server Hostname: localhost > >>> Server Port: 8080 > >>> > >>> Document Path: /wicketapp/products3/200 > >>> Document Length: 34689 bytes > >>> > >>> Concurrency Level: 32 > >>> Time taken for tests: 57.509 seconds > >>> Complete requests: 10240 > >>> Failed requests: 9670 > >>> (Connect: 0, Receive: 0, Length: 9670, Exceptions: 0) > >>> Write errors: 0 > >>> Total transferred: 354991304 bytes > >>> HTML transferred: 352820424 bytes > >>> Requests per second: 178.06 [#/sec] (mean) > >>> Time per request: 179.714 [ms] (mean) > >>> Time per request: 5.616 [ms] (mean, across all concurrent > requests) > >>> Transfer rate: 6028.16 [Kbytes/sec] received > >>> > >>> Connection Times (ms) > >>> min mean[+/-sd] median max > >>> Connect: 0 1 1.6 0 17 > >>> Processing: 11 179 123.2 163 1000 > >>> Waiting: 11 171 123.1 154 999 > >>> Total: 12 179 123.3 164 1000 > >>> > >>> Percentage of the requests served within a certain time (ms) > >>> 50% 164 > >>> 66% 218 > >>> 75% 248 > >>> 80% 268 > >>> 90% 337 > >>> 95% 418 > >>> 98% 491 > >>> 99% 552 > >>> 100% 1000 (longest request) > >>> > >>> The same test but then using 16 users: > >>> > >>> dashorst$ ab -c 16 -n 10240 > http://localhost:8080/wicketapp/products3/200 > >>> This is ApacheBench, Version 2.3 <$Revision: 655654 $> > >>> Copyright 1996 Adam Twiss, Zeus Technology Ltd, > http://www.zeustech.net/ > >>> Licensed to The Apache Software Foundation, http://www.apache.org/ > >>> > >>> Benchmarking localhost (be patient) > >>> Completed 1024 requests > >>> Completed 2048 requests > >>> Completed 3072 requests > >>> Completed 4096 requests > >>> Completed 5120 requests > >>> Completed 6144 requests > >>> Completed 7168 requests > >>> Completed 8192 requests > >>> Completed 9216 requests > >>> Completed 10240 requests > >>> Finished 10240 requests > >>> > >>> > >>> Server Software: Jetty(6.1.16) > >>> Server Hostname: localhost > >>> Server Port: 8080 > >>> > >>> Document Path: /wicketapp/products3/200 > >>> Document Length: 34290 bytes > >>> > >>> Concurrency Level: 16 > >>> Time taken for tests: 54.005 seconds > >>> Complete requests: 10240 > >>> Failed requests: 6161 > >>> (Connect: 0, Receive: 0, Length: 6161, Exceptions: 0) > >>> Write errors: 0 > >>> Total transferred: 355066608 bytes > >>> HTML transferred: 352895728 bytes > >>> Requests per second: 189.61 [#/sec] (mean) > >>> Time per request: 84.383 [ms] (mean) > >>> Time per request: 5.274 [ms] (mean, across all concurrent > requests) > >>> Transfer rate: 6420.58 [Kbytes/sec] received > >>> > >>> Connection Times (ms) > >>> min mean[+/-sd] median max > >>> Connect: 0 0 1.0 0 16 > >>> Processing: 11 84 51.2 74 397 > >>> Waiting: 8 77 49.4 66 397 > >>> Total: 11 84 51.2 75 398 > >>> > >>> Percentage of the requests served within a certain time (ms) > >>> 50% 75 > >>> 66% 97 > >>> 75% 113 > >>> 80% 124 > >>> 90% 154 > >>> 95% 182 > >>> 98% 217 > >>> 99% 239 > >>> 100% 398 (longest request) > >>> > >>> Now with 8 users: > >>> > >>> dashorst$ ab -c 8 -n 10240 > http://localhost:8080/wicketapp/products3/200 > >>> This is ApacheBench, Version 2.3 <$Revision: 655654 $> > >>> Copyright 1996 Adam Twiss, Zeus Technology Ltd, > http://www.zeustech.net/ > >>> Licensed to The Apache Software Foundation, http://www.apache.org/ > >>> > >>> Benchmarking localhost (be patient) > >>> Completed 1024 requests > >>> Completed 2048 requests > >>> Completed 3072 requests > >>> Completed 4096 requests > >>> Completed 5120 requests > >>> Completed 6144 requests > >>> Completed 7168 requests > >>> Completed 8192 requests > >>> Completed 9216 requests > >>> Completed 10240 requests > >>> Finished 10240 requests > >>> > >>> > >>> Server Software: Jetty(6.1.16) > >>> Server Hostname: localhost > >>> Server Port: 8080 > >>> > >>> Document Path: /wicketapp/products3/200 > >>> Document Length: 34290 bytes > >>> > >>> Concurrency Level: 8 > >>> Time taken for tests: 48.672 seconds > >>> Complete requests: 10240 > >>> Failed requests: 6308 > >>> (Connect: 0, Receive: 0, Length: 6308, Exceptions: 0) > >>> Write errors: 0 > >>> Total transferred: 355046233 bytes > >>> HTML transferred: 352875353 bytes > >>> Requests per second: 210.39 [#/sec] (mean) > >>> Time per request: 38.025 [ms] (mean) > >>> Time per request: 4.753 [ms] (mean, across all concurrent > requests) > >>> Transfer rate: 7123.72 [Kbytes/sec] received > >>> > >>> Connection Times (ms) > >>> min mean[+/-sd] median max > >>> Connect: 0 0 0.9 0 10 > >>> Processing: 11 38 19.9 33 203 > >>> Waiting: 9 35 18.6 30 203 > >>> Total: 11 38 20.0 33 204 > >>> > >>> Percentage of the requests served within a certain time (ms) > >>> 50% 33 > >>> 66% 41 > >>> 75% 47 > >>> 80% 53 > >>> 90% 65 > >>> 95% 77 > >>> 98% 93 > >>> 99% 105 > >>> 100% 204 (longest request) > >>> > >>> And with 4 users: > >>> > >>> dashorst$ ab -c 4 -n 10240 > http://localhost:8080/wicketapp/products3/200 > >>> This is ApacheBench, Version 2.3 <$Revision: 655654 $> > >>> Copyright 1996 Adam Twiss, Zeus Technology Ltd, > http://www.zeustech.net/ > >>> Licensed to The Apache Software Foundation, http://www.apache.org/ > >>> > >>> Benchmarking localhost (be patient) > >>> Completed 1024 requests > >>> Completed 2048 requests > >>> Completed 3072 requests > >>> Completed 4096 requests > >>> Completed 5120 requests > >>> Completed 6144 requests > >>> Completed 7168 requests > >>> Completed 8192 requests > >>> Completed 9216 requests > >>> Completed 10240 requests > >>> Finished 10240 requests > >>> > >>> > >>> Server Software: Jetty(6.1.16) > >>> Server Hostname: localhost > >>> Server Port: 8080 > >>> > >>> Document Path: /wicketapp/products3/200 > >>> Document Length: 34290 bytes > >>> > >>> Concurrency Level: 4 > >>> Time taken for tests: 45.238 seconds > >>> Complete requests: 10240 > >>> Failed requests: 6134 > >>> (Connect: 0, Receive: 0, Length: 6134, Exceptions: 0) > >>> Write errors: 0 > >>> Total transferred: 355063755 bytes > >>> HTML transferred: 352892875 bytes > >>> Requests per second: 226.36 [#/sec] (mean) > >>> Time per request: 17.671 [ms] (mean) > >>> Time per request: 4.418 [ms] (mean, across all concurrent > requests) > >>> Transfer rate: 7664.91 [Kbytes/sec] received > >>> > >>> Connection Times (ms) > >>> min mean[+/-sd] median max > >>> Connect: 0 0 0.2 0 7 > >>> Processing: 11 17 2.4 18 43 > >>> Waiting: 10 17 2.5 17 41 > >>> Total: 11 18 2.4 18 43 > >>> > >>> Percentage of the requests served within a certain time (ms) > >>> 50% 18 > >>> 66% 18 > >>> 75% 19 > >>> 80% 19 > >>> 90% 20 > >>> 95% 21 > >>> 98% 23 > >>> 99% 25 > >>> 100% 43 (longest request) > >>> > >>> And finally with 2 users: > >>> > >>> dashorst$ ab -c 2 -n 10240 > http://localhost:8080/wicketapp/products3/200 > >>> This is ApacheBench, Version 2.3 <$Revision: 655654 $> > >>> Copyright 1996 Adam Twiss, Zeus Technology Ltd, > http://www.zeustech.net/ > >>> Licensed to The Apache Software Foundation, http://www.apache.org/ > >>> > >>> Benchmarking localhost (be patient) > >>> Completed 1024 requests > >>> Completed 2048 requests > >>> Completed 3072 requests > >>> Completed 4096 requests > >>> Completed 5120 requests > >>> Completed 6144 requests > >>> Completed 7168 requests > >>> Completed 8192 requests > >>> Completed 9216 requests > >>> Completed 10240 requests > >>> Finished 10240 requests > >>> > >>> > >>> Server Software: Jetty(6.1.16) > >>> Server Hostname: localhost > >>> Server Port: 8080 > >>> > >>> Document Path: /wicketapp/products3/200 > >>> Document Length: 34691 bytes > >>> > >>> Concurrency Level: 2 > >>> Time taken for tests: 55.005 seconds > >>> Complete requests: 10240 > >>> Failed requests: 7532 > >>> (Connect: 0, Receive: 0, Length: 7532, Exceptions: 0) > >>> Write errors: 0 > >>> Total transferred: 355060137 bytes > >>> HTML transferred: 352889257 bytes > >>> Requests per second: 186.16 [#/sec] (mean) > >>> Time per request: 10.743 [ms] (mean) > >>> Time per request: 5.372 [ms] (mean, across all concurrent > requests) > >>> Transfer rate: 6303.74 [Kbytes/sec] received > >>> > >>> Connection Times (ms) > >>> min mean[+/-sd] median max > >>> Connect: 0 0 0.0 0 1 > >>> Processing: 6 11 2.8 11 25 > >>> Waiting: 6 10 2.8 10 22 > >>> Total: 7 11 2.8 11 25 > >>> > >>> Percentage of the requests served within a certain time (ms) > >>> 50% 11 > >>> 66% 12 > >>> 75% 13 > >>> 80% 14 > >>> 90% 15 > >>> 95% 15 > >>> 98% 16 > >>> 99% 16 > >>> 100% 25 (longest request) > >>> > >>> Looking at the results I think it is safe to say that with 16 users we > >>> hit the limit of my core i7, and with 4 users we hit the sweet spot > >>> (which is probably correct since I have 2 cores with each 2 > >>> hyperthreads). > >>> > >>> I expected 4 users to perform similar to 2 users, but it appears to be > >>> slower. I'll run it using yourkit to see if there's a blocker > >>> somewhere. > >>> > >>> In my overall experience, 1.5-SNAPSHOT performs comparable to 1.4. > >>> > >>> Notes from this test: > >>> - we report wrong length in several cases > >>> - exceptions occur when a request is broken off > >>> - performance seems on par with 1.4 > >>> > >>> Martijn > >>> > >> > > >
