Hi Roy.
Roy Wilson wrote:
> Jason,
>
> This is an impressive piece of work: very systematic and very
> well-documented.
I wrote it to be similar to the Volano Java benchmarks. (I really enjoy
those! :)
> I have several questions. I ran some similar tests
> (without adjusting configuration parameters) and noticed variability as
> large as 10% in throughput that surprised me, even when N = 10,000.
You mean your benchmarks in comparison to mine? Not sure what you're
comparing here..
> To
> reduce that, I felt compelled to use N = 100,000. Your data seems pretty
> stable: did you run more tests and just truncate the reporting?
I did run lots of test benchmarks to tune first.. but the numbers in my text
are "real", since right before the real (shown) tests I shut the server down
and restarted it. Something I didn't note in my text is that I wasn't
trying
to benchmark initialization.. I didn't want my benchmark numbers to
include first-request kinds of initialization lag times.. so between
restarting
the server and when I ran the benchmarks I used ab to make a few single
requests to make sure both Apache and Catalina were initialized enough
to not show horrible request times due mostly to initialization. So, the
first benchmark test shows slower numbers than the second does. I should
have noted this in the text, or I shouldn't have done it. It seems that no
matter how verbose you try to be with tests like this, you end up forgetting
something anyway. :)
Probably I should have just let the first test be slow, and explain what
happened. Either way, at least the second test for each kind of test
contains
some good solid numbers.
> Do you
> think it would be worthwhile to rerun the test with N = 100,000 maybe two
> or three times ( my tests took hours )?
With that many requests per benchmark, I can certainly see why the tests
took
hours! :) I came down to 10,000 requests because I felt that the
system must
have stabilized in that amount of requests (I could be wrong), but also
because
100,000 was simply taking too long. Whether the numbers would have turned
out very differently is anyone's guess until someone does it.. Maybe
I'll try.
> Costin's ab data shows a ramp up: the shell script I posted a while back
> was based on that approach.
I liked the scripts, but couldn't use them only because I found that I had
intentionally *not* installed the process accounting package when I
installed RedHat on my laptop.. :) So, I had to drop that part, and that
was the main purpose for the scripts as far as I could tell. I will
probably
install the process accounting package at some point so I can try it out.
I have another machine at home now that is bigger and faster, so I may
run more benchmarks on that machine, maybe even in an automated
fashion.
> I noticed doing my measurements that Apache
> throughput increased as C increased (up to C = 100), whereas TC
> throughput did not (actually it declined). I also wonder what the
> architectural explanation might be for the different scalability behavior
> (maybe this is obvious, I haven't thought about it yet). I wonder if you
> could (in your spare time :-)} repeat the test 3 times for C = 20, 40,
> 60, 80, 100 with N = 100,000.
I will try this as soon as I get a chance.. But, one thing that may be a
problem is the request-per-second limits of our boxes. If each of our
machines has a limit to the number of requests per second they can
handle and we run benchmarks beyond those numbers, then we're no
longer seeing accurate benchmarks -- we'll see some slow response
times, or even failed requests and it won't be because the server
software failed. Take a look at this:
http://www.enhydra.org/software/cvs/cvsweb.cgi/~checkout~/Enhydra/modules/EnhydraDirector/src/README.TUNING?rev=1.1&content-type=text/plain
And also this:
http://httpd.apache.org/docs/misc/descriptors.html
By reading those, you'll see that there are many upper limits that
you could hit without really knowing.. So, if you're running high
traffic benchmarks and you're not watching for these problems,
then your benchmark numbers are probably bad/wrong at the high
end.
I want to try your benchmarks like c=100 and N=100,000, but
first I want to make sure my machine is ready for that.. :)
> That's my plan, along with using sar and sa
> with C = 1 to get CPU demand measures.
I need to install those too..
> Maybe others will join this little band of performance freaks.
Oh I'm sure someone out there reading this will. :)
> BTW, little credit should go to me re: ab use (or should I say "abuse").
> Craig suggested using ab and Costin did it (and did it, ... :-)). Pier
> "inspired" me to look at ab code and think about how it can be used as a
> tool for getting information about both behavior and the resource usage
> that drives it.
Okay, they each get my special thanks as well!
> Your stuff (along with Costin's) might be a key part of a
> great guide to TCx performance and tuning ala Dean Gaudet's stuff on
> (Apache?) performance.
That sounds like a much needed guide. Maybe a HOWTO?
> Now I need to study your stuff more closely and
> plan for the next round of measurement (and, in my case, modeling).
>
> Roy
>
Have fun!
--
Jason Brittain
Software Engineer, Olliance Inc. http://www.Olliance.com
Current Maintainer, Locomotive Project http://www.Locomotive.org
