Re: [ANNOUNCE] Hello World Benchmarks, updated

2001-07-12 Thread Joshua Chamas

Joshua Chamas wrote:
> 
>  mod_caucho
> used to look a lot faster, but my testing methodology changed.
> I used to take the results of the second benchmark run, and
> publish those, but this time only ran the -test for minor
> caching after starting resin ( & tomcat ).  So, I'm guessing
> that mod_caucho compiles aggresively in the beginning, killing
> performance for a dry run ( even 60 seconds! ).  To improve
> the numbers for mod_caucho using this methodology might require
> and longer test cycle than 60 seconds.
> 

Perrin convinced me that getting "steady-state" numbers 
was important enough to ignore the compile penalties 
for such apps as mod_caucho which does runtime java
compilation.  

To this end, I have added a -prime switch to the bench.pl 
which will run all the tests for a couple seconds, before
doing the real benchmark where the results are scored.  
This allows any system caching to be done before the 
numbers start counting.

However, despite the new -prime setting, I was still not
getting reproducable resin/mod_caucho results, varying 
from 80 hits/sec one run to 280 hits/sec the next for
the Hello World 2000 JSP benchmark.

What I found is that in order for the mod_caucho results
to be reproducable to 5% variation from one run to the 
next, I need to run the benchmark for 10 minutes!  At 3-5 
minutes, there was a 10-15% variation, which seems too high.
Benchmarks < 120 seconds, which were my benchmarks from before
seem next to meaningless now.

Here's some #s to show what I mean:

Test Name   Test File  Hits/sec   # of Hits  Time(sec)  
secs/Hit   Bytes/Hit  
-   -  -  -  -  
-  -  
mod_caucho JSP 2000 h2000.jsp75.4   9069 120.33 
0.013269   28965
mod_caucho JSP 2000 h2000.jsp   249.4   2505  10.05 
0.004010   28965  
mod_caucho JSP 2000 h2000.jsp   274.5   4118  15.00 
0.003643   28965 
mod_caucho JSP 2000 h2000.jsp   191.9   5760  30.01 
0.005211   28965 
mod_caucho JSP 2000 h2000.jsp   133.7   8022  60.01 
0.007481   28965  
mod_caucho JSP 2000 h2000.jsp   104.8  12591 120.10 
0.009539   28965
mod_caucho JSP 2000 h2000.jsp96.0  11581 120.66 
0.010419   28965
mod_caucho JSP 2000 h2000.jsp   113.6  19937 175.51 
0.008803   28965  
mod_caucho JSP 2000 h2000.jsp   103.9  18283 175.98 
0.009625   28965 
mod_caucho JSP 2000 h2000.jsp91.4  27146 297.16 
0.010947   28965  
mod_caucho JSP 2000 h2000.jsp   105.0  31335 298.57 
0.009528   28965  
mod_caucho JSP 2000 h2000.jsp91.4  27210 297.76 
0.010943   28965  
mod_caucho JSP 2000 h2000.jsp88.8  52440 590.56 
0.011262   28965  
mod_caucho JSP 2000 h2000.jsp93.5  55129 589.64 
0.010696   28965  

The numbers seem to indicate that the resin/mod_caucho java 
environment gets slow the longer tests are run for, but also
that the results stabilize the longer tests are run for.

As a result, when I post benchmarks, it will be with
10 minute tests run in the future.  The idea here is
stable, repeatable benchmarks we can use.

Here were the 60 second #s I had posted before just to show
how far off the mod_caucho #s were.

> Test Name   Test File  Hits/sec   Total Hits Total Time 
>sec/Hits   Bytes/Hit
> -- -- -- -- 
>-- --
> Apache::ASP v2.19 2000  h2000.asp   226.2 13578 hits 60.02 sec  
>0.004421   28998 byte
> Apache::Registry v2.01 2000 mod_per h2000.reg   339.5 20376 hits 60.02 sec  
>0.002945   28179 byte
> HTML::Embperl v1.3.0 2000   h2000.epl   111.3 6677 hits  60.00 sec  
>0.008987   28841 byte
> HTML::Mason v1.03 2000  h2000.mas83.5 5014 hits  60.02 sec  
>0.011969   28799 byte
> HTML::Template v2.3 2000h2000.htmp   98.2 5892 hits  60.00 sec  
>0.010183   29152 byte
> mod_caucho JSP 2000 h2000.jsp76.6 4595 hits  60.01 sec  
>0.013060   28965 byte


--Josh



Re: [ANNOUNCE] Hello World Benchmarks, updated

2001-07-11 Thread Stas Bekman

On Wed, 11 Jul 2001, Philip Mak wrote:

> And sorry for my newbie-ish question, but what is the difference
> between "mod_perl handler" and "Apache::Registry mod_perl"?

http://perl.apache.org/guide/performance.html#Apache_Registry_PerlHandler_vs_
including the benchmarks


_
Stas Bekman  JAm_pH --   Just Another mod_perl Hacker
http://stason.org/   mod_perl Guide  http://perl.apache.org/guide
mailto:[EMAIL PROTECTED]   http://apachetoday.com http://eXtropia.com/
http://singlesheaven.com http://perl.apache.org http://perlmonth.com/





Re: [ANNOUNCE] Hello World Benchmarks, updated

2001-07-11 Thread Joshua Chamas

Perrin Harkins wrote:
> 
> > I do feel that compile time matters, but really with 60 seconds
> > and high MaxRequestsPerChild, these systems are getting plenty
> > of compiling caching.
> 
> The thing is, if mod_caucho takes 5 seconds the first time it hits each
> template, but is the fastest afterwards, these numbers don't give a very
> accurate picture of that.  Most people expect a hit on the first access.
> 

I get it, but I don't like it... why should an app get to spend
( hypothetically ) 60 seconds compiling a template into highly optimized
assembly at runtime, and not have to show this cost in the benchmark.

I'd be willing to run a test for longer, or run it multiple times,
but to entirely throw out the compilation phase results just seems
wrong to me.  I'd like to see some middle ground here.

> No need for anything that fancy.  I'd say just run the test once as a primer
> and throw away the results, like you were doing before.  It could be for 10
> seconds instead of 60.
> 

Another reason not to throw out the results is that it represents
a legitimate web request as part of the apps life cycle.  There
might be some user sitting at the end of that 5 second delay.

But then if we talk about throwing out highs & lows this starts
to sound almost scientific.  Like run 10 time slices, throw
out the highest & lowest times?

> I wouldn't worry about them interacting so much, but I did suggest running
> multiple times and averaging.  I think it helps smooth out random bad runs,
> which do happen now and then.
> 
> Any numbers on the new Apache::ASP CGI mode?
> 

Horrid.  Won't set up a benchmark yet, but its something like
3 hits/sec on my system compared with mod_cgi CGI.pm which is
11 hits/sec.  ASP won't be optimized for mod_cgi type execution
any time soon, which would require not loading in all the code
per request which it does now for best mod_perl use.

--Josh



Re: [ANNOUNCE] Hello World Benchmarks, updated

2001-07-11 Thread Perrin Harkins

> I do feel that compile time matters, but really with 60 seconds
> and high MaxRequestsPerChild, these systems are getting plenty
> of compiling caching.

The thing is, if mod_caucho takes 5 seconds the first time it hits each
template, but is the fastest afterwards, these numbers don't give a very
accurate picture of that.  Most people expect a hit on the first access.

> if we wanted to do away with compile
> time entirely, we'd make sure that each system got to precompile
> all its templates like Apache::RegistryLoader, Apache::ASP->Loader()
> & Embperl's Execute() in the parent httpd.

No need for anything that fancy.  I'd say just run the test once as a primer
and throw away the results, like you were doing before.  It could be for 10
seconds instead of 60.

> compile time is a very real problem for some
> types of apps like large web sites, and there is an increased
> burden on optimizing each benchmark, which implies an expertise
> in the development environment that many users may not
> have normally.  I have been trying for more out of box benchmarks,
> and not highly optimized benchmarks, using mostly the shipping
> config for a system.

I feel like allowing the templates to compile isn't tuning, just ignoring
startup costs.

> People have suggested before ( you? ) to do two runs, and
> average the results, and I think that this is a fair approach,
> and might be better than just doubling the test time because
> the systems might interact, better yet, do them in one order,
> and then reverse them, to average out sequential interactions
> of the tests on a system ( ??? )

I wouldn't worry about them interacting so much, but I did suggest running
multiple times and averaging.  I think it helps smooth out random bad runs,
which do happen now and then.

Any numbers on the new Apache::ASP CGI mode?

- Perrin




Re: [ANNOUNCE] Hello World Benchmarks, updated

2001-07-11 Thread Joshua Chamas

Perrin Harkins wrote:
>
> >  mod_caucho
> > used to look a lot faster, but my testing methodology changed.
> > I used to take the results of the second benchmark run, and
> > publish those, but this time only ran the -test for minor
> > caching after starting resin ( & tomcat ).  So, I'm guessing
> > that mod_caucho compiles aggresively in the beginning, killing
> > performance for a dry run ( even 60 seconds! ).  To improve
> > the numbers for mod_caucho using this methodology might require
> > and longer test cycle than 60 seconds.
> 
> Ouch!  I would think it's worth doing one full run to prime each system.  Or
> do you feel a need to include the initial compilation time?
> 

I do feel that compile time matters, but really with 60 seconds
and high MaxRequestsPerChild, these systems are getting plenty
of compiling caching., these are thousands of requests we are
talking about, what if we had lower MaxRequests, or big sites
with lots of templates ... if we wanted to do away with compile 
time entirely, we'd make sure that each system got to precompile 
all its templates like Apache::RegistryLoader, Apache::ASP->Loader() 
& Embperl's Execute() in the parent httpd.  

Gerald suggested this before & I think it could be good, but for 
two reasons: compile time is a very real problem for some 
types of apps like large web sites, and there is an increased 
burden on optimizing each benchmark, which implies an expertise
in the development environment that many users may not
have normally.  I have been trying for more out of box benchmarks, 
and not highly optimized benchmarks, using mostly the shipping 
config for a system.

I have thought about having lower MaxClients as an option to the 
bench to help test compile times, but this doesn't affect the 
java engines which effectively have their own backend web servers
running, like mod_proxy/mod_perl dual httpds.

People have suggested before ( you? ) to do two runs, and 
average the results, and I think that this is a fair approach,
and might be better than just doubling the test time because
the systems might interact, better yet, do them in one order,
and then reverse them, to average out sequential interactions
of the tests on a system ( ??? )

--Josh

_
Joshua Chamas   Chamas Enterprises Inc.
NodeWorks Founder   Huntington Beach, CA  USA 
http://www.nodeworks.com1-714-625-4051



Re: [ANNOUNCE] Hello World Benchmarks, updated

2001-07-11 Thread Perrin Harkins

Good work as usual, Joshua.

>  mod_caucho
> used to look a lot faster, but my testing methodology changed.
> I used to take the results of the second benchmark run, and
> publish those, but this time only ran the -test for minor
> caching after starting resin ( & tomcat ).  So, I'm guessing
> that mod_caucho compiles aggresively in the beginning, killing
> performance for a dry run ( even 60 seconds! ).  To improve
> the numbers for mod_caucho using this methodology might require
> and longer test cycle than 60 seconds.

Ouch!  I would think it's worth doing one full run to prime each system.  Or
do you feel a need to include the initial compilation time?

- Perrin




Re: [ANNOUNCE] Hello World Benchmarks, updated

2001-07-11 Thread Perrin Harkins

Good work as usual, Joshua.

>  mod_caucho
> used to look a lot faster, but my testing methodology changed.
> I used to take the results of the second benchmark run, and
> publish those, but this time only ran the -test for minor
> caching after starting resin ( & tomcat ).  So, I'm guessing
> that mod_caucho compiles aggresively in the beginning, killing
> performance for a dry run ( even 60 seconds! ).  To improve
> the numbers for mod_caucho using this methodology might require
> and longer test cycle than 60 seconds.

Ouch!  I would think it's worth doing one full run to prime each system.  Or
do you feel a need to include the initial compilation time?

- Perrin




Re: [ANNOUNCE] Hello World Benchmarks, updated

2001-07-11 Thread Joshua Chamas

Philip Mak wrote:
> 
> One thing caught my eye; how come "mod_perl handler" (808.4 hits per
> second) performed better than "HTML static" (768.2 hits per second)?
> 

Here are my comments on this from the original post:

 HTML static
for the first time, looks slower on my system than mod_perl.
It may be that the default 1.3.20 apache install/config does 
more work on the headers.  Also this is a dual proc system
which has slowed down the HTML static test before, relative
to single CPU systems.

> And sorry for my newbie-ish question, but what is the difference
> between "mod_perl handler" and "Apache::Registry mod_perl"?

Check out the full code in the benchmarks at:
  http://www.chamas.com/bench/hello.tar.gz

For the hello tests, one is an Apache::Registry run 
CGI script, and the other a small mod_perl module handler
written for the hello bench.  For the 2000 benchmark,
the Registry mod_perl is a .pl script, but uses the 
modperl API for things like headers.

-- Josh
_
Joshua Chamas   Chamas Enterprises Inc.
NodeWorks Founder   Huntington Beach, CA  USA 
http://www.nodeworks.com1-714-625-4051



Re: [ANNOUNCE] Hello World Benchmarks, updated

2001-07-11 Thread Philip Mak

One thing caught my eye; how come "mod_perl handler" (808.4 hits per
second) performed better than "HTML static" (768.2 hits per second)?

And sorry for my newbie-ish question, but what is the difference
between "mod_perl handler" and "Apache::Registry mod_perl"?





[ANNOUNCE] Hello World Benchmarks, updated

2001-07-11 Thread Joshua Chamas

Hey,

The latest Hello World benchmarks at available at:
  http://www.chamas.com/bench/hello.tar.gz

To reproduce the BELOW results on your platform, for
whatever tests are available, run:
 ./bench.pl -test
 ./bench.pl -version -time=60

--Josh


DISCLAIMER: these benchmarks test only what they test for, 
and I try to make no assertions about the fitness of any system
for your needs.  They are open source, feel free to critique
the tests as you like, and provide constructive feedback.

CHANGES: New HTML::Template 2000 test.  My system is newly
built with apache 1.3.20, mod_perl 1.25.

NOTES:
 mod_perl
The mod_perl environments are starting to look faster than PHP.
Maybe something in the default mod_perl 1.25 gave a performance boost?

 mod_caucho
used to look a lot faster, but my testing methodology changed.  
I used to take the results of the second benchmark run, and 
publish those, but this time only ran the -test for minor 
caching after starting resin ( & tomcat ).  So, I'm guessing 
that mod_caucho compiles aggresively in the beginning, killing 
performance for a dry run ( even 60 seconds! ).  To improve
the numbers for mod_caucho using this methodology might require
and longer test cycle than 60 seconds.

 HTML::Embperl
2.0 series is much faster ( than PHP & ASP too), but 1.3.0 is stable, 
& installable from CPAN, so I run that.

 HTML::Mason
< 1.0 version used to be much faster on 2000 benchmark

 mod_jserv
Why is the hello bench slower than the 2000 bench?  I don't know.
It may be a funny caching strategy.  It may be the JVMs were
cleaning up from the 2000 test during the hello test.  Speculation
only, as I don't know much about java.

 HTML static
for the first time, looks slower on my system than mod_perl.
It may be that the default 1.3.20 apache install/config does 
more work on the headers.  Also this is a dual proc system
which has slowed down the HTML static test before, relative
to single CPU systems.

Test Name   Test File  Hits/sec   Total Hits Total Time 
sec/Hits   Bytes/Hit  
-- -- -- -- 
-- -- 
Apache::ASP v2.19 2000  h2000.asp   226.2 13578 hits 60.02 sec  
0.004421   28998 byte 
Apache::Registry v2.01 2000 mod_per h2000.reg   339.5 20376 hits 60.02 sec  
0.002945   28179 byte 
HTML::Embperl v1.3.0 2000   h2000.epl   111.3 6677 hits  60.00 sec  
0.008987   28841 byte 
HTML::Mason v1.03 2000  h2000.mas83.5 5014 hits  60.02 sec  
0.011969   28799 byte 
HTML::Template v2.3 2000h2000.htmp   98.2 5892 hits  60.00 sec  
0.010183   29152 byte 
mod_caucho JSP 2000 h2000.jsp76.6 4595 hits  60.01 sec  
0.013060   28965 byte 
mod_jserv JSP 2000  h2000.jsp   150.0 9002 hits  60.02 sec  
0.006667   29408 byte 
mod_php PHP 2000h2000.php   228.4 13705 hits 60.01 sec  
0.004379   28866 byte 
Template v2.04 Toolkit 2000 h2000.tt 52.9 3176 hits  60.02 sec  
0.018900   28889 byte 
Apache::ASP v2.19   hello.asp   378.3 22706 hits 60.02 sec  
0.002643   242 bytes  
Apache::Dispatch v0.09 handler  hello/worl  587.6 35259 hits 60.01 sec  
0.001702   197 bytes  
Apache::ePerl   hello.eper  345.5 20742 hits 60.03 sec  
0.002894   218 bytes  
Apache::Registry v2.01 CGI Raw  hello_raw.  669.9 40196 hits 60.00 sec  
0.001493   52 bytes   
Apache::Registry v2.01 CGI.pm   hello.cgi   448.6 26924 hits 60.02 sec  
0.002229   217 bytes  
Apache::SSI v2.16   hello.shtm  533.8 32029 hits 60.00 sec  
0.001873   200 bytes  
HTML static hello.html  768.2 46120 hits 60.04 sec  
0.001302   312 bytes  
HTML::Embperl v1.3.0hello.epl   459.8 27595 hits 60.01 sec  
0.002175   221 bytes  
HTML::Mason v1.03   hello.mas   373.3 22406 hits 60.02 sec  
0.002679   198 bytes  
HTML::Template v2.3 hello.htmp  535.9 32165 hits 60.02 sec  
0.001866   199 bytes  
mod_caucho JSP  hello.jsp88.1 5321 hits  60.37 sec  
0.011345   231 bytes  
mod_cgi CGI Raw hello_raw.  153.5 9210 hits  60.00 sec  
0.006515   197 bytes  
mod_cgi CGI.pm  hello.cgi11.0 662 hits   60.02 sec  
0.090660   217 bytes  
mod_include SSI hello.shtm  223.2 13396 hits 60.03 sec  
0.004481   199 bytes  
mod_jserv JSP   hello.jsp53.5 3223 hits  60.29 sec  
0.018706   358 bytes  
mod_perl handlerhello.benc  808.4 48522 hits 60.02 sec  
0.001237   197 bytes  
mod_php PHP hello.php   665.9 39954 hits 60.00 sec  
0.001502   226 bytes  
mod_speedycgi   hello.