On Fri, 28 Jan 2000, Joshua Chamas wrote:
> I have updated the Hello World Web Application Benchmarks,
> now at http://www.chamas.com/bench/
I guess this is just my day to complain about benchmarks. I find these
benchmarks very confusing and not very useful. Benchmarks done on
different hardware can't be compared to tell anything about the relative
speeds of different languages and development tools. Most of the
contributors have benchmarked one or two things on their preferred system
and stopped, making it impossible to compare much of anything. You can't
say anything about Velocigen relative to anything else, because that guy
only did Velocigen testing. You can't say anything about Java servlets
under Resin relative to ModPerl handlers, because that guy only
benchmarked mod_perl doing CGI emulation. Even the benchmark tools are
different in many cases, which has a HUGE effect on numbers like these.
The end result of all this is that you have benchmark numbers which, while
sort of entertaining, should not be used to make any kind of decision. I
would hate to think that someone used these numbers as the basis for
making decisions and didn't benchmark the options himself. I almost fell
into that trap when I saw the low scores for certain things and wrote them
off without doing my own testing. (Yes, I know a good developer should
never believe other people's benchmarks.)
Although there is a disclaimer on the page, I wish it said more. In some
ways, it's worse to have these misleading benchmarks than no benchmarks at
all. I shudder to think what a naive manager might do with these.
Okay, I'm done complaining for now. Sorry Joshua, I know you put effort
into this, and I do appreciate it.
- Perrin
