> -----Original Message-----
> From: Peter Losher [mailto:[EMAIL PROTECTED]
> Sent: Monday, March 03, 2008 10:18 PM
> To: Ted Mittelstaedt
> Cc: [EMAIL PROTECTED]; freebsd-questions@freebsd.org
> Subject: Re: FreeBSD bind performance in FreeBSD 7
>
>
> Yeah, ISC just hates FreeBSD... <rolls eyes>

This final report here:

ftp://ftp.isc.org/isc/dns_perf/ISC-TN-2008-1.pdf

is LIGHTYEARS different than the draft here:

http://new.isc.org/proj/dnsperf/OStest.html


The draft contains the conclusion:

"...We will use Linux Gentoo 2.6.20.7 for further production testing. We
brought these numbers to the attention of the FreeBSD development team, and
will re-test when FreeBSD 7.1 is released..."

This is completely missing in the final.  Added is a bunch of
praise of bind on commodity hardware.  And also added is the line:

"...All computers in the testbed were loaded with identical copies of
FreeBSD 7.0-RELEASE..."

which is missing in the draft.

So in other words, it certainly appears that the final is
180 degrees opposite of it's discussion of FreeBSD.  The
draft appears to suggest to avoid it - the final appears to
suggest to embrace it.

So what, exactly may I ask, were you expecting after
writing that draft?  Everyone here to be happy?

It almost seems to me like the draft was a trial balloon
floated to get the FreeBSD developers to jump in and
do some coding for you at the last minute.

But, I'll say no more about that and turn towards the
report - because it has some significant problems.

I'll start with the beginning:

"...We have been particularly interested in the performance of DNSSEC
mechanisms within the DNS protocols and on the use of BIND as a production
server for major zones..."

OK, fine and good.  However, the conclusion is rather
different:

"...Commodity hardware with BIND 9 software is fast enough by a wide margin
to run large production name service..."

What is going on here?  This project started out as
purely observational - merely interested in BIND performance -
and ended up being a proof for the hypothesis that BIND
is good enough to run large nameservers on commodity hardware.

In short, the report is moving from an objective view to a
subjective goal of proving BIND is kick-ass.

It is interesting how the original draft conclusion IS NOT subjective
with regards to BIND (it is with regards to FreeBSD of course)
and uses the phrase "further production testing" implying
that BIND is still under development, while the final report
uses the language:

"...open-source software and commodity hardware are up to the task of
providing full-scale production name..."

which definitely implies that BIND is "done" and ready for
production.

Another thing of interest concernes the OS.

Microsoft Windows 2003 server is included in
the first breaking point test.  It is absent from the other
tests.  And the version chosen is old, old, it is NOT
even Server 2003 R2, nor the RC of Server 2008 which is
available.

Why were the Windows test results even left in the
published report at all?  What purpose do they serve
other than as a feel-good "bash Windows".  If you really
were interested in the results of testing, you would
have wanted to know how BIND did under Windows for the
other tests.  But, as I pointed out, by the time the
later tests were run the goal has stopped being the
pure objective observational goal, and become the
subjective "prove BIND is the best" goal.  And as the
Windows results for the breaking test were so low, it
was an embarassment to keep bothering with it, so it
was dropped.

The report also suffers from NOT listing out the
components of the HP servers and instead offering a
link to HP.  Yeah, how long is that link going to be
valid?  HP changes it's website and changes it's product
line up as often as I change my underpants - a year from
now, that product will be gone and a new reader will have
a snowball's chance in Hell of getting the actual server
specs, and I mean the chipsets in use for the disk controller,
nic card, video, etc.  You know, the stuff that actually
-affects- the performance of different operating systems.

But the biggest hole is the report conclusion and this
shift from objective, to subjective, reporting.  The conclusion
claims BIND is great on commodity hardware but what it
ONLY has proven is that BIND is great on this one specific
hardware platform running a couple specific operating systems.

If you really wanted to merely objectively observe BIND on
commodity hardware you should have had your testers stay out of the
setup of the OS and platform.  You should have called
up the developers of the various operating systems you
were going to use - Microsoft among them - and told them
to each send in a group that would build a server to their
spec.  You should have merely set a maximum limit that the
server could cost that was in line with commodity server
hardware costs - something like $2K and it had to be name-brand,
for example - and let all of the vested interest groups do
their best to create a server that would run as fast as they could
in those constraints.

In short, if the testers are setting out to prove BIND is
really powerful, they are essentially trying to write a
benchmark.  And the way you do that is by deliberatly pulling
all the stops out to make your stuff run as lickety-split as
possible - then you document the crap out of everything you
did to make it run lickety-split, so that anyone else can
come along, set up the stuff the same way you did, and then
get the same results.  Benchmarks are subjective and they
are expected to be subjective - but when you write them,
your admitting your testers are being subjective.  In that
case, there is no point in having an OS bake-off since
your going to have your testers select the OS that will
give the best shine to your product.

The report needs to make up it's mind what it's actually trying
to accomplish, objective, or subjective, reporting.

Ted

_______________________________________________
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "[EMAIL PROTECTED]"

Reply via email to