On Sat, 26 Aug 2006, Vincent Diepeveen wrote:
Find me 1 site that 'tests' hardware that's objective. Spec is the best
compromise.
I don't know about site testing hardware, but there are many objective
hardware tests on sites. In particular, lmbench is an excellent and
unbiased toolset (in my personal belief, having communicated fairly
extensively with Larry and Carl, knowing e.g. that Linus uses lmbench
routinely to test and tune the linux kernel). benchmaster (my own
microbenchmark suite, ex. cpu_rate) isn't finished, really, but it is
definitely objective. netpipe is quite objective. I've never heard
bonnie accused of being biased. Stream is objective, and damn simple
code at that. I really dunno about the new code being developed for
top500 testing, but I do trust to SOME extent the folks developing it --
the bias if any will be in that it is focussed on HPC and perhaps
certain classes of code WITHIN HPC.
Macro benchmarks are much tougher, as they are really in the category of
"is this program "like" my program" for the component(s) of the suite or
tool. They also tend to be relatively "rich" code environments, where
compilers are tested as much or more than just the hardware. This is
always true, I supposed, but it makes it difficult to separate
comparative analysis of different hardware from the compiler even when
the same compiler is used, especially across architectures. Leaving
aside the issue of whether or not one SHOULD try to differentiate
compiler from the hardware. Microbenchmarks are probably better for
that purpose as in many cases the core code fragments they time are
small enough that there isn't that much variation in their assembler
implementation, at any rate.
I am a longstanding, fairly passionate advocate of totally open (GPL)
benchmarking code. I was one of the folks who talked John into opening
up lmbench and liberalizing its fairly restrictive original license,
pointing out that the synergy obtained by freeing the code was of
greater importance than the control he was trying to maintain to prevent
vendor abuse. In the end, I don't think anyone has abused lmbench in
part because it has never become a major marketing tool (unlike certain
other benchmark suites I can name).
One of the things that has LONG irritated me is that SPEC isn't open
source, isn't extensible, isn't freely available. I'm certain that
there are reasons for this -- they could even be "good" reasons. That
doesn't mean I have to like it, or think that the world doesn't need a
truly open alternative. However, this needs at least one human to "own"
the project of creating the alternative. I myself cannot do it -- I
already own a languishing microbenchmark suite, a cluster monitoring
toolset, a random number tester, and an XML-based flashcard presentation
program, and none of them get enough attention as it is. If somebody
DID put one together and own it, though, I'd contribute, help, clap
loudly, cheer.
rgb
Vincent
----- Original Message ----- From: "Geoff Jacobs" <[EMAIL PROTECTED]>
To: <[email protected]>
Sent: Friday, August 25, 2006 9:01 PM
Subject: Re: [Beowulf] SPEC CPU 2006 released
Mark Hahn wrote:
The new SPEC cpu suite has been released, and initial results
are posted to:
is this one still stupidly not open-source?
I don't think SPEC has any open source benchmarks.
As far as I can tell, SPEC groups derive income by licensing their
benchmarks to integrators and such (although I know SpecViewPerf is free
to use). I would think they also have issues with proprietary 3rd party
software within their benchmarks.
I think they could be supported in the same manner as OSDL, but they are
not, so no open source from SPEC.
--
Geoffrey D. Jacobs
Go to the Chinese Restaurant,
Order the Special
_______________________________________________
Beowulf mailing list, [email protected]
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf
_______________________________________________
Beowulf mailing list, [email protected]
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf
--
Robert G. Brown http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567 Fax: 919-660-2525 email:[EMAIL PROTECTED]
_______________________________________________
Beowulf mailing list, [email protected]
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf
