* * * Sarcasm Alert * * *
Severe?
You get your hand slapped. Oooooo. And it used to be in private. Now you
end up with a permanent entry in the database labeled "non-compliant
results." (and there were complaints that this was too heavy handed!)
Not even a fine. TPC *fines* their cheaters.
Severe. Hmmph.
-gt
* * * End Sarcasm Alert * * *
P.S. Seriously, thanks for defending benchmarking. You are right to jump to
its defense. Just be prepared for a rebuttal from someone when you mention
penalties. The biggest impediment to cheating in the consortia, in my
opinion, is that everyone is watching and that most people care about being
fair. In SPEC, the arrival of Microsoft almost put that model at risk. We
can chat more about it sometime.
-----Original Message-----
From: Ezolt, Phillip
Sent: Thursday, January 20, 2000 9:41 AM
To: [EMAIL PROTECTED]
Subject: Re: Transmeta announces
Derek,
> Every computer science text I've ever read that had anything to say on the
> matter disagrees with you, owing to a variety of reasons. The biggest one
> being that benchmarking programs tend to become rather well-known, and
> vendors have a habit of optimizing their compilers to detect when they are
> being compiled, and to produce code that will yeild a substantial
> performance increase in the case of the benchmark, but in real-world
> situations the code that gets compiled runs much slower.
That is only the case with the unscrupulous vendor. "Benchmark
specials" hurt everybody, including the vendor. Users see poorer
performance, and the vendor loses credibility.
If there is a change going into our compilers for a SPEC benchmark, we
make sure that it is a general change that will help all programs
compiled by the compiler.
This is not even a problem is the SPEC benchmark is representative of
what a typical user does. If a system is optimized for a benchmark,
and benchmark represents what a user would do, then that system will be
optimized for the user as well.
In any event, this type of behavior is strictly prohibited by SPEC,
and penalties for violating it are severe.
--Phil
Compaq: High Performance Server Division/Benchmark Performance Engineering
---------------- Alpha, The Fastest Processor on Earth --------------------
[EMAIL PROTECTED] |C|O|M|P|A|Q| [EMAIL PROTECTED]
------------------- See the results at www.spec.org -----------------------
On Thu, 20 Jan 2000, Derek Martin wrote:
> On Thu, 20 Jan 2000, Phillip Ezolt wrote:
>
> > On Wed, 19 Jan 2000, Derek Martin wrote:
> > > So you're looking for what, MIPS? MIPS="Meaningless Information Per
> > > Second" or "Meaningless Indication of Processor Speed"
> > >
> > > Benchmarks are all pretty useless. They can be made/optimized to say
> > > whatever you want them to say. Let me demo one so I can SEE how it
> > > performs...
> >
> > Not true. All benchmarks give meaningfull information. It may or may
> > not be a piece of the system that the user cares about, but it DOES
> > say something about how the system performs. It is up to the user to
> > say "This does not apply to me".
>
> Every computer science text I've ever read that had anything to say on the
> matter disagrees with you, owing to a variety of reasons. The biggest one
> being that benchmarking programs tend to become rather well-known, and
> vendors have a habit of optimizing their compilers to detect when they are
> being compiled, and to produce code that will yeild a substantial
> performance increase in the case of the benchmark, but in real-world
> situations the code that gets compiled runs much slower.
>
> See ch. 2, Computer Organization & Design, Hennessey & Patterson, for
> example. ESR also has a number of remarks in the Jargon File, from whence
> my definitions of MIPS (above) came.
>
> --
> "Quis custodiet ipsos custodes?" "Who watches the watchmen?"
> -Juvenal, Satires, VI, 347
>
> Derek D. Martin | Senior UNIX Systems/Network Administrator
> Arris Interactive | A Nortel Company
> [EMAIL PROTECTED] | [EMAIL PROTECTED]
> -------------------------------------------------
>
>
> **********************************************************
> To unsubscribe from this list, send mail to
> [EMAIL PROTECTED] with the following text in the
> *body* (*not* the subject line) of the letter:
> unsubscribe gnhlug
> **********************************************************
>
>
**********************************************************
To unsubscribe from this list, send mail to
[EMAIL PROTECTED] with the following text in the
*body* (*not* the subject line) of the letter:
unsubscribe gnhlug
**********************************************************
**********************************************************
To unsubscribe from this list, send mail to
[EMAIL PROTECTED] with the following text in the
*body* (*not* the subject line) of the letter:
unsubscribe gnhlug
**********************************************************
