On Mon, 4 Sep 2006, Joe Landman wrote:
The most important benchmark is the one that uses the same code you use in the way you are going to use it.
True (and I generally agree with everything else Joe said as well except that I think he meant a one-dimensional projection, or even two, of a multidimensional function, not really zero).
Anything else is an entropy generator.
Well, no, not really. They can reduce the entropy of the decision space, they just don't live in the exact same projectional subspace as the problem itself, so at most they yield (hopefully important) components of the expected performance. How important those components are depends in large part in how close the projections are. A monte carlo benchmark CAN be a good predictor for other similar monte carlo programs, not just the one that generated the benchmark. Code that is bound by a given stream-like linear operation CAN be well-predicted by stream. The more mixed the code, the more unlike all the component benchmarks, the less useful. And to be fair, although people tend to quote e.g. "specint" or "specfp" performance (which are cumulative/projective) one can go in and look at performance on the components, which is perhaps more useful if one of them has a good projective overlap with your task. None of which changes my agreement with you regarding open source solutions and performing a fairly systematic search for a good constellation of macro benchmarks inside a harness that can test YOUR application in CONCERT with the constellation and perhaps build a functional performance relationship out of the performance spectrum. That is, it is quite possible to build a "benchmarking tool" that one can run on your own application, suitably encapsulated, and a set of micro and macro benchmarks that can (over time and with the accumulation of some "experience") be transformed into a multivariate map of the covariance of the distinct results with your application, i.e. a predictive map. This can then be used to extrapolate performance even on hardware where the result vector is known but your application performance cannot for some reason be directly measured. Probably quite accurately with as few as 2-3 distinct hardware platforms as model input data... (Just a little hint on a way to proceed to generate a genuinely NEW kind of "benchmarking tool" -- one that is good for something!) rgb
[/soap box]
-- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:[EMAIL PROTECTED] _______________________________________________ Beowulf mailing list, [email protected] To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
