On Sunday 06 December 2009 at 08:06, Geoffrey Broadwell wrote: > While I would love for HEAD to be always the fastest, if you mentally > graph this you can see that trunk is in the main grouping of total times > in the 32 to 40 second range. It's the outliers in the 55 to 74 second > range that really aren't looking good. > > To me there are two optimization goals: > > 1. Make sure HEAD is in the same grouping as the fastest time, for all > benchmarks we can find. (Don't ask me for an exact rule for grouping -- > but if you need a programmatic rule of thumb for a meta-bench tool, > perhaps cut off at 30% or even 50% worse than the best time.)
I don't trust time-related benchmarkings for this reason. Yes, stochastic analysis can give you some degree of probability that comparisons are all like-to-like, but the output of a single Callgrind run is much easier to compare (and gives a better explanation of *why* and *how* and *where* performance has changed). -- c _______________________________________________ http://lists.parrot.org/mailman/listinfo/parrot-dev
