Hi! Andy Wingo <wi...@pobox.com> skribis:
> On Wed 25 Apr 2012 22:39, l...@gnu.org (Ludovic Courtès) writes: > >>> So, those are the problems: benchmarks running for inappropriate, >>> inconsistent durations; >> >> I don’t really see such a problem. It doesn’t matter to me if >> ‘arithmetic.bm’ takes 2mn while ‘vlists.bm’ takes 40s, since I’m not >> comparing them. > > Running a benchmark for 2 minutes is not harmful to the results, but it > is a bit needless. One second is enough. Well, duration has to be chosen such that the jitter is small enough. Sometimes it could be 2mn, sometimes 1s. [...] >>> and benchmarks being optimized out. >> >> That should be fixed. > > In what way? It would make those benchmarks different. > > Thesis: anything for which you would want to turn off the optimizer is > not a good benchmark anyway. Yes, it depends on the benchmarks. For instance, I once added benchmarks for ‘1+’ and ‘1-’, because I wanted to see the impact of an optimization to the corresponding VM instructions. Nowadays peval would optimize those benchmarks out. Yet, the fact is that I was interested in the performance of the underlying VM instructions, regardless of what the compiler might be doing. Thanks, Ludo’.