Franck Pommereau wrote: >> I need some input for the benchmarking infrastructure. I'm nearly >> at the point where I need to have some place to run it before >> continuing (i.e. I need to try and use it, not just speculate). > > This is not an answer to your question but might interest you. > > We it comes to benchmarking, the following paper is certainly worth > reading: http://www-plan.cs.colorado.edu/diwan/asplos09.pdf > It explains how many benchmarks are biased and result in faulty measures > (in important proportions), and how this can be compensated.
Seconded. This is an excellent paper. Btw, issues like this might be a reason to really reuse parts of the unladen-swallow benchmark runner, since they seem to have put work into doing the right thing from a statistics point of view. Cheers, Carl Friedrich _______________________________________________ [email protected] http://codespeak.net/mailman/listinfo/pypy-dev
