Hi: On 05 Apr 2011, at 21:27, Nicolas Cellier wrote:
> But you probably have more ellaborated and well thought method :) I think we are talking here mostly about quick and dirty benchmarks to confirm intuitions. The provable best and most cost effective way to do it is: flip a coin (and that is serious, has the same or even better reliability and quality of result) If you think that is not good enough, well then the absolute minimum is something like this: - run your benchmark 5 times - 5 times with your optimization - 5 times without - look at the collected raw values i.e. measure runtime - and use your gut feeling to decide whether you have meaningful values and whether your time to do the optimization was well spent (don't eliminate outliers) If that is still to much to do, well, again, just flip a coin, and you have a high chance that the result of that are actually more reliable. (the approach above does not account for systematic errors...) However, once you got your optimization into the VM your favorite build server should do its thing and that should include performance regression testing which is using a "slightly" more sophisticated approach than the described one. Best regards Stefan -- Stefan Marr Software Languages Lab Vrije Universiteit Brussel Pleinlaan 2 / B-1050 Brussels / Belgium http://soft.vub.ac.be/~smarr Phone: +32 2 629 2974 Fax: +32 2 629 3525
