You may also consider that a benchmark that varies greatly between runs may be a flawed benchmark.
I think it should be considered, but only on the running side, and act accordingly (too high a deviation: discard run, reconsider benchmark, reconsider environment or whatever). Miquel 2010/2/26 Stefan Behnel <[email protected]>: > Carl Friedrich Bolz, 25.02.2010 18:38: >> On 02/25/2010 04:10 PM, Miquel Torres wrote: >>> As some of you already know, there is a new performance site online: >>> speed.pypy.org. >> [...] >> >> I'm quite impressed, this is very cool work and a good improvement over >> the current plots. Thanks for doing this! >> >> One thing that I would really find useful are error bars in the timeline >> view. This would help us judge whether up-and-down movements are within >> the margin of randomness, or whether it is a real effect. I don't know >> how annoying they are to implement though, no clue whether the plot lib >> supports that. There should be enough information about errors in the >> json files, as far as I remember. > > The standard deviation is commonly considered meaningless for benchmark > results. All that really matters is the fastest run, everything else is > just fog. > > Read the docs on timeit, for example. > > Stefan > > _______________________________________________ > [email protected] > http://codespeak.net/mailman/listinfo/pypy-dev > _______________________________________________ [email protected] http://codespeak.net/mailman/listinfo/pypy-dev
