On 5 July 2016 at 20:08, Antoine Pitrou <solip...@pitrou.net> wrote: > On Tue, 5 Jul 2016 11:35:30 +0200 > Victor Stinner <victor.stin...@gmail.com> > wrote: >> In practice, it almost never occurs to have all samples with the same >> value. There is always a statistic distribution, usually as a gaussian >> curse. > > If it's a gaussian curve (not a curse, probably :-)), then you can > summarize it with two values: the mean and the stddev. But it's > probably not a gaussian, because of system noise and other factors, so > your assumption is wrong :-)
If you haven't already, I highly recommend reading the discussion in https://github.com/haypo/perf/issues/1 that led to Victor adopting the current median + stddev approach As Mahmoud noted there, in terms of really understanding the benchmark results, there's no substitute for actually looking at the histograms with the result distributions. The numeric results are never going to be able to do more than provide a "flavour" for those results, since the distributions aren't Guassian, but trying to characterise and describe them properly would inevitably confuse folks that aren't already expert statisticians. The median + stddev approach helps convey a "typical" result better than the minimum or mean do, while also providing an indication when the variation in results is too high for the median to really be meaningful. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia _______________________________________________ Speed mailing list Speed@python.org https://mail.python.org/mailman/listinfo/speed
