On Tue, 5 Jul 2016 22:04:51 +1000 Nick Coghlan <ncogh...@gmail.com> wrote: > > The median + stddev approach helps convey a "typical" result better > than the minimum or mean do, while also providing an indication when > the variation in results is too high for the median to really be > meaningful.
This is missing the primary goal, which is to compare results between implementations. For this you need a single number, and the median is a poor indication of overall performance (because it totally ignores the actual distribution shape). Providing detailed statistical information (median, mean, deviation, quartiles, etc.) about each benchmark run is useful in itself, but a secondary concern for most uses of the benchmark suite. Regards Antoine. _______________________________________________ Speed mailing list Speed@python.org https://mail.python.org/mailman/listinfo/speed
