On 14 March 2017 at 17:14, Serhiy Storchaka <storch...@gmail.com> wrote:
> On 13.03.17 22:38, Antoine Pitrou wrote: > >> Additionally, while mean and std dev are generally quite well >> understood, the properties of the median absolute deviation are >> generally little known. >> > > Std dev is well understood for the distribution close to normal. But when > the distribution is too skewed or multimodal (as in your quick example) > common assumptions (that 2/3 of samples are in the range of the std dev, > 95% of samples are in the range of two std devs, 99% of samples are in the > range of three std devs) are no longer valid. That would suggest that the implicit assumption of a measure-of-centrality with a measure-of-symmetric-deviation may need to be challenged, as at least some meaningful performance problems are going to show up as non-normal distributions in the benchmark results. Network services typically get around the "inherent variance" problem by looking at a few key percentiles like 50%, 90% and 95%. Perhaps that would be appropriate here as well? Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia
_______________________________________________ Speed mailing list Speed@python.org https://mail.python.org/mailman/listinfo/speed
