On Wed, 15 Mar 2017 00:42:07 +1000 Nick Coghlan <ncogh...@gmail.com> wrote: > > That would suggest that the implicit assumption of a measure-of-centrality > with a measure-of-symmetric-deviation may need to be challenged, as at > least some meaningful performance problems are going to show up as > non-normal distributions in the benchmark results.
Well, the real issue here is that an important contributor to non-normality is not the benchmark itself, but measurement noise due to various issues (such as system noise, which has of course a highly skewed distribution). Victor is trying to eliminate the effects of system noise by using the median, but if that's the primary goal, using the minimum is arguably better, since the system noise is always a positive contributor (i.e. it can only increase the runtimes). The median is arguably a bastardized solution, which satisfies neither the requirement of eliminating system noise, nor the requirement of faithfully representing performance variations due to non-deterministic effects in the Python runtime and/or benchmark itself. Regards Antoine. _______________________________________________ Speed mailing list Speed@python.org https://mail.python.org/mailman/listinfo/speed
