STINNER Victor added the comment:
Maciej Fijalkowski also sent me the following article a few months ago, it also
explains indirectly why using the minimum for benchmarks is not reliable:
"Virtual Machine Warmup Blows Hot and Cold"
Even if the article is more focused on JIT compilers, it shows that benchmarks
are not straightforward but always full of bad surprises.
A benchmark doesn't have a single value but a *distribution*. The best question
is how to summarize the full distribution without loosing too much information.
In the perf module I decided to not take a decision: a JSON file stores *all*
data :-D But by default, perf displays mean +- std dev.
Python tracker <rep...@bugs.python.org>
Python-bugs-list mailing list