STINNER Victor added the comment:

Maciej Fijalkowski also sent me the following article a few months ago, it also 
explains indirectly why using the minimum for benchmarks is not reliable:

"Virtual Machine Warmup Blows Hot and Cold"

Even if the article is more focused on JIT compilers, it shows that benchmarks 
are not straightforward but always full of bad surprises.

A benchmark doesn't have a single value but a *distribution*. The best question 
is how to summarize the full distribution without loosing too much information.

In the perf module I decided to not take a decision: a JSON file stores *all* 
data :-D But by default, perf displays mean +- std dev.


Python tracker <>
Python-bugs-list mailing list

Reply via email to