Marc-Andre Lemburg <m...@egenix.com> added the comment:
On the topic average vs. minimum, it's interesting to see this pop up every now and then. When I originally wrote pybench late in 1997, I used average, since it gave good results on my PC at the time. Later on, before pybench was added to Tools/ in Python 2.5 in 2006, people started complaining about sometimes getting weird results (e.g. negative times due to the calibration runs not being stable enough). A lot of noise was made from the minimum fans, so I switched to minimum, which then made the results more stable, but I left in the average figures as well. Then some years later, people complained about pybench not being good enough for comparing to e.g. PyPy, since those other implementations optimized away some of the micro-benchmarks which were used in pybench. It was then eventually removed, to not confuse people not willing to try to understand what the benchmark suite was meant for, nor understand the issues around running such benchmarks on more modern CPUs. CPUs have advanced a lot since the days pybench was written and so reliable timings are not easy to get unless you invest in dedicated hardware, custom OS and CPU settings and lots of time to calibrate everything. See Victor's research for more details. What we have here is essentially the same issue. timeit() is mostly being used for micro-benchmarks, but those need to be run in dedicated environments. timeit() is good for quick checks, but not really up to the task of providing reliable timing results. One of these days, we should ask the PSF or one of its sponsors to provide funding and devtime to set up such a reliable testing environment. One which runs not only high end machines, but also average and lower end machines, and using different OSes, so that we can detect performance regressions early and easily on different platforms. ---------- nosy: +lemburg _______________________________________ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue45261> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com