[issue45261] Unreliable (?) results from timeit (cache issue?)

Change by STINNER Victor : -- resolution: -> not a bug ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https

[issue45261] Unreliable (?) results from timeit (cache issue?)

Change by Tim Holy : -- resolution: not a bug -> ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mai

[issue45261] Unreliable (?) results from timeit (cache issue?)

Raymond Hettinger added the comment: Given that there isn't an actual bug here, please move this discussion to python-ideas. -- nosy: +rhettinger resolution: -> not a bug stage: -> resolved status: open -> closed ___ Python tracker

[issue45261] Unreliable (?) results from timeit (cache issue?)

Tim Holy added the comment: > And I hope that Tim Holy is interested too :-) Sure, I'll bite :-). On the topic of which statistic to show, I am a real fan of the histogram. As has been pointed out, timing in the real world can be pretty complicated, and I don't think it does anyone good to h

[issue45261] Unreliable (?) results from timeit (cache issue?)

Marc-Andre Lemburg added the comment: On the topic average vs. minimum, it's interesting to see this pop up every now and then. When I originally wrote pybench late in 1997, I used average, since it gave good results on my PC at the time. Later on, before pybench was added to Tools/ in Python

[issue45261] Unreliable (?) results from timeit (cache issue?)

Tim Holy added the comment: To make sure it's clear, it's not 0.08ns/function call, it's a loop and it comes out to 0.08ns/element. The purpose of quoting that number was to compare to the CPU clock interval, which on my machine is ~0.4ns. Any operation that's less than 1 clock cycle is susp

[issue45261] Unreliable (?) results from timeit (cache issue?)

STINNER Victor added the comment: I suggest to close this issue. timeit works as expected. It has limitations, but that's ok. Previous attempts to enhance timeit were rejected. -- ___ Python tracker ___

[issue45261] Unreliable (?) results from timeit (cache issue?)

STINNER Victor added the comment: > but you give no reason for why you should not use the minimum. See https://pyperf.readthedocs.io/en/latest/analyze.html#minimum-vs-average I'm not really interested to convince you. Use the minimum if you believe that it better fits your needs. But pyperf

[issue45261] Unreliable (?) results from timeit (cache issue?)

Steven D'Aprano added the comment: Thank you Victor, I don't know whether I can convince you or you will convince me, but this is very interesting one way or the other. And I hope that Tim Holy is interested too :-) On Wed, Sep 22, 2021 at 01:51:35PM +, STINNER Victor wrote: > > But tim

[issue45261] Unreliable (?) results from timeit (cache issue?)

STINNER Victor added the comment: > But timing results are not like that, the measurement errors are one-sided, not two: (..) I suggest you to run tons of benchmarks and look at the distribution. The reality is more complex than what you may think. > measurement = true value + random error

[issue45261] Unreliable (?) results from timeit (cache issue?)

Change by Dong-hee Na : -- nosy: +corona10 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.pytho

[issue45261] Unreliable (?) results from timeit (cache issue?)

Steven D'Aprano added the comment: Thanks Victor for the explanation about pyperf's addition features. They do sound very useful. Perhaps we should consider adding some of them to timeit? However, in my opinion using the average is statistically wrong. Using the mean is good when errors are

[issue45261] Unreliable (?) results from timeit (cache issue?)

STINNER Victor added the comment: PyPy emits a warning when the timeit module is used, suggesting to use pyperf. timeit uses the minimum, whereas pyperf uses the average (arithmetic mean). timeit uses a single process, pyperf spawns 21 processes: 1 just for the loop calibration, 20 to comput

[issue45261] Unreliable (?) results from timeit (cache issue?)

Steven D'Aprano added the comment: Running timeit with number=1 for fast running processes is not likely to be reliable. The problem is that modern computers are *incredibly noisy*, there are typically hundreds of processes running on them at any one time. Trying to get deterministic times f

[issue45261] Unreliable (?) results from timeit (cache issue?)

New submission from Tim Holy : This is a lightly-edited reposting of https://stackoverflow.com/questions/69164027/unreliable-results-from-timeit-cache-issue I am interested in timing certain operations in numpy and skimage, but I'm occasionally seeing surprising transitions (not entirely repr