[issue25423] Deprecate benchmarks that execute too quickly

[Brett Cannon]

Brett Cannon added the comment:

I trust you fixed it. :)

--
resolution:  -> fixed
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25423] Deprecate benchmarks that execute too quickly

[STINNER Victor]

STINNER Victor added the comment:

I believe that this issue was indirectly fixed by the issue #26275. The new 
flavor of the benchmark suite now calibrates (computes the number of outter 
loop iterations) each benchmark based on time, no more using a fixed number of 
loop iteartions.

Brett: Can you please confirm that the issue was fixed in the new performance 
benchmark suite?
https://github.com/python/performance


$ python3 -m performance run -b silent_logging,unpack_sequence --fast -o json
Python benchmark suite 0.1.3dev
(...)
Report on Linux smithers 4.5.7-300.fc24.x86_64 #1 SMP Wed Jun 8 18:12:45 UTC 
2016 x86_64 x86_64
Total CPU cores: 8

### logging/no_output ###
Median +- std dev: 809 ns +- 16 ns

### unpack_sequence ###
Median +- std dev: 120 ns +- 4 ns


* silent_logging used 10 inner-loops and 2^14 outter-loops
* unpack_sequence used 400 inner-loops and 4096 outter-loops

You can you this using the "python3 -m perf dump -v json" command on the output 
JSON file (search for "loop").

--
nosy: +haypo

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25423] Deprecate benchmarks that execute too quickly

[Ezio Melotti]

Changes by Ezio Melotti :


--
keywords: +easy
nosy: +ezio.melotti
stage:  -> needs patch
type:  -> behavior

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25423] Deprecate benchmarks that execute too quickly

[Antoine Pitrou]

Antoine Pitrou added the comment:

I haven't seen that:

$ ./perf.py -f -b silent_logging `which python2.7` `which python3.4`
INFO:root:Automatically selected timer: time
Running silent_logging...
INFO:root:Running `/home/antoine/.local/bin/python3.4 
./performance/bm_logging.py -n 5 --timer time no_output`
INFO:root:Running `/usr/bin/python2.7 ./performance/bm_logging.py -n 5 --timer 
time no_output`

Report on Linux fsol 3.19.0-30-generic #34-Ubuntu SMP Fri Oct 2 22:08:41 UTC 
2015 x86_64 x86_64
Total CPU cores: 4

### silent_logging ###
Min: 0.056003 -> 0.075588: 1.35x slower
Avg: 0.056043 -> 0.076493: 1.36x slower
Significant (t=-38.61)
Stddev: 0.3 -> 0.00118: 47.2238x larger


In any case, if it runs too fast, we should simply increase the number of 
iterations.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25423] Deprecate benchmarks that execute too quickly

[Brett Cannon]

New submission from Brett Cannon:

I have seen both silent_logging and unpack_sequence reliably complete so 
quickly in fast mode as to be measured at 0 time. That indicates to me that the 
benchmarks are no longer measuring something of consequence.

--
components: Benchmarks
messages: 253078
nosy: brett.cannon, pitrou
priority: normal
severity: normal
status: open
title: Deprecate benchmarks that execute too quickly

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com