[issue26284] Fix telco benchmark

[STINNER Victor]

STINNER Victor added the comment:

Sorry, I'm not sure if you want to make any change to bm_telco.py of the 
performance. If yes, please open an issue at 
https://github.com/python/performance

> So LGTM.

Hum, it looks like you are happy, so I now close this issue :-) I'm slowly 
trying to move discussions related to benchmarking to the performance module on 
GitHub.

--
resolution:  -> fixed
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26284] Fix telco benchmark

[Stefan Krah]

Stefan Krah added the comment:

Wow, on my machine this is very stable, great.

The output should be like

   http://www.bytereef.org/software/mpdecimal/benchmarks/telco.py ,

but printing one number only should be okay. The important thing is that some 
decimal is printed at all to test the formatting speed.


So LGTM.

--
assignee: skrah -> haypo

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26284] Fix telco benchmark

[STINNER Victor]

STINNER Victor added the comment:

@Stefan: Can you please check bm_telco.py and maybe propose a pull request if 
something is wrong?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26284] Fix telco benchmark

[STINNER Victor]

STINNER Victor added the comment:

I worked on a new version of the benchmark suite, it is now called performance 
and moved to GitHub:
https://github.com/python/performance

In performance 0.1.2, bm_telco.py uses BytesIO for input and six.StringIO for 
output. The output is just one number per line, it's different than  
http://bugs.python.org/file41802/telco_haypo.py output.

I don't know which benchmark is "right".

I believe that bm_telco.py of performance is now much more stable thanks to the 
perf module which runs the benchmark in multiple processes (20 by default). It 
helps to get a better distribution of samples.

Example:

$ python3 -m performance run -b telco -o telco.json
(...)

$ python3 -m perf show --hist --stats --metadata telco.json 
Metadata:
- aslr: Full randomization
- cpu_config: 0-7=driver:intel_pstate, intel_pstate:no turbo, governor:powersave
- cpu_count: 8
- cpu_model_name: Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz
- description: Telco decimal benchmark
- hostname: smithers
- loops: 8
- name: telco
- perf_version: 0.7.4
- platform: Linux-4.5.7-300.fc24.x86_64-x86_64-with-fedora-24-Twenty_Four
- python_executable: 
/home/haypo/prog/python/performance/venv/cpython3.5-da07a9f70715/bin/python
- python_implementation: cpython
- python_version: 3.5.1 (64bit)
- timer: clock_gettime(CLOCK_MONOTONIC), resolution: 1.00 ns

22.7 ms: 18 #
23.0 ms: 27 
###
23.4 ms:  7 
23.7 ms:  1 ###
24.1 ms:  2 ##
24.4 ms:  1 ###
24.7 ms:  0 |
25.1 ms:  1 ###
25.4 ms:  0 |
25.8 ms:  0 |
26.1 ms:  0 |
26.5 ms:  0 |
26.8 ms:  0 |
27.1 ms:  0 |
27.5 ms:  0 |
27.8 ms:  0 |
28.2 ms:  0 |
28.5 ms:  0 |
28.9 ms:  1 ###
29.2 ms:  1 ###
29.5 ms:  1 ###

Total duration: 15.1 sec
Start date: 2016-08-30T18:09:49
End date: 2016-08-30T18:10:07
Raw sample minimum: 182 ms
Raw sample maximum: 237 ms

Number of runs: 20
Total number of samples: 60
Number of samples per run: 3
Number of warmups per run: 1
Loop iterations per sample: 8

Minimum: 22.8 ms (-2%)
Median +- std dev: 23.2 ms +- 1.4 ms
Mean +- std dev: 23.6 ms +- 1.4 ms
Maximum: 29.7 ms (+28%)

Median +- std dev: 23.2 ms +- 1.4 ms


The histogram helps to see that there is no such "minimum", but more a gaussian 
curve. The perf module uses the median of samples rather than the minimum, it 
also displays and computes the standard deviation by default.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26284] Fix telco benchmark

[STINNER Victor]

STINNER Victor added the comment:

> Unfortunately, replacing io.BytesIO(data) with indexing does not make the 
> benchmark faster or more stable on my machine.

Ah, I didn't check. I expected BytesIO.read() to be slower than bytes string 
slicing.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26284] Fix telco benchmark

[Brett Cannon]

Brett Cannon added the comment:

If you think the string conversion should go back in, Stefan, feel free to put 
it back (unless Victor wants to say why he took it out).

--
assignee:  -> skrah

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26284] FIx telco benchmark

[Stefan Krah]

Stefan Krah added the comment:

Unfortunately, replacing io.BytesIO(data) with indexing does not make the 
benchmark faster or more stable on my machine.

BTW, string conversion of the result is actually a crucial part of
the benchmark, it was taken out in 
http://bugs.python.org/file41802/telco_haypo.py.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26284] Fix telco benchmark

[Stefan Krah]

Changes by Stefan Krah :


--
title: FIx telco benchmark -> Fix telco benchmark

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26284] FIx telco benchmark

[Stefan Krah]

New submission from Stefan Krah:

The telco benchmark is unstable. It needs some of Victor's changes from #26275 
and probably a larger data set:

http://speleotrove.com/decimal/expon180-1e6b.zip is too big for
_pydecimal, but the one that is used is probably too small for
_decimal.

--
components: Benchmarks
messages: 259569
nosy: brett.cannon, haypo, pitrou, skrah, yselivanov
priority: normal
severity: normal
stage: needs patch
status: open
title: FIx telco benchmark
type: behavior
versions: Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com