[issue22881] show median in benchmark results

[Stefan Behnel]

Changes by Stefan Behnel :


--
resolution:  -> fixed
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22881] show median in benchmark results

[STINNER Victor]

STINNER Victor added the comment:

The new https://github.com/python/performance benchmark suite now displays the 
median rather than the arithmetic mean (average) by default (it also displays 
the standard deviation).

See the perf issue for a discussion about median vs mean:
https://github.com/haypo/perf/issues/1

Can we now close this issue?

--
nosy: +haypo

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22881] show median in benchmark results

[Stefan Behnel]

Stefan Behnel added the comment:

I'm actually not sure how it relates to the minimum. The more runs you have, 
the higher the chance of hitting the actual minimum at least once. And if none 
of the runs hits the real minimum, you're simply out of luck.

However, it should tend to give a much better result than the (currently 
printed) average, which suffers from outliers. And outliers are almost always 
too high for benchmarks and never too low, due to various external influences.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22881
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22881] show median in benchmark results

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

May be just drop 5% of largest values to avoid the impact of outliers?

See also issue23552.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22881
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22881] show median in benchmark results

[Antoine Pitrou]

Antoine Pitrou added the comment:

 May be just drop 5% of largest values to avoid the impact of outliers?

In fast mode (option -f), there may not be enough samples for that.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22881
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22881] show median in benchmark results

[Stefan Behnel]

Stefan Behnel added the comment:

Well, we can apply a kludge, or apply statistics.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22881
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22881] show median in benchmark results

[Antoine Pitrou]

Antoine Pitrou added the comment:

Then let's just replace the average with the median? I don't think it makes 
sense to add more statistical information to the output (IMHO, there is already 
too much of it :-)).

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22881
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22881] show median in benchmark results

[Antoine Pitrou]

Antoine Pitrou added the comment:

Have you found the median to be more stable than the minimum here?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22881
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22881] show median in benchmark results

[Stefan Behnel]

Stefan Behnel added the comment:

 for the even number case, I think you shouldn't do // 2, but / 2.

Right. I updated the patch.

--
Added file: http://bugs.python.org/file39279/show_median.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22881
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22881] show median in benchmark results

[Stefan Behnel]

Stefan Behnel added the comment:

Any more comments on the patch, or can it be applied?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22881
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22881] show median in benchmark results

[Wolfgang Maier]

Wolfgang Maier added the comment:

for the even number case, I think you shouldn't do // 2, but / 2.

In general, wouldn't it be good to let the statistics module do all the stats 
calculations?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22881
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22881] show median in benchmark results

[Wolfgang Maier]

Wolfgang Maier added the comment:

ah sorry, it's late here already and I forgot what file this change is about. 
So forget my last comment then.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22881
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22881] show median in benchmark results

[Wolfgang Maier]

Changes by Wolfgang Maier wolfgang.ma...@biologie.uni-freiburg.de:


--
nosy: +wolma

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22881
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22881] show median in benchmark results

[Stefan Behnel]

Stefan Behnel added the comment:

 In general, wouldn't it be good to let the statistics module do all the stats 
 calculations?

It's not available in older Python versions, e.g. 2.6.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22881
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22881] show median in benchmark results

[Wolfgang Maier]

Wolfgang Maier added the comment:

 It's not available in older Python versions, e.g. 2.6.

I know, I was talking about 3.5+, of course. This would not be backported to 
Python2 anyway, would it?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22881
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22881] show median in benchmark results

[Stefan Behnel]

New submission from Stefan Behnel:

The median tends to give a better idea about benchmark results than an average 
as it inherently ignores outliers.

--
components: Benchmarks
files: show_median.patch
keywords: patch
messages: 231239
nosy: pitrou, scoder
priority: normal
severity: normal
status: open
title: show median in benchmark results
type: enhancement
Added file: http://bugs.python.org/file37206/show_median.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22881
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22881] show median in benchmark results

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

In case of even number of samples the median value is calculated as arithmetic 
mean of two middle samples.

med_base = (base_times[len(base_times)//2] + 
base_times[(len(base_times)+1)//2]) / 2

--
nosy: +serhiy.storchaka
stage:  - patch review

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22881
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22881] show median in benchmark results

[Serhiy Storchaka]

Changes by Serhiy Storchaka storch...@gmail.com:


--
Removed message: http://bugs.python.org/msg231240

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22881
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22881] show median in benchmark results

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

In case of even number of samples the median value is calculated as arithmetic 
mean of two middle samples.

med_base = (base_times[len(base_times)//2] + 
base_times[(len(base_times)-1)//2]) / 2

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22881
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22881] show median in benchmark results

[Stefan Behnel]

Stefan Behnel added the comment:

Fair enough, patch updated.

--
Added file: http://bugs.python.org/file37207/show_median.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22881
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com