[Impala-ASF-CR] IMPALA-7426: Use Mann-Whitney U to compare benchmarks
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11194 ) Change subject: IMPALA-7426: Use Mann-Whitney U to compare benchmarks .. Patch Set 5: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/11194 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8d6631ebeba1422b832def5cd68537624f672fa0 Gerrit-Change-Number: 11194 Gerrit-PatchSet: 5 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Wed, 12 Sep 2018 21:10:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7426: Use Mann-Whitney U to compare benchmarks
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11194 ) Change subject: IMPALA-7426: Use Mann-Whitney U to compare benchmarks .. IMPALA-7426: Use Mann-Whitney U to compare benchmarks The Mann-Whitney test can be used to compare samples taken from non-normal distributions, and so can more accurately reflect performance changes than a T-test. This patch does not remove t-tests from the benchmark reporting, it just supplements them by including the Mann-Whitney test result as well. Change-Id: I8d6631ebeba1422b832def5cd68537624f672fa0 Reviewed-on: http://gerrit.cloudera.org:8080/11194 Reviewed-by: Jim Apple Tested-by: Impala Public Jenkins --- M tests/benchmark/report_benchmark_results.py M tests/util/calculation_util.py 2 files changed, 85 insertions(+), 37 deletions(-) Approvals: Jim Apple: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/11194 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I8d6631ebeba1422b832def5cd68537624f672fa0 Gerrit-Change-Number: 11194 Gerrit-PatchSet: 6 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Philip Zeyliger
[Impala-ASF-CR] IMPALA-7426: Use Mann-Whitney U to compare benchmarks
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11194 ) Change subject: IMPALA-7426: Use Mann-Whitney U to compare benchmarks .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3149/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/11194 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8d6631ebeba1422b832def5cd68537624f672fa0 Gerrit-Change-Number: 11194 Gerrit-PatchSet: 5 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Wed, 12 Sep 2018 17:47:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7426: Use Mann-Whitney U to compare benchmarks
Jim Apple has posted comments on this change. ( http://gerrit.cloudera.org:8080/11194 ) Change subject: IMPALA-7426: Use Mann-Whitney U to compare benchmarks .. Patch Set 5: Code-Review+2 Carry Philip's +2 on rebase -- To view, visit http://gerrit.cloudera.org:8080/11194 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8d6631ebeba1422b832def5cd68537624f672fa0 Gerrit-Change-Number: 11194 Gerrit-PatchSet: 5 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Wed, 12 Sep 2018 17:46:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7426: Use Mann-Whitney U to compare benchmarks
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/11194 ) Change subject: IMPALA-7426: Use Mann-Whitney U to compare benchmarks .. Patch Set 4: Code-Review+2 Thanks! -- To view, visit http://gerrit.cloudera.org:8080/11194 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8d6631ebeba1422b832def5cd68537624f672fa0 Gerrit-Change-Number: 11194 Gerrit-PatchSet: 4 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Mon, 10 Sep 2018 21:57:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7426: Use Mann-Whitney U to compare benchmarks
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11194 ) Change subject: IMPALA-7426: Use Mann-Whitney U to compare benchmarks .. Patch Set 3: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/562/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/11194 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8d6631ebeba1422b832def5cd68537624f672fa0 Gerrit-Change-Number: 11194 Gerrit-PatchSet: 3 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Sat, 01 Sep 2018 02:51:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7426: Use Mann-Whitney U to compare benchmarks
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11194 ) Change subject: IMPALA-7426: Use Mann-Whitney U to compare benchmarks .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/11194/3/tests/benchmark/report_benchmark_results.py File tests/benchmark/report_benchmark_results.py: http://gerrit.cloudera.org:8080/#/c/11194/3/tests/benchmark/report_benchmark_results.py@573 PS3, Line 573: a flake8: W503 line break before binary operator -- To view, visit http://gerrit.cloudera.org:8080/11194 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8d6631ebeba1422b832def5cd68537624f672fa0 Gerrit-Change-Number: 11194 Gerrit-PatchSet: 3 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Sat, 01 Sep 2018 02:33:09 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7426: Use Mann-Whitney U to compare benchmarks
Hello Philip Zeyliger, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11194 to look at the new patch set (#3). Change subject: IMPALA-7426: Use Mann-Whitney U to compare benchmarks .. IMPALA-7426: Use Mann-Whitney U to compare benchmarks The Mann-Whitney test can be used to compare samples taken from non-normal distributions, and so can more accurately reflect performance changes than a T-test. This patch does not remove t-tests from the benchmark reporting, it just supplements them by including the Mann-Whitney test result as well. Change-Id: I8d6631ebeba1422b832def5cd68537624f672fa0 --- M tests/benchmark/report_benchmark_results.py M tests/util/calculation_util.py 2 files changed, 85 insertions(+), 37 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/94/11194/3 -- To view, visit http://gerrit.cloudera.org:8080/11194 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8d6631ebeba1422b832def5cd68537624f672fa0 Gerrit-Change-Number: 11194 Gerrit-PatchSet: 3 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Philip Zeyliger
[Impala-ASF-CR] IMPALA-7426: Use Mann-Whitney U to compare benchmarks
Philip Zeyliger has posted comments on this change. (
http://gerrit.cloudera.org:8080/11194 )
Change subject: IMPALA-7426: Use Mann-Whitney U to compare benchmarks
..
Patch Set 2:
(7 comments)
http://gerrit.cloudera.org:8080/#/c/11194/2/tests/benchmark/report_benchmark_results.py
File tests/benchmark/report_benchmark_results.py:
http://gerrit.cloudera.org:8080/#/c/11194/2/tests/benchmark/report_benchmark_results.py@a568
PS2, Line 568:
is removing num_clients intentional?
http://gerrit.cloudera.org:8080/#/c/11194/2/tests/benchmark/report_benchmark_results.py@129
PS2, Line 129: dest="max_percent_change_threshold",
default=float("inf"),
Is the default change intentional?
http://gerrit.cloudera.org:8080/#/c/11194/2/tests/benchmark/report_benchmark_results.py@394
PS2, Line 394: if not options.hive_results:
: try:
: save_runtime_diffs(results, ref_results,
self.perf_change, self.zval, self.tval)
: except Exception as e:
: LOG.error('Could not generate an html diff:
{0}'.format(e))
This code used to never run because it was indented incorrectly? It looks ok to
me, but I'm not clear how it ever ran before?
http://gerrit.cloudera.org:8080/#/c/11194/2/tests/benchmark/report_benchmark_results.py@563
PS2, Line 563: 'Base StdDev(%)', 'Iters',
'Median Diff(%)', 'Zval',
Should this be "MW Z-Val" or something to indicate what kind of zvalue this is?
http://gerrit.cloudera.org:8080/#/c/11194/2/tests/benchmark/report_benchmark_results.py@1037
PS2, Line 1037: prefix = ('reg' if zval >= 0 and tval >=
This is different than __check_perf_change_significance's implementation. Does
that make sense?
http://gerrit.cloudera.org:8080/#/c/11194/2/tests/benchmark/report_benchmark_results.py@1037
PS2, Line 1037: prefix = ('reg' if zval >= 0 and tval >=
: 0 else 'imp' if zval <= 0 and tval <= 0 else
'???')
I think this would be more legible as multiple lines. (I have a deep hatred of
python's non-ternary, but when it's nested, I can't make heads or tails of it.)
http://gerrit.cloudera.org:8080/#/c/11194/2/tests/util/calculation_util.py
File tests/util/calculation_util.py:
http://gerrit.cloudera.org:8080/#/c/11194/2/tests/util/calculation_util.py@74
PS2, Line 74: Calculates the Mann-Whitney U Test Z value for the given
samples and reference.
Have you compared this to
https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.mannwhitneyu.html?
Do we have a significant number of sample (at least 20 per that page) to use
this?
After installing python-scipy, I ran the following. Of note, the U calculation
agrees (good). The scipy thing gives a p-value and we are returning a z-value
and then taking its absolute value and looking at a threshold. Are we doing the
right thing?
You might want to update the pydoc here to explain what to do with one of these
values.
$cat test.py
import math
def calculate_mwu(samples, ref_samples):
"""
Calculates the Mann-Whitney U Test Z value for the given samples and
reference.
"""
tag_a = [(s, 'A') for s in samples]
tab_b = [(s, 'B') for s in ref_samples]
ab = tag_a + tab_b
ab.sort()
# Assume no ties
u = 0
count_b = 0
for v in ab:
if v[1] == 'A':
u += count_b
else:
count_b += 1
# u is normally distributed with the following mean and standard deviation:
mean = len(samples) * len(ref_samples) / 2.0
stddev = math.sqrt(len(samples) * len(ref_samples) * (1 + len(ab)) / 12.0)
print u, mean, stddev
return (u - mean) / stddev
from scipy.stats import mannwhitneyu
a = range(30)
b = [ z + 0.5 for z in a ]
print mannwhitneyu(a, b)
print mannwhitneyu(a, b, use_continuity=False)
print mannwhitneyu(a, b, alternative="two-sided")
print mannwhitneyu(a, b, alternative="less")
print mannwhitneyu(a, b, alternative="less",use_continuity=False)
print mannwhitneyu(a, b, alternative="two-sided",use_continuity=False)
print calculate_mwu(a, b)
print 1 - abs(calculate_mwu(a, b))
09:28:24 pannier cmf [master !x?*] ~/src/cmf
$python test.py
MannwhitneyuResult(statistic=435.0, pvalue=0.8302552839111963)
MannwhitneyuResult(statistic=435.0, pvalue=0.82449575165477107)
MannwhitneyuResult(statistic=435.0, pvalue=0.8302552839111963)
MannwhitneyuResult(statistic=435.0, pvalue=0.41512764195559815)
MannwhitneyuResult(statistic=435.0, pvalue=0.41224787582738553)
MannwhitneyuResult(statistic=435.0, pvalue=0.82449575165477107)
435 450.0 67.6387462923
-0.221766381286
435 450.0 67.6387462923
0.778233618714
--
To view, visit http://gerrit.cloudera.org:8080/11194
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8d6631ebeba1422b832def5cd68537624f672fa0
Gerrit-Change-Number: 11194
[Impala-ASF-CR] IMPALA-7426: Use Mann-Whitney U to compare benchmarks
Jim Apple has posted comments on this change. ( http://gerrit.cloudera.org:8080/11194 ) Change subject: IMPALA-7426: Use Mann-Whitney U to compare benchmarks .. Patch Set 2: (9 comments) http://gerrit.cloudera.org:8080/#/c/11194/1/tests/benchmark/report_benchmark_results.py File tests/benchmark/report_benchmark_results.py: http://gerrit.cloudera.org:8080/#/c/11194/1/tests/benchmark/report_benchmark_results.py@121 PS1, Line 121: > flake8: E502 the backslash is redundant between brackets Done http://gerrit.cloudera.org:8080/#/c/11194/1/tests/benchmark/report_benchmark_results.py@380 PS1, Line 380: > flake8: E703 statement ends with a semicolon Done http://gerrit.cloudera.org:8080/#/c/11194/1/tests/benchmark/report_benchmark_results.py@396 PS1, Line 396: , > flake8: F821 undefined name 'ref_results' This whole section was, even in HEAD, indented 4 spaces, which puts it at the class level, not the method level. I indented in 6 spaces now, but something was already fishy here. http://gerrit.cloudera.org:8080/#/c/11194/1/tests/benchmark/report_benchmark_results.py@573 PS1, Line 573: a > flake8: W503 line break before binary operator This is now flipped? https://github.com/python/peps/commit/c59c4376ad233a62ca4b3a6060c81368bd21e85b I'll send another patch. http://gerrit.cloudera.org:8080/#/c/11194/1/tests/benchmark/report_benchmark_results.py@1016 PS1, Line 1016: > flake8: E302 expected 2 blank lines, found 1 Done http://gerrit.cloudera.org:8080/#/c/11194/1/tests/benchmark/report_benchmark_results.py@1036 PS1, Line 1036: > flake8: E501 line too long (94 > 90 characters) Done http://gerrit.cloudera.org:8080/#/c/11194/1/tests/util/calculation_util.py File tests/util/calculation_util.py: http://gerrit.cloudera.org:8080/#/c/11194/1/tests/util/calculation_util.py@71 PS1, Line 71: > flake8: E302 expected 2 blank lines, found 1 Done http://gerrit.cloudera.org:8080/#/c/11194/1/tests/util/calculation_util.py@75 PS1, Line 75: > flake8: E231 missing whitespace after ',' Done http://gerrit.cloudera.org:8080/#/c/11194/1/tests/util/calculation_util.py@76 PS1, Line 76: , > flake8: E231 missing whitespace after ',' Done -- To view, visit http://gerrit.cloudera.org:8080/11194 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8d6631ebeba1422b832def5cd68537624f672fa0 Gerrit-Change-Number: 11194 Gerrit-PatchSet: 2 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Comment-Date: Sun, 12 Aug 2018 22:08:07 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7426: Use Mann-Whitney U to compare benchmarks
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11194 ) Change subject: IMPALA-7426: Use Mann-Whitney U to compare benchmarks .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/300/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11194 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8d6631ebeba1422b832def5cd68537624f672fa0 Gerrit-Change-Number: 11194 Gerrit-PatchSet: 2 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Sun, 12 Aug 2018 18:11:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7426: Use Mann-Whitney U to compare benchmarks
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11194 ) Change subject: IMPALA-7426: Use Mann-Whitney U to compare benchmarks .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/11194/2/tests/benchmark/report_benchmark_results.py File tests/benchmark/report_benchmark_results.py: http://gerrit.cloudera.org:8080/#/c/11194/2/tests/benchmark/report_benchmark_results.py@573 PS2, Line 573: a flake8: W503 line break before binary operator -- To view, visit http://gerrit.cloudera.org:8080/11194 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8d6631ebeba1422b832def5cd68537624f672fa0 Gerrit-Change-Number: 11194 Gerrit-PatchSet: 2 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Sun, 12 Aug 2018 17:39:53 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7426: Use Mann-Whitney U to compare benchmarks
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11194 to look at the new patch set (#2). Change subject: IMPALA-7426: Use Mann-Whitney U to compare benchmarks .. IMPALA-7426: Use Mann-Whitney U to compare benchmarks The Mann-Whitney test can be used to compare samples taken from non-normal distributions, and so can more accurately reflect performance changes than a T-test. This patch does not remove t-tests from the benchmark reporting, it just supplements them by including the Mann-Whitney test result as well. Change-Id: I8d6631ebeba1422b832def5cd68537624f672fa0 --- M tests/benchmark/report_benchmark_results.py M tests/util/calculation_util.py 2 files changed, 82 insertions(+), 37 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/94/11194/2 -- To view, visit http://gerrit.cloudera.org:8080/11194 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8d6631ebeba1422b832def5cd68537624f672fa0 Gerrit-Change-Number: 11194 Gerrit-PatchSet: 2 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-7426: Use Mann-Whitney U to compare benchmarks
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11194 ) Change subject: IMPALA-7426: Use Mann-Whitney U to compare benchmarks .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/298/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11194 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8d6631ebeba1422b832def5cd68537624f672fa0 Gerrit-Change-Number: 11194 Gerrit-PatchSet: 1 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Sat, 11 Aug 2018 18:34:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7426: Use Mann-Whitney U to compare benchmarks
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11194 ) Change subject: IMPALA-7426: Use Mann-Whitney U to compare benchmarks .. Patch Set 1: (14 comments) http://gerrit.cloudera.org:8080/#/c/11194/1/tests/benchmark/report_benchmark_results.py File tests/benchmark/report_benchmark_results.py: http://gerrit.cloudera.org:8080/#/c/11194/1/tests/benchmark/report_benchmark_results.py@121 PS1, Line 121: \ flake8: E502 the backslash is redundant between brackets http://gerrit.cloudera.org:8080/#/c/11194/1/tests/benchmark/report_benchmark_results.py@380 PS1, Line 380: ; flake8: E703 statement ends with a semicolon http://gerrit.cloudera.org:8080/#/c/11194/1/tests/benchmark/report_benchmark_results.py@396 PS1, Line 396: s flake8: F821 undefined name 'save_runtime_diffs' http://gerrit.cloudera.org:8080/#/c/11194/1/tests/benchmark/report_benchmark_results.py@396 PS1, Line 396: r flake8: F821 undefined name 'results' http://gerrit.cloudera.org:8080/#/c/11194/1/tests/benchmark/report_benchmark_results.py@396 PS1, Line 396: r flake8: F821 undefined name 'ref_results' http://gerrit.cloudera.org:8080/#/c/11194/1/tests/benchmark/report_benchmark_results.py@396 PS1, Line 396: s flake8: F821 undefined name 'self' http://gerrit.cloudera.org:8080/#/c/11194/1/tests/benchmark/report_benchmark_results.py@396 PS1, Line 396: s flake8: F821 undefined name 'self' http://gerrit.cloudera.org:8080/#/c/11194/1/tests/benchmark/report_benchmark_results.py@396 PS1, Line 396: s flake8: F821 undefined name 'self' http://gerrit.cloudera.org:8080/#/c/11194/1/tests/benchmark/report_benchmark_results.py@573 PS1, Line 573: a flake8: W503 line break before binary operator http://gerrit.cloudera.org:8080/#/c/11194/1/tests/benchmark/report_benchmark_results.py@1016 PS1, Line 1016: def save_runtime_diffs(results, ref_results, change_significant, zval, tval): flake8: E302 expected 2 blank lines, found 1 http://gerrit.cloudera.org:8080/#/c/11194/1/tests/benchmark/report_benchmark_results.py@1036 PS1, Line 1036: ? flake8: E501 line too long (94 > 90 characters) http://gerrit.cloudera.org:8080/#/c/11194/1/tests/util/calculation_util.py File tests/util/calculation_util.py: http://gerrit.cloudera.org:8080/#/c/11194/1/tests/util/calculation_util.py@71 PS1, Line 71: def calculate_mwu(samples, ref_samples): flake8: E302 expected 2 blank lines, found 1 http://gerrit.cloudera.org:8080/#/c/11194/1/tests/util/calculation_util.py@75 PS1, Line 75: , flake8: E231 missing whitespace after ',' http://gerrit.cloudera.org:8080/#/c/11194/1/tests/util/calculation_util.py@76 PS1, Line 76: , flake8: E231 missing whitespace after ',' -- To view, visit http://gerrit.cloudera.org:8080/11194 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8d6631ebeba1422b832def5cd68537624f672fa0 Gerrit-Change-Number: 11194 Gerrit-PatchSet: 1 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Sat, 11 Aug 2018 18:01:08 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7426: Use Mann-Whitney U to compare benchmarks
Jim Apple has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11194 Change subject: IMPALA-7426: Use Mann-Whitney U to compare benchmarks .. IMPALA-7426: Use Mann-Whitney U to compare benchmarks The Mann-Whitney test can be used to compare samples taken from non-normal distributions, and so can more accurately reflect performance changes than a T-test. This patch does not remove t-tests from the benchmark reporting, it just supplements them by including the Mann-Whitney test result as well. Change-Id: I8d6631ebeba1422b832def5cd68537624f672fa0 --- M tests/benchmark/report_benchmark_results.py M tests/util/calculation_util.py 2 files changed, 75 insertions(+), 33 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/94/11194/1 -- To view, visit http://gerrit.cloudera.org:8080/11194 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I8d6631ebeba1422b832def5cd68537624f672fa0 Gerrit-Change-Number: 11194 Gerrit-PatchSet: 1 Gerrit-Owner: Jim Apple
