Add a contrib script for comparing the performance of two sets of
compiler runs.
Usage documentation is in the script.
The script produces output of the form:
$ compare_two_ftime_report_sets Log0/*perf Log3/*perf
Arithmetic sample for timevar log files
Log0/*perf
and selecting lines containing TOTAL with desired confidence 95 is
trial count is 4, mean is 443.022 (95% confidence in 440.234 to 445.811),
std.deviation is 1.75264, std.error is 0.876322
Arithmetic sample for timevar log files
Log3/*perf
and selecting lines containing TOTAL with desired confidence 95 is
trial count is 4, mean is 441.302 (95% confidence in 436.671 to 445.934),
std.deviation is 2.91098, std.error is 1.45549
The first sample appears to be 0.39% larger,
with 60% confidence of being larger.
To reach 95% confidence, you need roughly 14 trials,
assuming the standard deviation is stable, which is iffy.
Tested on x86_64 builds.
Okay for trunk?
Index: contrib/ChangeLog
2012-11-05 Lawrence Crowl cr...@google.com
* compare_two_ftime_report_sets: New.
Index: contrib/compare_two_ftime_report_sets
===
--- contrib/compare_two_ftime_report_sets (revision 0)
+++ contrib/compare_two_ftime_report_sets (revision 0)
@@ -0,0 +1,605 @@
+#!/usr/bin/python
+
+# Script to statistically compare two sets of log files with -ftime-report
+# output embedded within them.
+
+# Contributed by Lawrence Crowl cr...@google.com
+#
+# Copyright (C) 2012 Free Software Foundation, Inc.
+#
+# This file is part of GCC.
+#
+# GCC is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3, or (at your option)
+# any later version.
+#
+# GCC is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with GCC; see the file COPYING. If not, write to
+# the Free Software Foundation, 51 Franklin Street, Fifth Floor,
+# Boston, MA 02110-1301, USA.
+
+
+ Compare two sets of compile-time performance numbers.
+
+The intent of this script is to compare compile-time performance of two
+different versions of the compiler. Each version of the compiler must be
+run at least three times with the -ftime-report option. Each log file
+represents a data point, or trial. The set of trials for each compiler
+version constitutes a sample. The ouput of the script is a description
+of the statistically significant difference between the two version of
+the compiler.
+
+The parameters to the script are:
+
+ Two file patterns that each match a set of log files. You will probably
+ need to quote the patterns before passing them to the script.
+
+Each pattern corresponds to a version of the compiler.
+
+ A regular expression that finds interesting lines in the log files.
+ If you want to match the beginning of the line, you will need to add
+ the ^ operator. The filtering uses Python regular expression syntax.
+
+The default is TOTAL.
+
+All of the interesting lines in a single log file are summed to produce
+a single trial (data point).
+
+ A desired statistical confidence within the range 60% to 99.9%. Due to
+ the implementation, this confidence will be rounded down to one of 60%,
+ 70%, 80%, 90%, 95%, 98%, 99%, 99.5%, 99.8%, and 99.9%.
+
+The default is 95.
+
+If the computed confidence is lower than desired, the script will
+estimate the number of trials needed to meet the desired confidence.
+This estimate is not very good, as the variance tends to change as
+you increase the number of trials.
+
+The most common use of the script is total compile-time comparison between
+logfiles stored in different directories.
+
+compare_two_ftime_report_sets Log1/*perf Log2/*perf
+
+One can also look at parsing time, but expecting a lower confidence.
+
+compare_two_ftime_report_sets Log1/*perf Log2/*perf ^phase parsing 75
+
+
+
+
+import os
+import sys
+import fnmatch
+import glob
+import re
+import math
+
+
+### Utility
+
+
+def divide(dividend, divisor):
+ Return the quotient, avoiding division by zero.
+
+ if divisor == 0:
+return sys.float_info.max
+ else:
+return dividend / divisor
+
+
+# File and Line
+
+
+# Should you repurpose this script, this code might help.
+#
+#def find_files(topdir, filepat):
+# Find a set of file names, under a given directory,
+# matching a Unix shell file pattern.
+# Returns an iterator over the file names.
+#
+# for path, dirlist, filelist in os.walk(topdir):
+#for name in fnmatch.filter(filelist, filepat):
