RE: [PATCH] RE: gcc parallel make check
Jakub, First of all, the -j2 testing shows more tests tested in gcc and libstdc++: -# of expected passes 10133 +# of expected passes 10152 +PASS: 23_containers/set/modifiers/erase/abi_tag.cc (test for excess errors) [...] Not sure where the bug is, could be e.g. in i386.exp for gcc, but for libstdc++ less likely to be there rather than in the split. I looked into this, and believe this problem is already in current trunk, and not due to my patch. I.e. unmodified trunk also has these tests executed several times: libstdc++-v3/testsuite/normal4/libstdc++.log.sep:PASS: 23_containers/map/modifiers/erase/abi_tag.cc libstdc++-v3/testsuite/normal1/libstdc++.log.sep:PASS: 23_containers/map/modifiers/erase/abi_tag.cc I believe the current trunk pattern could indeed match those twice (Makefile.in in trunk): normal1) \ dirs=`cd $$srcdir; echo [ab]* de* [ep]*/*`;; \ normal4) \ dirs=`cd $$srcdir; echo 23_*/[a-km-tw-z]*`;; \ could it be that the pattern in normal1 should have been '[ab]*/ de*/ [ep]*/*' ? Joost
RE: [PATCH] RE: gcc parallel make check
could it be that the pattern in normal1 should have been '[ab]*/ de*/ [ep]*/*' ? I've checked that this fixes the bug in the current trunk split. I.e. files are stil tested, but now only once. Consider this change added to the previously submitted patch.
Re: [PATCH] RE: gcc parallel make check
On 11 September 2014 07:22, VandeVondele Joost wrote: Jakub, First of all, the -j2 testing shows more tests tested in gcc and libstdc++: -# of expected passes 10133 +# of expected passes 10152 +PASS: 23_containers/set/modifiers/erase/abi_tag.cc (test for excess errors) [...] Not sure where the bug is, could be e.g. in i386.exp for gcc, but for libstdc++ less likely to be there rather than in the split. I looked into this, and believe this problem is already in current trunk, and not due to my patch. I.e. unmodified trunk also has these tests executed several times: libstdc++-v3/testsuite/normal4/libstdc++.log.sep:PASS: 23_containers/map/modifiers/erase/abi_tag.cc libstdc++-v3/testsuite/normal1/libstdc++.log.sep:PASS: 23_containers/map/modifiers/erase/abi_tag.cc I believe the current trunk pattern could indeed match those twice (Makefile.in in trunk): normal1) \ dirs=`cd $$srcdir; echo [ab]* de* [ep]*/*`;; \ normal4) \ dirs=`cd $$srcdir; echo 23_*/[a-km-tw-z]*`;; \ could it be that the pattern in normal1 should have been '[ab]*/ de*/ [ep]*/*' ? Yes, we are running these tests multiple times: PASS: 23_containers/map/modifiers/erase/abi_tag.cc (test for excess errors) PASS: 23_containers/multimap/modifiers/erase/abi_tag.cc (test for excess errors) PASS: 23_containers/multiset/modifiers/erase/abi_tag.cc (test for excess errors) PASS: 23_containers/set/modifiers/erase/abi_tag.cc (test for excess errors) PASS: 26_numerics/complex/abi_tag.cc (test for excess errors) I'll fix that.
RE: [PATCH] RE: gcc parallel make check
could it be that the pattern in normal1 should have been '[ab]*/ de*/ [ep]*/*' ? Yes, we are running these tests multiple times: PASS: 23_containers/map/modifiers/erase/abi_tag.cc (test for excess errors) PASS: 23_containers/multimap/modifiers/erase/abi_tag.cc (test for excess errors) PASS: 23_containers/multiset/modifiers/erase/abi_tag.cc (test for excess errors) PASS: 23_containers/set/modifiers/erase/abi_tag.cc (test for excess errors) PASS: 26_numerics/complex/abi_tag.cc (test for excess errors) I'll fix that. Actually, the proper pattern should presumably be '[ab]*/* de*/* [ep]*/*' even though it seems to make no difference in testing. I'll have this included in yet another version of the parallel make check patch (plus some further reschuffling as requested by Jakub), so I think there is no need for you to fix this now.
Re: [PATCH] RE: gcc parallel make check
On 11 September 2014 15:45, VandeVondele Joost joost.vandevond...@mat.ethz.ch wrote: could it be that the pattern in normal1 should have been '[ab]*/ de*/ [ep]*/*' ? Yes, we are running these tests multiple times: PASS: 23_containers/map/modifiers/erase/abi_tag.cc (test for excess errors) PASS: 23_containers/multimap/modifiers/erase/abi_tag.cc (test for excess errors) PASS: 23_containers/multiset/modifiers/erase/abi_tag.cc (test for excess errors) PASS: 23_containers/set/modifiers/erase/abi_tag.cc (test for excess errors) PASS: 26_numerics/complex/abi_tag.cc (test for excess errors) I'll fix that. Actually, the proper pattern should presumably be '[ab]*/* de*/* [ep]*/*' even though it seems to make no difference in testing. Yes, that's what I'm testing. I'll have this included in yet another version of the parallel make check patch (plus some further reschuffling as requested by Jakub), so I think there is no need for you to fix this now. This can (and should) be fixed now, without waiting for some other change.
Re: [PATCH] RE: gcc parallel make check
On Tue, Sep 09, 2014 at 03:14:08PM +, VandeVondele Joost wrote: Attached is a further revision of the patch, now dealing with check-c++. Roughly 50% speedup here at '-j32' (18m vs 12m). For my setup (--enable-languages=c,c++,fortran) I have now improved all targets called in 'make -j32 -k check'. The latter is now 30% faster (15m vs 20m). Note that there are +- 1m fluctuations in these numbers, easily. I currently have no plans to work on other check targets before this patch is committed. OK for trunk ? So, I've tested your patch on a few years old 16way AMD, --enable-checking=yes and with make check -j1 and make check -j2 (the former doesn't parallelize, the latter does). First of all, the -j2 testing shows more tests tested in gcc and libstdc++: === gcc Summary === -# of expected passes 108100 +# of expected passes 108130 # of unexpected failures 175 # of unexpected successes 33 # of expected failures 262 ... === libstdc++ Summary === -# of expected passes 10133 +# of expected passes 10152 In gcc testsuite the difference seems to be (according to sorted PASS lines from *.sum files): +PASS: gcc.target/i386/vect-args.c -m3dnow -O (test for excess errors) +PASS: gcc.target/i386/vect-args.c -m3dnow -O (test for excess errors) +PASS: gcc.target/i386/vect-args.c -m3dnow -O (test for excess errors) +PASS: gcc.target/i386/vect-args.c -m3dnow (test for excess errors) +PASS: gcc.target/i386/vect-args.c -m3dnow (test for excess errors) +PASS: gcc.target/i386/vect-args.c -m3dnow (test for excess errors) +PASS: gcc.target/i386/vect-args.c -mmmx -O (test for excess errors) +PASS: gcc.target/i386/vect-args.c -mmmx -O (test for excess errors) +PASS: gcc.target/i386/vect-args.c -mmmx -O (test for excess errors) +PASS: gcc.target/i386/vect-args.c -mmmx (test for excess errors) +PASS: gcc.target/i386/vect-args.c -mmmx (test for excess errors) +PASS: gcc.target/i386/vect-args.c -mmmx (test for excess errors) +PASS: gcc.target/i386/vect-args.c -msse2 -O (test for excess errors) +PASS: gcc.target/i386/vect-args.c -msse2 -O (test for excess errors) +PASS: gcc.target/i386/vect-args.c -msse2 -O (test for excess errors) +PASS: gcc.target/i386/vect-args.c -msse2 (test for excess errors) +PASS: gcc.target/i386/vect-args.c -msse2 (test for excess errors) +PASS: gcc.target/i386/vect-args.c -msse2 (test for excess errors) +PASS: gcc.target/i386/vect-args.c -msse -O (test for excess errors) +PASS: gcc.target/i386/vect-args.c -msse -O (test for excess errors) +PASS: gcc.target/i386/vect-args.c -msse -O (test for excess errors) +PASS: gcc.target/i386/vect-args.c -msse (test for excess errors) +PASS: gcc.target/i386/vect-args.c -msse (test for excess errors) +PASS: gcc.target/i386/vect-args.c -msse (test for excess errors) +PASS: gcc.target/i386/vect-args.c -O (test for excess errors) +PASS: gcc.target/i386/vect-args.c -O (test for excess errors) +PASS: gcc.target/i386/vect-args.c -O (test for excess errors) +PASS: gcc.target/i386/vect-args.c (test for excess errors) +PASS: gcc.target/i386/vect-args.c (test for excess errors) +PASS: gcc.target/i386/vect-args.c (test for excess errors) and in libstdc++: +PASS: 23_containers/map/modifiers/erase/abi_tag.cc scan-assembler _ZNSt3mapIiiSt4lessIiESaISt4pairIKiiEEE5eraseB5cxx11ESt17_Rb_tree_iteratorIS4_E +PASS: 23_containers/map/modifiers/erase/abi_tag.cc (test for excess errors) +PASS: 23_containers/multimap/modifiers/erase/abi_tag.cc scan-assembler _ZNSt8multimapIiiSt4lessIiESaISt4pairIKiiEEE5eraseB5cxx11ESt17_Rb_tree_iteratorIS4_E +PASS: 23_containers/multimap/modifiers/erase/abi_tag.cc (test for excess errors) +PASS: 23_containers/multiset/modifiers/erase/abi_tag.cc scan-assembler _ZNSt8multisetIiSt4lessIiESaIiEE5eraseB5cxx11ESt23_Rb_tree_const_iteratorIiE +PASS: 23_containers/multiset/modifiers/erase/abi_tag.cc scan-assembler _ZNSt8multisetIiSt4lessIiESaIiEE5eraseB5cxx11ESt23_Rb_tree_const_iteratorIiES5_ +PASS: 23_containers/multiset/modifiers/erase/abi_tag.cc (test for excess errors) +PASS: 23_containers/set/modifiers/erase/abi_tag.cc scan-assembler _ZNSt3setIiSt4lessIiESaIiEE5eraseB5cxx11ESt23_Rb_tree_const_iteratorIiE +PASS: 23_containers/set/modifiers/erase/abi_tag.cc scan-assembler _ZNSt3setIiSt4lessIiESaIiEE5eraseB5cxx11ESt23_Rb_tree_const_iteratorIiES5_ +PASS: 23_containers/set/modifiers/erase/abi_tag.cc (test for excess errors) +PASS: 26_numerics/complex/abi_tag.cc scan-assembler _ZNKSt7complexIdE4imagB5cxx11Ev +PASS: 26_numerics/complex/abi_tag.cc scan-assembler _ZNKSt7complexIdE4realB5cxx11Ev +PASS: 26_numerics/complex/abi_tag.cc scan-assembler _ZNKSt7complexI[eg]E4imagB5cxx11Ev +PASS: 26_numerics/complex/abi_tag.cc scan-assembler _ZNKSt7complexI[eg]E4realB5cxx11Ev +PASS: 26_numerics/complex/abi_tag.cc scan-assembler _ZNKSt7complexIfE4imagB5cxx11Ev +PASS: 26_numerics/complex/abi_tag.cc scan-assembler
Re: [PATCH] RE: gcc parallel make check
On Wed, Sep 10, 2014 at 01:57:01PM +, VandeVondele Joost wrote:
Thanks for testing.
The vect-args.c I explained earlier, and is indeed due to i386.exp hardcoding
those.
IMHO the best fix for that is following, use the same predicate whether to
run the vect-args.c tests or not as is used for all other tests.
Ok for trunk?
2014-09-10 Jakub Jelinek ja...@redhat.com
* gcc.target/i386/i386.exp: Only run vect-args.c tests
if runtest_file_p says they should be run.
--- gcc/testsuite/gcc.target/i386/i386.exp.jj 2014-06-10 07:58:09.0
+0200
+++ gcc/testsuite/gcc.target/i386/i386.exp 2014-09-10 16:09:31.572493879
+0200
@@ -316,13 +316,16 @@ if ![info exists DEFAULT_CFLAGS] then {
dg-init
clearcap-init
+global runtests
# Special case compilation of vect-args.c so we don't have to
# replicate it 10 times.
-foreach type { -mmmx -m3dnow -msse -msse2 } {
- foreach level { -O } {
-set flags $type $level
-verbose -log Testing vect-args, $flags 1
-dg-test $srcdir/$subdir/vect-args.c $flags
+if [runtest_file_p $runtests $srcdir/$subdir/vect-args.c] {
+ foreach type { -mmmx -m3dnow -msse -msse2 } {
+foreach level { -O } {
+ set flags $type $level
+ verbose -log Testing vect-args, $flags 1
+ dg-test $srcdir/$subdir/vect-args.c $flags
+}
}
}
Jakub
RE: [PATCH] RE: gcc parallel make check
You mean enhancing the script to split across arbitrarily long prefixes? That would be great. I've now a script that does something like that: ~/test$ find /data/vjoost/gnu/gcc_trunk/gcc/gcc/testsuite/gfortran.dg/ -maxdepth 1 -type f -printf %f\n | ./generate_patterns.py 500 foo All 3947 files matched the pattern ^[0-9A-Za-z_#+-]*([.][0-9A-Za-z_#+-]+)+ without exception Final 12 patterns and match count: (^[j-z_#+-][p-z_#+-][0-9A-Za-i][0-9A-Za-z_#+-]*([.][0-9A-Za-z_#+-]+)+|^[j-z_#+-][0-9A-Za-o][0-9A-Za-m]([.][0-9A-Za-z_#+-]+)+) matching 469 files (^[0-9A-Za-i][0-9A-Za-n][0-9A-Za-n][0-9A-Za-o][0-9A-Za-z_#+-]*([.][0-9A-Za-z_#+-]+)+|^([.][0-9A-Za-z_#+-]+)+) matching 433 files (^[j-z_#+-][0-9A-Za-o][n-z_#+-][0-9A-Za-z_#+-]*([.][0-9A-Za-z_#+-]+)+|^[0-9A-Za-i][0-9A-Za-n][o-z_#+-]([.][0-9A-Za-z_#+-]+)+) matching 400 files (^[j-z_#+-][p-z_#+-][j-z_#+-][0-9A-Za-z_#+-]*([.][0-9A-Za-z_#+-]+)+|^[0-9A-Za-i]([.][0-9A-Za-z_#+-]+)+) matching 371 files (^[0-9A-Za-i][o-z_#+-][s-z_#+-][0-9A-Za-z_#+-]*([.][0-9A-Za-z_#+-]+)+|^[0-9A-Za-i][0-9A-Za-n][0-9A-Za-n]([.][0-9A-Za-z_#+-]+)+) matching 323 files (^[0-9A-Za-i][o-z_#+-][0-9A-Za-r][o-z_#+-][0-9A-Za-z_#+-]*([.][0-9A-Za-z_#+-]+)+|^[j-z_#+-][p-z_#+-]([.][0-9A-Za-z_#+-]+)+) matching 314 files (^[0-9A-Za-i][o-z_#+-][0-9A-Za-r][0-9A-Za-n][0-9A-Za-z_#+-]*([.][0-9A-Za-z_#+-]+)+|^[j-z_#+-][0-9A-Za-o]([.][0-9A-Za-z_#+-]+)+) matching 314 files (^[j-z_#+-][0-9A-Za-o][0-9A-Za-m][0-9A-Za-i][0-9A-Za-z_#+-]*([.][0-9A-Za-z_#+-]+)+|^[j-z_#+-]([.][0-9A-Za-z_#+-]+)+) matching 272 files (^[0-9A-Za-i][0-9A-Za-n][0-9A-Za-n][p-z_#+-][0-9A-Za-z_#+-]*([.][0-9A-Za-z_#+-]+)+|^[0-9A-Za-i][o-z_#+-]([.][0-9A-Za-z_#+-]+)+) matching 270 files (^[0-9A-Za-i][0-9A-Za-n][o-z_#+-][0-9A-Za-l][0-9A-Za-z_#+-]*([.][0-9A-Za-z_#+-]+)+|^[0-9A-Za-i][0-9A-Za-n]([.][0-9A-Za-z_#+-]+)+) matching 265 files (^[0-9A-Za-i][0-9A-Za-n][o-z_#+-][m-z_#+-][0-9A-Za-z_#+-]*([.][0-9A-Za-z_#+-]+)+|^[0-9A-Za-i][o-z_#+-][0-9A-Za-r]([.][0-9A-Za-z_#+-]+)+) matching 260 files ^[j-z_#+-][0-9A-Za-o][0-9A-Za-m][j-z_#+-][0-9A-Za-z_#+-]*([.][0-9A-Za-z_#+-]+)+ matching 256 files It is a set of patterns that will match any file of the form '^[0-9A-Za-z_#+-]*([.][0-9A-Za-z_#+-]+)+', but such that it splits a list of input files roughly in equal chunks (e.g. between 500 and 500/2 in this example), even if files have long overlapping prefixes. However, I'm unsure if/how this can be integrated, i.e. what precisely is allowed for testsuite filenames, and if this regexp format can be employed in gcc makefiles / tcl / expect harness, suggestions/help appreciated.
RE: [PATCH] RE: gcc parallel make check
Attached is an extended version of the patch, it brings a 100% improvement in
make -j32 -k check-gcc (down from 20min to 10min) by modification of
check_gcc_parallelize.
It includes one non-trivial part, namely a split of the target exps. They are
now all split using a common choice (based on i386), which I believe is
reasonable as it is the target with most tests, and the patterns will be
somewhat similar for other targets (e.g. split of p(rxxx)). The implementation
of this in the makefile uses an odd looking technique to substitute spaces with
commas in a variable, if this can be done more elegantly, I'm happy to make the
change.
Bootstrap and testing revealed one issue, i386.exp hard-codes a loop for the
testcase 'vect-args.c' in order to test 10 different combinations of options.
With the current split (i.e. target x4) this test will thus be executed 4
times. There are two easy options
1) keep the current setup, overhead is small
2) keep the .exp file simple and just replicate this test 10x
I've selected 1), but I can update a patch with 2). Ideally dg-options in the
testcase file itself could be repeated, but I haven't found an example of this.
The script now includes sorting and compression of the ranges, and an
additional sanity check on the input, i.e. that file names start with
[0-9A-Za-z]. Some (few) files seem to start with _ or # (in ./gcc.dg/cpp/).
I'll follow up with a separate patch to improve check_g++_parallelize.
Full 'make -j k32 check' is now dominated by libstdc++ testing, which contains
single goals that run ~1100s (e.g. regex related tests). These uses a slightly
different syntax (see gcc/libstdc++-v3/testsuite/Makefile.am) and I'm not yet
sure how to deal with the .am files.
current patch OK for trunk ?
Joost
patch-speedup-checkfortran-v05.CL
Description: patch-speedup-checkfortran-v05.CL
Index: contrib/generate_tcl_patterns.sh
===
--- contrib/generate_tcl_patterns.sh (revision 0)
+++ contrib/generate_tcl_patterns.sh (revision 0)
@@ -0,0 +1,114 @@
+#! /bin/sh
+
+#
+# based on a list of filenames as input, starting with [0-9A-Za-z],
+# generate regexps that match subsets trying to not exceed a
+# 'maxcount' parameter. Most useful to generate the
+# check_LANG_parallelize assignments needed to split
+# testsuite directories, defining prefix appropriately.
+#
+# Example usage:
+# cd gcc/gcc/testsuite/gfortran.dg
+# ls -1 | ../../../contrib/generate_tcl_patterns.sh 300 dg.exp=gfortran.dg/
+#
+# the first parameter is the maximum number of files.
+# the second parameter the prefix used for printing.
+#
+
+# Copyright (C) 2014 Free Software Foundation
+# Contributed by Joost VandeVondele joost.vandevond...@mat.ethz.ch
+#
+# This file is part of GCC.
+#
+# GCC is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3, or (at your option)
+# any later version.
+#
+# GCC is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with GCC; see the file COPYING. If not, write to
+# the Free Software Foundation, 51 Franklin Street, Fifth Floor,
+# Boston, MA 02110-1301, USA.
+
+gawk -v maxcount=$1 -v prefix=$2 '
+BEGIN{
+ # list of allowed starting chars for a file name in a dir to split
+ achars=0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz
+ ranget=112233
+}
+{
+ if (index(achars,substr($1,1,1))==0){
+ print file : $1 does not start with an allowed character.
+ _assert_exit = 1
+ exit 1
+ }
+ nfiles++ ; files[nfiles]=$1
+}
+END{
+ if (_assert_exit) exit 1
+ for(i=1; i=length(achars); i++) count[substr(achars,i,1)]=0
+ for(i=1; i=nfiles; i++) {
+ if (length(files[i]0)) { count[substr(files[i],1,1)]++ }
+ };
+ asort(count,ordered)
+ countsingle=0
+ groups=0
+ label=
+ for(i=length(achars);i=1;i--) {
+countsingle=countsingle+ordered[i]
+for(j=1;j=length(achars);j++) {
+ if(count[substr(achars,j,1)]==ordered[i]) found=substr(achars,j,1)
+}
+count[found]=-1
+label=label found
+if(i==1) { val=maxcount+1 } else { val=ordered[i-1] }
+if(countsingle+valmaxcount) {
+ subset[label]=countsingle
+ print Adding label: , label, matching files: countsingle
+ groups++
+ countsingle=0
+ label=
+}
+ }
+ print patterns:
+ asort(subset,ordered)
+ for(i=groups;i=1;i--) {
+for(j in subset){
+ if(subset[j]==ordered[i]) found=j
+}
+subset[found]=-1
+if (length(found)==1) {
+ printf(%s%s* \\\n,prefix,found)
+} else {
+ sortandcompress()
+
Re: [PATCH] RE: gcc parallel make check
On 09/09/2014 10:51 AM, VandeVondele Joost wrote: Attached is an extended version of the patch, it brings a 100% improvement in make -j32 -k check-gcc First of all, many thanks for working on this. +# ls -1 | ../../../contrib/generate_tcl_patterns.sh 300 dg.exp=gfortran.dg/ How does this work with subdirectories? Can we replace ls with find? -check_p_numbers=1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 +check_p_numbers=1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 \ + 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 $(shell seq 1 40) ? + if (_assert_exit) exit 1 Haven't you already exited above? A second part of the patch is a new file 'contrib/generate_tcl_patterns.sh' which generates the needed regexp Can we provide a Makefile target to automatically update Makefile.in? -Y
Re: [PATCH] RE: gcc parallel make check
On Tue, Sep 09, 2014 at 02:02:18PM +0400, Yury Gribov wrote: On 09/09/2014 10:51 AM, VandeVondele Joost wrote: Attached is an extended version of the patch, it brings a 100% improvement in make -j32 -k check-gcc First of all, many thanks for working on this. +# ls -1 | ../../../contrib/generate_tcl_patterns.sh 300 dg.exp=gfortran.dg/ How does this work with subdirectories? Can we replace ls with find? Generally, if the argument to *.exp doesn't contain a particular subdirectory, then the wildcard is taken against basenames of the tests. -check_p_numbers=1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 +check_p_numbers=1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 \ + 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 $(shell seq 1 40) ? Would that be sufficiently portable to weirdo hosts (M$Win, Darwin, ...)? We require GNU make, so if it can be written in GNU make text functions, fine, otherwise it is better to keep as is. + if (_assert_exit) exit 1 Haven't you already exited above? A second part of the patch is a new file 'contrib/generate_tcl_patterns.sh' which generates the needed regexp Can we provide a Makefile target to automatically update Makefile.in? No. As I wrote earlier, splitting on filenames and test counts only is only very rough split, all the splits really need to be backed out by real timing data from popular targets. Also, I'm afraid of some tests being left out unintentionally (e.g. the wildcards created at some point, then a new test is added with a weird starting character that hasn't been used before and suddenly it will not be tested with make -j?). Jakub
RE: [PATCH] RE: gcc parallel make check
+# ls -1 | ../../../contrib/generate_tcl_patterns.sh 300
dg.exp=gfortran.dg/
How does this work with subdirectories? Can we replace ls with find?
The input to the script is general, you can use this to your advantage. For
example, I've been using:
ls -1 g++.*/* | cut -c5- | ../../../contrib/generate_tcl_patterns.sh 700
old-deja.exp=g++.old-deja/g++.
to split at a deeper level or
find . -name [0-9A-Za-z]* -type f -printf %f\n |
../../../../contrib/generate_tcl_patterns.sh 300 dg-torture.exp=torture/
to collect statistics also from subdirs.
+ if (_assert_exit) exit 1
Haven't you already exited above?
yes, but the END{} block in awk is nevertheless executed, unless protected as
above.
RE: [PATCH] RE: gcc parallel make check
No. As I wrote earlier, splitting on filenames and test counts only is only very rough split, all the splits really need to be backed out by real timing data from popular targets. I'm actually doing quite some testing trying to get a reasonable balance, checking 'completed in' in all *.log.sep files. However, it is important that the procedure is semi-automatic, otherwise few people will be interested in doing so. Furthermore, for parallel performance, it is not so important that times are distributed evenly (it is anyway unlikely the number of goals is exactly divided by N of -jN), but rather that the goals are ordered (executed) from slow to fast (similar to omp schedule guided). Most of the real bottlenecks are single letter patterns (e.g. p* since pr is such a common filename), and this is ultimately limiting. In the project (CP2K) I'm working on, we also parallelize testing over directories, but we keep a list of approximate runtimes per directory, and keep that (global) list sorted. Testing follows that list. As a result, we have near perfect parallel speedup, despite (or because) timings per directory ranging from a few 100s to 1s. Also, I'm afraid of some tests being left out unintentionally (e.g. the wildcards created at some point, then a new test is added with a weird starting character that hasn't been used before and suddenly it will not be tested with make -j?). I agree this is an issue, partially addressed by not having to write patterns by hand anymore (i.e. a script does this), and by having the script check its input. There are something like 10 testnames that do not fall in [0-9A-Za-z], as mentioned in a previous email.
Re: [PATCH] RE: gcc parallel make check
On Tue, Sep 09, 2014 at 10:57:09AM +, VandeVondele Joost wrote: No. As I wrote earlier, splitting on filenames and test counts only is only very rough split, all the splits really need to be backed out by real timing data from popular targets. Furthermore, for parallel performance, it is not so important that times are distributed evenly (it is anyway unlikely the number of goals is exactly divided by N of -jN), but rather that the goals are ordered (executed) from slow to fast (similar to omp schedule guided). Most of the real bottlenecks are single letter patterns (e.g. p* since pr is such a common filename), and this is ultimately limiting. I disagree. If e.g. in gcc.dg/ more than a third of testcases are pr*.c, then running dg.exp=p* in one job and dg.exp=a* in another one etc. is simply a bad idea, the pr*.c should be split more and some other letters just be done together. Even that can be done semi-automatically. If you get whitespace right, one can provide multiple different wildcards to a single *.exp file, e.g. make check-gcc RUNTESTFLAGS=dg.exp='p[0-9A-Za-qs-z]* pr[9A-Za-z]*' should cover all tests starting with p other than pr[0-8]*.c (where you could split say pr[0-2]* into another job, pr[3-5]* into another and pr[6-8]* into another. The fact that some check-gcc or check-gfortran test job is early in the list doesn't mean it will be started early, you need to consider also all other potentially long jobs like check-g++, check-target-libgomp, check-target-libstdc++-v3 etc. Jakub
RE: [PATCH] RE: gcc parallel make check
If you get whitespace right, one can provide multiple different wildcards to a single *.exp file, e.g. make check-gcc RUNTESTFLAGS=dg.exp='p[0-9A-Za-qs-z]* pr[9A-Za-z]*' should cover all tests starting with p other than pr[0-8]*.c (where you could split say pr[0-2]* into another job, pr[3-5]* into another and pr[6-8]* into another. I think this confirms that it becomes very delicate to try and write these more complex patterns. The above would miss p_test.c, p-1.c, etc ? For other classes of files the difference is even further down the filename (e.g. using dates as in 20020508-3.c going from 2000 to 2014, or avx*), making the automatic generation of the patterns more complicated. I certainly don't want to claim that the patch I have now is perfect, it is rather an incremental improvement on the current setup.
Re: [PATCH] RE: gcc parallel make check
On 09/09/2014 06:14 PM, VandeVondele Joost wrote: I certainly don't want to claim that the patch I have now is perfect, it is rather an incremental improvement on the current setup. I'd second this. Writing patterns manually seems rather inefficient and error-prone (not undoable of course but unnecessarily complicated). And with current (crippled) version Joost already got 100% test time improvement. -Y
Re: [PATCH] RE: gcc parallel make check
On Tue, Sep 09, 2014 at 06:27:10PM +0400, Yury Gribov wrote: On 09/09/2014 06:14 PM, VandeVondele Joost wrote: I certainly don't want to claim that the patch I have now is perfect, it is rather an incremental improvement on the current setup. I'd second this. Writing patterns manually seems rather inefficient and error-prone (not undoable of course but unnecessarily complicated). And with current (crippled) version Joost already got 100% test time improvement. But if there are jobs that just take 1s to complete, then clearly it doesn't make sense to split them off as separate job. I think we don't need 100% even split, but at least roughly is highly desirable. Jakub
Re: [PATCH] RE: gcc parallel make check
On 09/09/2014 06:33 PM, Jakub Jelinek wrote: On Tue, Sep 09, 2014 at 06:27:10PM +0400, Yury Gribov wrote: On 09/09/2014 06:14 PM, VandeVondele Joost wrote: I certainly don't want to claim that the patch I have now is perfect, it is rather an incremental improvement on the current setup. I'd second this. Writing patterns manually seems rather inefficient and error-prone (not undoable of course but unnecessarily complicated). And with current (crippled) version Joost already got 100% test time improvement. But if there are jobs that just take 1s to complete, then clearly it doesn't make sense to split them off as separate job. I think we don't need 100% even split, but at least roughly is highly desirable. You mean enhancing the script to split across arbitrarily long prefixes? That would be great. -Y
RE: [PATCH] RE: gcc parallel make check
Now with gzipped figure.. why do these bounce ? But if there are jobs that just take 1s to complete, then clearly it doesn't make sense to split them off as separate job. I think we don't need 100% even split, but at least roughly is highly desirable. Let me add some data, attached is a graph (logscale y) showing the runtime of tests before and after my changes (including a new patch for c++). There is virtually no change for tests running shorter than 50s, only slowly running tests have been split. Now, there are only very few slow tests remaining: gcc_trunk/obj.new find . -name *.log | xargs grep completed in | sort -n -k 5 | tail -n 10 ./gcc/testsuite/gcc/gcc.log:testcase /data/vjoost/gnu/gcc_trunk/gcc/gcc/testsuite/gcc.dg/torture/dg-torture.exp completed in 521 seconds ./x86_64-unknown-linux-gnu/libstdc++-v3/testsuite/libstdc++.log:testcase /data/vjoost/gnu/gcc_trunk/gcc/libstdc++-v3/testsuite/libstdc++-dg/conformance.exp completed in 530 seconds ./x86_64-unknown-linux-gnu/libstdc++-v3/testsuite/libstdc++.log:testcase /data/vjoost/gnu/gcc_trunk/gcc/libstdc++-v3/testsuite/libstdc++-dg/conformance.exp completed in 553 seconds ./x86_64-unknown-linux-gnu/libgomp/testsuite/libgomp.log:testcase /data/vjoost/gnu/gcc_trunk/gcc/libgomp/testsuite/libgomp.fortran/fortran.exp completed in 561 seconds ./gcc/testsuite/gcc/gcc.log:testcase /data/vjoost/gnu/gcc_trunk/gcc/gcc/testsuite/gcc.c-torture/compile/compile.exp completed in 625 seconds ./x86_64-unknown-linux-gnu/libstdc++-v3/testsuite/libstdc++.log:testcase /data/vjoost/gnu/gcc_trunk/gcc/libstdc++-v3/testsuite/libstdc++-dg/conformance.exp completed in 683 seconds ./gcc/testsuite/g++/g++.log:testcase /data/vjoost/gnu/gcc_trunk/gcc/gcc/testsuite/g++.dg/dg.exp completed in 702 seconds ./x86_64-unknown-linux-gnu/libstdc++-v3/testsuite/libstdc++.log:testcase /data/vjoost/gnu/gcc_trunk/gcc/libstdc++-v3/testsuite/libstdc++-dg/conformance.exp completed in 726 seconds ./gcc/testsuite/gcc/gcc.log:testcase /data/vjoost/gnu/gcc_trunk/gcc/gcc/testsuite/gcc.c-torture/execute/execute.exp completed in 752 seconds ./x86_64-unknown-linux-gnu/libstdc++-v3/testsuite/libstdc++.log:testcase /data/vjoost/gnu/gcc_trunk/gcc/libstdc++-v3/testsuite/libstdc++-dg/conformance.exp completed in 904 seconds They, of course, limit the ultimate speedup. timings.png.gz Description: timings.png.gz
RE: [PATCH] RE: gcc parallel make check
Attached is a further revision of the patch, now dealing with check-c++. Roughly 50% speedup here at '-j32' (18m vs 12m). For my setup (--enable-languages=c,c++,fortran) I have now improved all targets called in 'make -j32 -k check'. The latter is now 30% faster (15m vs 20m). Note that there are +- 1m fluctuations in these numbers, easily. I currently have no plans to work on other check targets before this patch is committed. OK for trunk ? Joost contrib/ChangeLog 2014-09-09 Joost VandeVondele vond...@gcc.gnu.org * generate_tcl_patterns.sh: New file. gcc/fortran/ChangeLog 2014-09-09 Joost VandeVondele vond...@gcc.gnu.org * Make-lang.in (check_gfortran_parallelize): Improved parallelism. gcc/Changelog 2014-09-09 Joost VandeVondele vond...@gcc.gnu.org * Makefile.in (check_gcc_parallelize): Improved parallelism. (check_p_numbers): Increase maximum value. (dg_target_exps): Mention targets as separate words only. (null,space,comma,dg_target_exps_p1,dg_target_exps_p2, dg_target_exps_p3,dg_target_exps_p4): New variables. gcc/cp/ChangeLog 2014-09-09 Joost VandeVondele vond...@gcc.gnu.org * Make-lang.in (check_g++_parallelize): Improved parallelism. libstdc++-v3/ChangeLog 2014-09-09 Joost VandeVondele vond...@gcc.gnu.org * testsuite/Makefile.am (check_DEJAGNU_normal_targets): Add check-DEJAGNUnormal[11-15]. (check-DEJAGNU): Split into 15 jobs for parallel testing. * testsuite/Makefile.in: Regenerated. Index: libstdc++-v3/testsuite/Makefile.am === --- libstdc++-v3/testsuite/Makefile.am (revision 215017) +++ libstdc++-v3/testsuite/Makefile.am (working copy) @@ -101,7 +101,7 @@ new-abi-baseline: @test ! -f $*/site.exp || mv $*/site.exp $*/site.bak @mv $*/site.exp.tmp $*/site.exp -check_DEJAGNU_normal_targets = $(patsubst %,check-DEJAGNUnormal%,0 1 2 3 4 5 6 7 8 9 10) +check_DEJAGNU_normal_targets = $(patsubst %,check-DEJAGNUnormal%,0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15) $(check_DEJAGNU_normal_targets): check-DEJAGNUnormal%: normal%/site.exp # Run the testsuite in normal mode. @@ -111,7 +111,7 @@ check-DEJAGNU $(check_DEJAGNU_normal_tar if [ -z $*$(filter-out --target_board=%, $(RUNTESTFLAGS)) ] \ [ $(filter -j, $(MFLAGS)) = -j ]; then \ $(MAKE) $(AM_MAKEFLAGS) $(check_DEJAGNU_normal_targets); \ - for idx in 0 1 2 3 4 5 6 7 8 9 10; do \ + for idx in 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15; do \ mv -f normal$$idx/libstdc++.sum normal$$idx/libstdc++.sum.sep; \ mv -f normal$$idx/libstdc++.log normal$$idx/libstdc++.log.sep; \ done; \ @@ -138,25 +138,35 @@ check-DEJAGNU $(check_DEJAGNU_normal_tar fi; \ dirs=`cd $$srcdir; echo [013-9][0-9]_*/*`;; \ normal1) \ - dirs=`cd $$srcdir; echo [ab]* de* [ep]*/*`;; \ + dirs=`cd $$srcdir; echo e*/*`;; \ normal2) \ - dirs=`cd $$srcdir; echo 2[01]_*/*`;; \ + dirs=`cd $$srcdir; echo 28_*/a*`;; \ normal3) \ - dirs=`cd $$srcdir; echo 22_*/*`;; \ + dirs=`cd $$srcdir; echo 23_*/[lu]*`;; \ normal4) \ - dirs=`cd $$srcdir; echo 23_*/[a-km-tw-z]*`;; \ + dirs=`cd $$srcdir; echo 2[459]_*/*`;; \ normal5) \ - dirs=`cd $$srcdir; echo 23_*/[luv]*`;; \ + dirs=`cd $$srcdir; echo 2[01]_*/*`;; \ normal6) \ - dirs=`cd $$srcdir; echo 2[459]_*/*`;; \ + dirs=`cd $$srcdir; echo 23_*/[m-tw-z]*`;; \ normal7) \ - dirs=`cd $$srcdir; echo 26_*/* 28_*/[c-z]*`;; \ + dirs=`cd $$srcdir; echo 26_*/*`;; \ normal8) \ dirs=`cd $$srcdir; echo 27_*/*`;; \ normal9) \ - dirs=`cd $$srcdir; echo 28_*/[ab]*`;; \ + dirs=`cd $$srcdir; echo 22_*/*`;; \ normal10) \ dirs=`cd $$srcdir; echo t*/*`;; \ + normal11) \ + dirs=`cd $$srcdir; echo 28_*/b*`;; \ + normal12) \ + dirs=`cd $$srcdir; echo 28_*/[c-z]*`;; \ + normal13) \ + dirs=`cd $$srcdir; echo de* p*/*`;; \ + normal14) \ + dirs=`cd $$srcdir; echo [ab]* 23_*/v*`;; \ + normal15) \ + dirs=`cd $$srcdir; echo 23_*/[a-k]*`;; \ esac; \ if [ -n $* ]; then cd $*; fi; \ if $(SHELL) -c $$runtest --version /dev/null 21; then \ Index: libstdc++-v3/testsuite/Makefile.in === --- libstdc++-v3/testsuite/Makefile.in (revision 215017) +++ libstdc++-v3/testsuite/Makefile.in (working copy) @@ -301,7 +301,7 @@ lists_of_files = \ extract_symvers = $(glibcxx_builddir)/scripts/extract_symvers baseline_subdir := $(shell $(CXX) $(baseline_subdir_switch)) -check_DEJAGNU_normal_targets = $(patsubst %,check-DEJAGNUnormal%,0 1 2 3 4 5 6 7 8 9 10) +check_DEJAGNU_normal_targets = $(patsubst %,check-DEJAGNUnormal%,0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15) # Runs the testsuite, but in compile only mode. # Can be used to test sources with non-GNU FE's at various warning @@ -562,7 +562,7 @@ check-DEJAGNU $(check_DEJAGNU_normal_tar if [ -z
RE: [PATCH] RE: gcc parallel make check
Please sort the letters (LC_ALL=C sort) and where consecutive, use ranges.
Thus \[0-9A-Zhjqvx-z\]*
OK, works fine with the attached patch, and looks cleaner in Make-lang.in.
Now, with the proper email address for gcc-patches... I wonder how many time
I'll be punished for typos.
unmodified CL.
Joost
Index: contrib/generate_tcl_patterns.sh
===
--- contrib/generate_tcl_patterns.sh (revision 0)
+++ contrib/generate_tcl_patterns.sh (revision 0)
@@ -0,0 +1,108 @@
+#! /bin/sh
+
+#
+# based on a list of filenames as input,
+# generate regexps that match subsets trying to not exceed a
+# 'maxcount' parameter. Most useful to generate the
+# check_LANG_parallelize assignments needed to split
+# testsuite directories, defining prefix appropriately.
+#
+# Example usage:
+# cd gcc/gcc/testsuite/gfortran.dg
+# ls -1 | ../../../contrib/generate_tcl_patterns.sh 300 dg.exp=gfortran.dg/
+#
+# the first parameter is the maximum number of files.
+# the second parameter the prefix used for printing.
+#
+
+# Copyright (C) 2014 Free Software Foundation
+# Contributed by Joost VandeVondele joost.vandevond...@mat.ethz.ch
+#
+# This file is part of GCC.
+#
+# GCC is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3, or (at your option)
+# any later version.
+#
+# GCC is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with GCC; see the file COPYING. If not, write to
+# the Free Software Foundation, 51 Franklin Street, Fifth Floor,
+# Boston, MA 02110-1301, USA.
+
+gawk -v maxcount=$1 -v prefix=$2 '
+BEGIN{
+ # list of allowed starting chars for a file name in a dir to split
+ achars=0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz
+ ranget=112233
+}
+{
+ nfiles++ ; files[nfiles]=$1
+}
+END{
+ for(i=1; i=length(achars); i++) count[substr(achars,i,1)]=0
+ for(i=1; i=nfiles; i++) {
+ if (length(files[i]0)) { count[substr(files[i],1,1)]++ }
+ };
+ asort(count,ordered)
+ countsingle=0
+ groups=0
+ label=
+ for(i=length(achars);i=1;i--) {
+countsingle=countsingle+ordered[i]
+for(j=1;j=length(achars);j++) {
+ if(count[substr(achars,j,1)]==ordered[i]) found=substr(achars,j,1)
+}
+count[found]=-1
+label=label found
+if(i==1) { val=maxcount+1 } else { val=ordered[i-1] }
+if(countsingle+valmaxcount) {
+ subset[label]=countsingle
+ print Adding label: , label, matching files: countsingle
+ groups++
+ countsingle=0
+ label=
+}
+ }
+ print patterns:
+ asort(subset,ordered)
+ for(i=groups;i=1;i--) {
+for(j in subset){
+ if(subset[j]==ordered[i]) found=j
+}
+subset[found]=-1
+if (length(found)==1) {
+ printf(%s%s* \\\n,prefix,found)
+} else {
+ sortandcompress()
+ printf(%s\\[%s\\]* \\\n,prefix,found)
+}
+ }
+}
+function sortandcompress(i,n,tmp,bestj)
+{
+ n=length(found)
+ for(i=1; i=n; i++) tmp[i]=substr(found,i,1)
+ asort(tmp)
+ for(i=1;i=n;i++){
+ipos=index(achars,tmp[i])
+for(j=i;j=n;j++){
+ jpos=index(achars,tmp[j])
+ if (jpos-ipos==j-i substr(ranget,ipos,1)==substr(ranget,jpos,1)) bestj=j
+}
+if (bestj-i3) {
+ tmp[i+1]=-
+ for(j=i+2;jbestj;j++) tmp[j]=
+}
+i=bestj
+ }
+ found=
+ for(i=1; i=n; i++) found=found tmp[i]
+ gsub(/ /,,found)
+}
+'
Index: gcc/fortran/Make-lang.in
===
--- gcc/fortran/Make-lang.in (revision 214949)
+++ gcc/fortran/Make-lang.in (working copy)
@@ -168,12 +168,22 @@ check-fortran-subtargets : check-gfortra
lang_checks += check-gfortran
lang_checks_parallelized += check-gfortran
# For description see comment above check_gcc_parallelize in gcc/Makefile.in.
-check_gfortran_parallelize = dg.exp=gfortran.dg/\[adAD\]* \
- dg.exp=gfortran.dg/\[bcBC\]* \
- dg.exp=gfortran.dg/\[nopNOP\]* \
- dg.exp=gfortran.dg/\[isuvISUV\]* \
- dg.exp=gfortran.dg/\[efhkqrxzEFHKQRXZ\]* \
- dg.exp=gfortran.dg/\[0-9gjlmtwyGJLMTWY\]*
+check_gfortran_parallelize = execute.exp \
+ dg.exp=gfortran.dg/p* \
+ dg.exp=gfortran.dg/c* \
+ dg.exp=gfortran.dg/a* \
+ dg.exp=gfortran.dg/i* \
+ dg.exp=gfortran.dg/\[glow\]* \
+ dg.exp=gfortran.dg/\[mu\]* \
+ dg.exp=gfortran.dg/d* \
+ dg.exp=gfortran.dg/s* \
+ dg.exp=gfortran.dg/b* \
+ dg.exp=gfortran.dg/t* \
+ dg.exp=gfortran.dg/f* \
+ dg.exp=gfortran.dg/e* \
+ dg.exp=gfortran.dg/r* \
+ dg.exp=gfortran.dg/n* \
+ dg.exp=gfortran.dg/\[0-9A-Zhjkqvxyz\]*
# GFORTRAN
