Re: [PATCH 2/3] Run profile feedback tests with autofdo
On 07/25/2016 10:28 PM, Andi Kleen wrote: But it's not expected that it is not deterministic, so that it changes run to run. Do you see that? Or some other problem? Please describe it exactly. It definitely changes run to run for me. And do you have autofdo installed? (create_gcov) No. jeff
Re: [PATCH 2/3] Run profile feedback tests with autofdo
On 07/26/2016 06:28 AM, Andi Kleen wrote: > And do you have autofdo installed? (create_gcov) > > -Andi Ah, sorry for the false alarm, create_gcov is really missing on my distribution. Martin
Re: [PATCH 2/3] Run profile feedback tests with autofdo
> >But it's not expected that it is not deterministic, so that it changes > >run to run. Do you see that? Or some other problem? Please describe > >it exactly. > It definitely changes run to run for me. And do you have autofdo installed? (create_gcov) -Andi
Re: [PATCH 2/3] Run profile feedback tests with autofdo
On 07/25/2016 10:21 PM, Andi Kleen wrote: On Mon, Jul 25, 2016 at 01:08:43PM +0200, Martin Liška wrote: On 07/15/2016 10:37 AM, Bin.Cheng wrote: On Thu, Jul 14, 2016 at 11:33 PM, Andi Kleen wrote: I haven't seen that. Unstable in what way? For GCC doesn't support FDO, it run below tests as you said: PASS: gcc.dg/tree-prof/20041218-1.c compilation, -g UNSUPPORTED: gcc.dg/tree-prof/20041218-1.c: Cannot run create_gcov --binary /data/work/trunk/build/gcc/testsuite/gcc/20041218-1.gcda Normally, it doesn't create gcov data file, thus the next test is unsupported. I guess in parallel test, the second test picks up gcov data files from other process, which results in random pass. Is it possible to not have these when fdo is supported? Hmm, typo: s/supported/not supported/. I don't see how this can happen: as you can see the .gcda file name is unique for each test case. I think there may be problems with the perf.data, could add a postfix. But you should only see that with real autofdo. Don't know. I only configured/built GCC on x86_64 with below command lines: $ ../gcc/configure --prefix=... --enable-languages=c,c++,fortran,go,java,lto,objc,obj-c++ --with-build-config=bootstrap-O3 --disable-werror $ make && make install Not sure what the status for autofdo is in this case. "make check -k" is stable for me, but "make check -k -j#" gives unstable result in tree-prof.exp tests. Anything I did wrong? Thanks, bin I can confirm that I see the same problem on a Xeon machine (../configure && make && make check -k -jX). I see many: g++.dg/tree-prof/pr35545.C: Cannot run create_gcov --binary /home/marxin/Programming/gcc/objdir/gcc/testsuite/g++/pr35545.x01 --profile=perf.data -gcov_version=1 --gcov=/home/marxin/Programming/gcc/objdir/gcc/testsuite/g++/pr35545.gcda Well that's expected if you don't have autofdo installed or perf with lbrs is not working. The test case will be UNSUP in this case. But it's not expected that it is not deterministic, so that it changes run to run. Do you see that? Or some other problem? Please describe it exactly. It definitely changes run to run for me. Jeff
Re: [PATCH 2/3] Run profile feedback tests with autofdo
On Mon, Jul 25, 2016 at 01:08:43PM +0200, Martin Liška wrote: > On 07/15/2016 10:37 AM, Bin.Cheng wrote: > > On Thu, Jul 14, 2016 at 11:33 PM, Andi Kleen wrote: > > I haven't seen that. Unstable in what way? > For GCC doesn't support FDO, it run below tests as you said: > > PASS: gcc.dg/tree-prof/20041218-1.c compilation, -g > UNSUPPORTED: gcc.dg/tree-prof/20041218-1.c: Cannot run create_gcov > --binary /data/work/trunk/build/gcc/testsuite/gcc/20041218-1.gcda > > Normally, it doesn't create gcov data file, thus the next test is > unsupported. I guess in parallel test, the second test picks up gcov > data files from other process, which results in random pass. > Is it possible to not have these when fdo is supported? > >>> Hmm, typo: s/supported/not supported/. > >> > >> I don't see how this can happen: as you can see the .gcda file name > >> is unique for each test case. > >> > >> I think there may be problems with the perf.data, could add a postfix. > >> But you should only see that with real autofdo. > > Don't know. I only configured/built GCC on x86_64 with below command lines: > > > > $ ../gcc/configure --prefix=... > > --enable-languages=c,c++,fortran,go,java,lto,objc,obj-c++ > > --with-build-config=bootstrap-O3 --disable-werror > > $ make && make install > > > > Not sure what the status for autofdo is in this case. "make check -k" > > is stable for me, but "make check -k -j#" gives unstable result in > > tree-prof.exp tests. Anything I did wrong? > > > > Thanks, > > bin > > > > I can confirm that I see the same problem on a Xeon machine (../configure && > make && make check -k -jX). > I see many: > > g++.dg/tree-prof/pr35545.C: Cannot run create_gcov --binary > /home/marxin/Programming/gcc/objdir/gcc/testsuite/g++/pr35545.x01 > --profile=perf.data -gcov_version=1 > --gcov=/home/marxin/Programming/gcc/objdir/gcc/testsuite/g++/pr35545.gcda Well that's expected if you don't have autofdo installed or perf with lbrs is not working. The test case will be UNSUP in this case. But it's not expected that it is not deterministic, so that it changes run to run. Do you see that? Or some other problem? Please describe it exactly. If yes, did it start before r238320 or only afer? -Andi
Re: [PATCH 2/3] Run profile feedback tests with autofdo
On 07/15/2016 10:37 AM, Bin.Cheng wrote: > On Thu, Jul 14, 2016 at 11:33 PM, Andi Kleen wrote: > I haven't seen that. Unstable in what way? For GCC doesn't support FDO, it run below tests as you said: PASS: gcc.dg/tree-prof/20041218-1.c compilation, -g UNSUPPORTED: gcc.dg/tree-prof/20041218-1.c: Cannot run create_gcov --binary /data/work/trunk/build/gcc/testsuite/gcc/20041218-1.gcda Normally, it doesn't create gcov data file, thus the next test is unsupported. I guess in parallel test, the second test picks up gcov data files from other process, which results in random pass. Is it possible to not have these when fdo is supported? >>> Hmm, typo: s/supported/not supported/. >> >> I don't see how this can happen: as you can see the .gcda file name >> is unique for each test case. >> >> I think there may be problems with the perf.data, could add a postfix. >> But you should only see that with real autofdo. > Don't know. I only configured/built GCC on x86_64 with below command lines: > > $ ../gcc/configure --prefix=... > --enable-languages=c,c++,fortran,go,java,lto,objc,obj-c++ > --with-build-config=bootstrap-O3 --disable-werror > $ make && make install > > Not sure what the status for autofdo is in this case. "make check -k" > is stable for me, but "make check -k -j#" gives unstable result in > tree-prof.exp tests. Anything I did wrong? > > Thanks, > bin > I can confirm that I see the same problem on a Xeon machine (../configure && make && make check -k -jX). I see many: g++.dg/tree-prof/pr35545.C: Cannot run create_gcov --binary /home/marxin/Programming/gcc/objdir/gcc/testsuite/g++/pr35545.x01 --profile=perf.data -gcov_version=1 --gcov=/home/marxin/Programming/gcc/objdir/gcc/testsuite/g++/pr35545.gcda Thanks, Martin
Re: [PATCH 2/3] Run profile feedback tests with autofdo
> >Not sure what the status for autofdo is in this case. "make check -k" > >is stable for me, but "make check -k -j#" gives unstable result in > >tree-prof.exp tests. Anything I did wrong? > I'm seeing unstable results as well, but haven't dug into it at all. > It's definitely autofdo testing though -- If I remove the > gcc-auto-profile script I get consistent results from one run to the > next. Ok. I'll investigate. -Andi
Re: [PATCH 2/3] Run profile feedback tests with autofdo
On 07/15/2016 02:37 AM, Bin.Cheng wrote: On Thu, Jul 14, 2016 at 11:33 PM, Andi Kleen wrote: I haven't seen that. Unstable in what way? For GCC doesn't support FDO, it run below tests as you said: PASS: gcc.dg/tree-prof/20041218-1.c compilation, -g UNSUPPORTED: gcc.dg/tree-prof/20041218-1.c: Cannot run create_gcov --binary /data/work/trunk/build/gcc/testsuite/gcc/20041218-1.gcda Normally, it doesn't create gcov data file, thus the next test is unsupported. I guess in parallel test, the second test picks up gcov data files from other process, which results in random pass. Is it possible to not have these when fdo is supported? Hmm, typo: s/supported/not supported/. I don't see how this can happen: as you can see the .gcda file name is unique for each test case. I think there may be problems with the perf.data, could add a postfix. But you should only see that with real autofdo. Don't know. I only configured/built GCC on x86_64 with below command lines: $ ../gcc/configure --prefix=... --enable-languages=c,c++,fortran,go,java,lto,objc,obj-c++ --with-build-config=bootstrap-O3 --disable-werror $ make && make install Not sure what the status for autofdo is in this case. "make check -k" is stable for me, but "make check -k -j#" gives unstable result in tree-prof.exp tests. Anything I did wrong? I'm seeing unstable results as well, but haven't dug into it at all. It's definitely autofdo testing though -- If I remove the gcc-auto-profile script I get consistent results from one run to the next. jeff
Re: [PATCH 2/3] Run profile feedback tests with autofdo
On Thu, Jul 14, 2016 at 11:33 PM, Andi Kleen wrote: >> >> I haven't seen that. Unstable in what way? >> > For GCC doesn't support FDO, it run below tests as you said: >> > >> > PASS: gcc.dg/tree-prof/20041218-1.c compilation, -g >> > UNSUPPORTED: gcc.dg/tree-prof/20041218-1.c: Cannot run create_gcov >> > --binary /data/work/trunk/build/gcc/testsuite/gcc/20041218-1.gcda >> > >> > Normally, it doesn't create gcov data file, thus the next test is >> > unsupported. I guess in parallel test, the second test picks up gcov >> > data files from other process, which results in random pass. >> > Is it possible to not have these when fdo is supported? >> Hmm, typo: s/supported/not supported/. > > I don't see how this can happen: as you can see the .gcda file name > is unique for each test case. > > I think there may be problems with the perf.data, could add a postfix. > But you should only see that with real autofdo. Don't know. I only configured/built GCC on x86_64 with below command lines: $ ../gcc/configure --prefix=... --enable-languages=c,c++,fortran,go,java,lto,objc,obj-c++ --with-build-config=bootstrap-O3 --disable-werror $ make && make install Not sure what the status for autofdo is in this case. "make check -k" is stable for me, but "make check -k -j#" gives unstable result in tree-prof.exp tests. Anything I did wrong? Thanks, bin
Re: [PATCH 2/3] Run profile feedback tests with autofdo
> >> I haven't seen that. Unstable in what way? > > For GCC doesn't support FDO, it run below tests as you said: > > > > PASS: gcc.dg/tree-prof/20041218-1.c compilation, -g > > UNSUPPORTED: gcc.dg/tree-prof/20041218-1.c: Cannot run create_gcov > > --binary /data/work/trunk/build/gcc/testsuite/gcc/20041218-1.gcda > > > > Normally, it doesn't create gcov data file, thus the next test is > > unsupported. I guess in parallel test, the second test picks up gcov > > data files from other process, which results in random pass. > > Is it possible to not have these when fdo is supported? > Hmm, typo: s/supported/not supported/. I don't see how this can happen: as you can see the .gcda file name is unique for each test case. I think there may be problems with the perf.data, could add a postfix. But you should only see that with real autofdo. -Andi
Re: [PATCH 2/3] Run profile feedback tests with autofdo
On Thu, Jul 14, 2016 at 12:06 PM, Bin.Cheng wrote: > On Thu, Jul 14, 2016 at 11:55 AM, Andi Kleen wrote: >>> After this patch, I got below test results with command line: make >>> check-gcc RUNTESTFLAGS="tree-prof.exp" -k >> >> That is expected if you don't have autofdo. You would prefer to hide it? >> >>> Also I got unstable test result in tree-prof.exp if I run >>> aforementioned command line with -jnum (parallelly). Does the patch >>> has problem in parallel testing? >> >> I haven't seen that. Unstable in what way? > For GCC doesn't support FDO, it run below tests as you said: > > PASS: gcc.dg/tree-prof/20041218-1.c compilation, -g > UNSUPPORTED: gcc.dg/tree-prof/20041218-1.c: Cannot run create_gcov > --binary /data/work/trunk/build/gcc/testsuite/gcc/20041218-1.gcda > > Normally, it doesn't create gcov data file, thus the next test is > unsupported. I guess in parallel test, the second test picks up gcov > data files from other process, which results in random pass. > Is it possible to not have these when fdo is supported? Hmm, typo: s/supported/not supported/. Thanks, bin
Re: [PATCH 2/3] Run profile feedback tests with autofdo
On Thu, Jul 14, 2016 at 11:55 AM, Andi Kleen wrote: >> After this patch, I got below test results with command line: make >> check-gcc RUNTESTFLAGS="tree-prof.exp" -k > > That is expected if you don't have autofdo. You would prefer to hide it? > >> Also I got unstable test result in tree-prof.exp if I run >> aforementioned command line with -jnum (parallelly). Does the patch >> has problem in parallel testing? > > I haven't seen that. Unstable in what way? For GCC doesn't support FDO, it run below tests as you said: PASS: gcc.dg/tree-prof/20041218-1.c compilation, -g UNSUPPORTED: gcc.dg/tree-prof/20041218-1.c: Cannot run create_gcov --binary /data/work/trunk/build/gcc/testsuite/gcc/20041218-1.gcda Normally, it doesn't create gcov data file, thus the next test is unsupported. I guess in parallel test, the second test picks up gcov data files from other process, which results in random pass. Is it possible to not have these when fdo is supported? Thanks, bin > > -Andi
Re: [PATCH 2/3] Run profile feedback tests with autofdo
> After this patch, I got below test results with command line: make > check-gcc RUNTESTFLAGS="tree-prof.exp" -k That is expected if you don't have autofdo. You would prefer to hide it? > Also I got unstable test result in tree-prof.exp if I run > aforementioned command line with -jnum (parallelly). Does the patch > has problem in parallel testing? I haven't seen that. Unstable in what way? -Andi
Re: [PATCH 2/3] Run profile feedback tests with autofdo
On Thu, Jun 23, 2016 at 3:37 PM, Andi Kleen wrote:
> From: ak
>
> Extend the existing bprob and tree-prof tests to also run with autofdo.
> The test runtimes are really a bit too short for autofdo, but it's
> a reasonable sanity check.
>
> This only works natively for now.
>
> dejagnu doesn't seem to support a wrapper for unix tests, so I had
> to open code running these tests. That should be ok due to the
> native run restrictions.
>
> gcc/testsuite/:
>
> 2016-06-23 Andi Kleen
>
> * g++.dg/bprob/bprob.exp: Support autofdo.
> * g++.dg/tree-prof/tree-prof.exp: dito.
> * gcc.dg/tree-prof/tree-prof.exp: dito.
> * gcc.misc-tests/bprob.exp: dito.
> * gfortran.dg/prof/prof.exp: dito.
> * lib/profopt.exp: dito.
> * lib/target-supports.exp: Check for autofdo.
>
> git-svn-id: svn+ssh://gcc.gnu.org/svn/gcc/trunk@237732
> 138bc75d-0d04-0410-961f-82ee72b054a4
> ---
> gcc/testsuite/ChangeLog | 10
> gcc/testsuite/g++.dg/bprob/bprob.exp | 8 +++
> gcc/testsuite/g++.dg/tree-prof/tree-prof.exp | 8 +++
> gcc/testsuite/gcc.dg/tree-prof/tree-prof.exp | 8 +++
> gcc/testsuite/gcc.misc-tests/bprob.exp | 7 +++
> gcc/testsuite/gfortran.dg/prof/prof.exp | 7 +++
> gcc/testsuite/lib/profopt.exp| 82
> +++-
> gcc/testsuite/lib/target-supports.exp| 37 +
> 8 files changed, 164 insertions(+), 3 deletions(-)
>
> diff --git a/gcc/testsuite/ChangeLog b/gcc/testsuite/ChangeLog
> index 287baf6..55f8dbf 100644
> --- a/gcc/testsuite/ChangeLog
> +++ b/gcc/testsuite/ChangeLog
> @@ -1,3 +1,13 @@
> +2016-06-23 Andi Kleen
> +
> + * g++.dg/bprob/bprob.exp: Support autofdo.
> + * g++.dg/tree-prof/tree-prof.exp: dito.
> + * gcc.dg/tree-prof/tree-prof.exp: dito.
> + * gcc.misc-tests/bprob.exp: dito.
> + * gfortran.dg/prof/prof.exp: dito.
> + * lib/profopt.exp: dito.
> + * lib/target-supports.exp: Check for autofdo.
> +
> 2016-06-23 Martin Liska
>
> * gcc.dg/pr71619.c: New test.
> diff --git a/gcc/testsuite/g++.dg/bprob/bprob.exp
> b/gcc/testsuite/g++.dg/bprob/bprob.exp
> index d07..4818298 100644
> --- a/gcc/testsuite/g++.dg/bprob/bprob.exp
> +++ b/gcc/testsuite/g++.dg/bprob/bprob.exp
> @@ -53,6 +53,7 @@ if $tracelevel then {
>
> set profile_options "-fprofile-arcs"
> set feedback_options "-fbranch-probabilities"
> +set profile_wrapper ""
>
> # Main loop.
> foreach profile_option $profile_options feedback_option $feedback_options {
> @@ -65,4 +66,11 @@ foreach profile_option $profile_options feedback_option
> $feedback_options {
> }
> }
>
> +foreach src [lsort [glob -nocomplain $srcdir/$subdir/*.C]] {
> +if ![runtest_file_p $runtests $src] then {
> +continue
> +}
> +auto-profopt-execute $src
> +}
> +
> set PROFOPT_OPTIONS $bprob_save_profopt_options
> diff --git a/gcc/testsuite/g++.dg/tree-prof/tree-prof.exp
> b/gcc/testsuite/g++.dg/tree-prof/tree-prof.exp
> index 7a4b5cb..26ee0b3 100644
> --- a/gcc/testsuite/g++.dg/tree-prof/tree-prof.exp
> +++ b/gcc/testsuite/g++.dg/tree-prof/tree-prof.exp
> @@ -44,6 +44,7 @@ set PROFOPT_OPTIONS [list {}]
> # profile data.
> set profile_option "-fprofile-generate -D_PROFILE_GENERATE"
> set feedback_option "-fprofile-use -D_PROFILE_USE"
> +set profile_wrapper ""
>
> foreach src [lsort [glob -nocomplain $srcdir/$subdir/*.C]] {
> # If we're only testing specific files and this isn't one of them, skip
> it.
> @@ -53,4 +54,11 @@ foreach src [lsort [glob -nocomplain $srcdir/$subdir/*.C]]
> {
> profopt-execute $src
> }
>
> +foreach src [lsort [glob -nocomplain $srcdir/$subdir/*.C]] {
> +if ![runtest_file_p $runtests $src] then {
> +continue
> +}
> +auto-profopt-execute $src
> +}
> +
> set PROFOPT_OPTIONS $treeprof_save_profopt_options
> diff --git a/gcc/testsuite/gcc.dg/tree-prof/tree-prof.exp
> b/gcc/testsuite/gcc.dg/tree-prof/tree-prof.exp
> index 650ad8d..aaccf19 100644
> --- a/gcc/testsuite/gcc.dg/tree-prof/tree-prof.exp
> +++ b/gcc/testsuite/gcc.dg/tree-prof/tree-prof.exp
> @@ -44,6 +44,7 @@ set PROFOPT_OPTIONS [list {}]
> # profile data.
> set profile_option "-fprofile-generate -D_PROFILE_GENERATE"
> set feedback_option "-fprofile-use -D_PROFILE_USE"
> +set profile_wrapper ""
>
> foreach src [lsort [glob -nocomplain $srcdir/$subdir/*.c]] {
> # If we're only testing specific files and this isn't one of them, skip
> it.
> @@ -53,4 +54,11 @@ foreach src [lsort [glob -nocomplain $srcdir/$subdir/*.c]]
> {
> profopt-execute $src
> }
>
> +foreach src [lsort [glob -nocomplain $srcdir/$subdir/*.c]] {
> +if ![runtest_file_p $runtests $src] then {
> +continue
> +}
> +auto-profopt-execute $src
> +}
> +
> set PROFOPT_OPTIONS $treeprof_save_profopt_options
> diff --git a/gcc/testsuite/gcc.misc-tests/bprob.exp
> b/gcc/testsuite/gcc.misc-tests/bprob.exp
> index 52dcb1f..132bfe3 100644
>
Re: [PATCH 2/3] Run profile feedback tests with autofdo
On 06/22/2016 06:37 AM, Andi Kleen wrote: From: Andi Kleen Extend the existing bprob and tree-prof tests to also run with autofdo. The test runtimes are really a bit too short for autofdo, but it's a reasonable sanity check. This only works natively for now. dejagnu doesn't seem to support a wrapper for unix tests, so I had to open code running these tests. That should be ok due to the native run restrictions. gcc/testsuite/: 2016-06-22 Andi Kleen * g++.dg/bprob/bprob.exp: Support autofdo. * g++.dg/tree-prof/tree-prof.exp: dito. * gcc.dg/tree-prof/tree-prof.exp: dito. * gcc.misc-tests/bprob.exp: dito. * gfortran.dg/prof/prof.exp: dito. * lib/profopt.exp: dito. * lib/target-supports.exp: Check for autofdo. I'm generally OK with this as well. My only concern is that we get something sensible on targets which don't support autofdo and on systems without perf installed. ISTM the right result is for the autofdo versions of those tests to be UNSUPPORTED on both those situations. Can you confirm that's what happens in those two cases? If we do indeed get UNSUPPORTED for the autofdo versions of those tests in those cases, then this patch is OK as well. jeff
