On Thu, 2014-07-03 at 22:14 +0200, Jakub Jelinek wrote: > On Thu, Jul 03, 2014 at 10:04:35PM +0200, Mark Wielaard wrote: > > On Thu, 2014-07-03 at 21:52 +0200, Richard Biener wrote: > > > On July 3, 2014 8:38:14 PM CEST, Jakub Jelinek <ja...@redhat.com> wrote: > > > >On Thu, Jul 03, 2014 at 08:37:07PM +0200, Richard Biener wrote: > > > >> Well, simply removing the regression testing for LTO is a > > > >maintainance nightmare as well. > > > >> > > > >> The guality testsuite is very noisy anyway with all the xfail and > > > >xpass. > > > > > > > >Let's keep it as is then? > > > > > > That works for me. > > > > I don't find that very satisfactory. I want to add more guality tests, > > but the fact that they are unreliable and by default introduce even more > > FAILs when lto is enabled makes that not very attractive. I do like > > Jakub's suggestion to disable the guality tests be run with lto by > > default, but provide an environment variable to enable them for those > > that want to try them anyway. Shall I implement that? > > They aren't that unrealiable (at least, if people committing patches don't > ignore regressions in there). Just one should diff contrib/test_summary > output from earlier builds to the latest, that way it is clear what is a > regression and what is not.
The are much more unreliable than any other test. With guality.exp disabled one can just eyeball the results and investigate new FAILS. There are only a handful. When you include guality.exp you can easily get the impression the gcc testsuite is really bad (and it isn't!) And the problem is that it makes adding new tests a pain. See my new tests, they introduce new FAILs because LTO is enabled by default for guality.exp at the moment. It just results in a slow increase of FAILs that people have to ignore. And I am afraid that will just result in people missing real regressions. I don't mind if there is active work to fix LTO DWARF debuginfo generation issues and the guality.exp LTO failures will soon disappear, but if there is no active work on reducing the amount of failures and introducing new guality.exp testcases will keep adding more FAILs I think we are much better off disabling them for now. Thanks, Mark
