Re: Principles of the C99 testsuite conversion

[Florian Weimer]

These changes are now in, for i686-linux-gnu, powerpc64le-linux-gnu,
x86_64-linux-gnu.  For aarch64-linux-gnu, there's one change that would
benefit from maintainer review:

  [PATCH] aarch64: Call named function in gcc.target/aarch64/aapcs64/ice_1.c
  


(On aarch64-linux-gnu, we also need a libgcc fix, but that is a
different matter.)

Thanks,
Florian

Re: Principles of the C99 testsuite conversion

[Florian Weimer]

* Florian Weimer:

> * C89 test cases that appear to make explicit use of C89-only features.
>   I plan to add /* { dg-additional-options "-std=gnu89" } */ to those
>   (or add -std=gnu89 to an existing dg-options line).  These fixes can
>   be submitted early.

Example:



>
> * Overly reduced test cases where I can verify that the less-reduced
>   test case still reproduces the bug.  Can be submitted early.



> * Similar test cases where I cannot verify that the test case is still
>   reproducing the bug, but where I assume that the unreduction is
>   harmless as far as the test objective is concerned.  Can be submitted
>   early (but should be in a separate patch from the previous one).



Thanks,
Florian

Re: Principles of the C99 testsuite conversion

[Florian Weimer]

* Jeff Law:

> On 10/11/23 04:39, Florian Weimer wrote:
>> I've started to look at what it is required to convert the testsuite to
>> C99 (without implicit ints, without implicit function declarations, and
>> a few other legacy language features).

> I bet those older tests originating from c-torture will be a bit
> painful.  Torbjorn liked having them minimized, to the point of
> squashing out nearly everything he considered extraneous.  I'd bet
> many of those older tests are going to need lots of changes.

We can use them to exercise the old C89 features and build with
-std=gnu89.  That may not have been the original intent, but I think
it's what these tests are testing today (also noted by Richard
Earnshaw).

>> * Overly reduced test cases where I can verify that the less-reduced
>>test case still reproduces the bug.  Can be submitted early.

> Seems like this could be a huge time sink, but that's your call.

Well, if the verification means I can commit these fixes as obvious, it
may be worth it …

>> * Similar test cases where I cannot verify that the test case is
>> still
>>reproducing the bug, but where I assume that the unreduction is
>>harmless as far as the test objective is concerned.  Can be submitted
>>early (but should be in a separate patch from the previous one).
> I suspect many of the older tests will fall into this bucket.

Yes, for the newer test cases, verification is generally possible.

One ongoing issue is that gcc/testsuite/gcc.c-torture/ appears to be
used for new tests, but this subtree (or at least parts of it) defaults
to -w, so if you don't use dg-options, can easily check in something
that has a lingering C99 compatibility issue today.  Here's an example
from a few weeks ago:

diff --git a/gcc/testsuite/gcc.c-torture/execute/pr111331-1.c 
b/gcc/testsuite/gcc.c-torture/execute/pr111331-1.c
index 4c7f4fdbaa9..05a6db6dce6 100644
--- a/gcc/testsuite/gcc.c-torture/execute/pr111331-1.c
+++ b/gcc/testsuite/gcc.c-torture/execute/pr111331-1.c
@@ -10,7 +10,7 @@ int c(int d, int e, int f) {
 int main() {
   int g = -1;
   a = c(b + 30, 29, g + 29);
-  volatile t = a;
+  volatile int t = a;
   if (t != 28)
 __builtin_abort();
   return 0;

Thanks,
Florian

Re: Principles of the C99 testsuite conversion

[Jakub Jelinek]

On Wed, Oct 11, 2023 at 08:17:49AM -0600, Jeff Law wrote:
> 
> 
> On 10/11/23 08:10, Richard Earnshaw (lists) wrote:
> > On 11/10/2023 14:56, Jeff Law wrote:
> > > 
> > > 
> > > On 10/11/23 04:39, Florian Weimer wrote:
> > > > I've started to look at what it is required to convert the testsuite to
> > > > C99 (without implicit ints, without implicit function declarations, and
> > > > a few other legacy language features).
> > > I bet those older tests originating from c-torture will be a bit painful. 
> > >  Torbjorn liked having them minimized, to the point of squashing out 
> > > nearly everything he considered extraneous.  I'd bet many of those older 
> > > tests are going to need lots of changes.
> > > 
> > 
> > I've often wondered just how much of the original c-torture suite is still 
> > relevant today.  Most of those tests were written at a  time when the 
> > compiler expanded tree directly into RTL and I suspect that today the tests 
> > never get even close to tickling the original bug they were intended to 
> > validate.
> I'm sure it's a mixed bag.  Some I do see pop up regularly during
> development testing.
> 
> The real way to try and answer that question would be with gcov. Something
> like take an existing coverage report, run test and see if it added any
> additional coverage.  If not, flag it as potentially irrelevant.  Do this on
> x86 or aarch64.
> 
> Given that list of potentially irrelevant tests, then look at them from a
> target standpoint, potentially testing with the same methodology on several
> targets, perhaps a standardized set meant to cover 16->64 bit targets,
> big/little endian and a few other key features.

I think over the years we've seen tons of cases where a test written for one
problem at some point is able to trigger some completely different problem a
few years ago.  So, I don't think most of the tests are irrelevant.

Jakub

Re: Principles of the C99 testsuite conversion

[Jeff Law]

On 10/11/23 08:10, Richard Earnshaw (lists) wrote:

On 11/10/2023 14:56, Jeff Law wrote:



On 10/11/23 04:39, Florian Weimer wrote:

I've started to look at what it is required to convert the testsuite to
C99 (without implicit ints, without implicit function declarations, and
a few other legacy language features).

I bet those older tests originating from c-torture will be a bit painful.  
Torbjorn liked having them minimized, to the point of squashing out nearly 
everything he considered extraneous.  I'd bet many of those older tests are 
going to need lots of changes.



I've often wondered just how much of the original c-torture suite is still 
relevant today.  Most of those tests were written at a  time when the compiler 
expanded tree directly into RTL and I suspect that today the tests never get 
even close to tickling the original bug they were intended to validate.
I'm sure it's a mixed bag.  Some I do see pop up regularly during 
development testing.


The real way to try and answer that question would be with gcov. 
Something like take an existing coverage report, run test and see if it 
added any additional coverage.  If not, flag it as potentially 
irrelevant.  Do this on x86 or aarch64.


Given that list of potentially irrelevant tests, then look at them from 
a target standpoint, potentially testing with the same methodology on 
several targets, perhaps a standardized set meant to cover 16->64 bit 
targets, big/little endian and a few other key features.


jeff

Re: Principles of the C99 testsuite conversion

[Richard Earnshaw (lists)]

On 11/10/2023 14:56, Jeff Law wrote:
> 
> 
> On 10/11/23 04:39, Florian Weimer wrote:
>> I've started to look at what it is required to convert the testsuite to
>> C99 (without implicit ints, without implicit function declarations, and
>> a few other legacy language features).
> I bet those older tests originating from c-torture will be a bit painful.  
> Torbjorn liked having them minimized, to the point of squashing out nearly 
> everything he considered extraneous.  I'd bet many of those older tests are 
> going to need lots of changes.
> 

I've often wondered just how much of the original c-torture suite is still 
relevant today.  Most of those tests were written at a  time when the compiler 
expanded tree directly into RTL and I suspect that today the tests never get 
even close to tickling the original bug they were intended to validate.

R.

Re: Principles of the C99 testsuite conversion

[Jeff Law]

On 10/11/23 04:39, Florian Weimer wrote:

I've started to look at what it is required to convert the testsuite to
C99 (without implicit ints, without implicit function declarations, and
a few other legacy language features).
I bet those older tests originating from c-torture will be a bit 
painful.  Torbjorn liked having them minimized, to the point of 
squashing out nearly everything he considered extraneous.  I'd bet many 
of those older tests are going to need lots of changes.




I plan to bundle a bunch of fixes together and submit patches
incrementally.  So far, I have identified the following change
categories:

* C89 test cases that appear to make explicit use of C89-only features.
   I plan to add /* { dg-additional-options "-std=gnu89" } */ to those
   (or add -std=gnu89 to an existing dg-options line).  These fixes can
   be submitted early.

Sounds quite sensible.



* Overly reduced test cases where I can verify that the less-reduced
   test case still reproduces the bug.  Can be submitted early.

Seems like this could be a huge time sink, but that's your call.



* Similar test cases where I cannot verify that the test case is still
   reproducing the bug, but where I assume that the unreduction is
   harmless as far as the test objective is concerned.  Can be submitted
   early (but should be in a separate patch from the previous one).

I suspect many of the older tests will fall into this bucket.



* Test cases which have to build with -fpermissive.  These include
   expected-warnings tests, and over-reduced test cases where unreducing
   might conceivably interfere with the test objective.  These fixes have
   to wait until -fpermissive for C lands (or they could wait until the
   first relevant use of permerror).

Also seems quite sensible.



* Test cases which check (for example) error locations from diagnostics
   that used be warnings, but that are now errors, and other test cases
   where it seems to make sense to test for errors now.  Here I plan to
   change dg-warning into dg-error as appropriate.  These changes would
   be submitted along the patch that introduces the relevant permerror
   calls.  In some cases, I plan to duplicate tests into warning variants
   (now with -fpermissive) and error variants (with dg-warning changed to
   dg-error).

Sure, this seems quite sensible as well.



* There's the matter of target-specific tests that I don't know how to
   cover.  I can probably run the testsuite on x86-64, POWER9 and
   AArch64, but beyond that, I'm pretty much out of options.  I will have
   to rely on help from the community, and perhaps target maintainers
   adjusting their tests after the permerror changes according to the
   principles above.
The good news is we'll see these pop in the tester for all those 
embedded targets.  So I'm not too concerned here.


Seems like a lot of (long needed) maintenance work.   I can't drive it, 
but I can certainly help with identification and fixing of the embedded 
targets when we get to that point.


jeff

Principles of the C99 testsuite conversion

[Florian Weimer]

I've started to look at what it is required to convert the testsuite to
C99 (without implicit ints, without implicit function declarations, and
a few other legacy language features).

I plan to bundle a bunch of fixes together and submit patches
incrementally.  So far, I have identified the following change
categories:

* C89 test cases that appear to make explicit use of C89-only features.
  I plan to add /* { dg-additional-options "-std=gnu89" } */ to those
  (or add -std=gnu89 to an existing dg-options line).  These fixes can
  be submitted early.

* Overly reduced test cases where I can verify that the less-reduced
  test case still reproduces the bug.  Can be submitted early.

* Similar test cases where I cannot verify that the test case is still
  reproducing the bug, but where I assume that the unreduction is
  harmless as far as the test objective is concerned.  Can be submitted
  early (but should be in a separate patch from the previous one).

* Test cases which have to build with -fpermissive.  These include
  expected-warnings tests, and over-reduced test cases where unreducing
  might conceivably interfere with the test objective.  These fixes have
  to wait until -fpermissive for C lands (or they could wait until the
  first relevant use of permerror).

* Test cases which check (for example) error locations from diagnostics
  that used be warnings, but that are now errors, and other test cases
  where it seems to make sense to test for errors now.  Here I plan to
  change dg-warning into dg-error as appropriate.  These changes would
  be submitted along the patch that introduces the relevant permerror
  calls.  In some cases, I plan to duplicate tests into warning variants
  (now with -fpermissive) and error variants (with dg-warning changed to
  dg-error).

* There's the matter of target-specific tests that I don't know how to
  cover.  I can probably run the testsuite on x86-64, POWER9 and
  AArch64, but beyond that, I'm pretty much out of options.  I will have
  to rely on help from the community, and perhaps target maintainers
  adjusting their tests after the permerror changes according to the
  principles above.

Overall, I'm quite happy with the -fpermissive and permerror behavior I
see with Jason's patch so far.  As far as I can tell, permerrors are
flagged in system headers, which is what we want from a distribution
perspective at least.  I could get -std=gnu89 -pedantic-errors to work
(it should still flag these things as errors, as before), and -std=gnu89
-fno-permissive works out of the box, too.

(Beyond the testsuite, we will need some libgcc fixes.  I've started to
submit patches for that, but they have been bogged down in discussions
how to best fix the broken code unfortunately.)

Thanks,
Florian