Re: `make check -j` fails when building GNU Make from source

[Paul Smith]

On Mon, 2023-04-03 at 14:36 +0200, Alejandro Colomar wrote:
> -  Being part of the make targets, it would enable running them in
>    parallel, taking around 1/4th the time it takes now.

Up until this morning this could not work because all the tests run in
the same directory and they create temporary files, etc. that would
interfere with each other.

Since I implemented the "each test runs in a different directory"
today, it COULD work.  It would still require major changes to the test
driver, which is implemented to assume that it "owns" all the tests
(when you start it it cleans up everything from all previous runs etc.)

> -  If you run them several times, you only get rerun the ones that
>    failed (especially if you run with -k) or didn't yet run.  This
>    allows interrupting the job and restarting (as I did).

This is the point I already addressed: it's highly unlikely that you
run the tests "several times" without making some change that means you
should restart them all anyway.  The only time that would happen is if
(a) you killed them partway through like you did, which is extremely
unusual and I don't think supporting that situation is worth spending a
lot of effort on, or (b) you're developing a single test and want to
re-run just that one, in which case you can just invoke it directly
(you probably want to do this anyway since when you do rebuild make you
don't want to re-run everything).

Re: `make check -j` fails when building GNU Make from source

[Alejandro Colomar]

Hi Paul,

On 4/3/23 14:26, Paul Smith wrote:
> On Sun, 2023-04-02 at 18:14 +0200, Alejandro Colomar wrote:
>> I wonder if you could use the Makefile to run the tests, rather
>> than calling a script from a .PHONY target that runs them all the
>> time.  Why not run them only once?  You can touch empty files
>> when a test succeeds to make make(1) happy (I usually call them
>> *.touch files).
> 
> I don't quite understand what you mean by "why not run them only once".
> They are run only once...?
> 
> Maybe you mean, if (a) the make binary hasn't been rebuilt and (b) the
> test script hasn't been changed, then we wouldn't need to re-run the
> test?

Yes, I meant that.

>  Perhaps, but realistically I'm not sure it's worth the effort to
> make this change because almost the only time we ever run these tests
> is when we've rebuilt the make binary, and it's easy enough to run just
> a single test if we want to.

A few beneifts:

-  Being part of the make targets, it would enable running them in
   parallel, taking around 1/4th the time it takes now.

-  If you run them several times, you only get rerun the ones that
   failed (especially if you run with -k) or didn't yet run.  This
   allows interrupting the job and restarting (as I did).

But of course, I'm not going to tell you all of the benefits that a
Makefile has over a shell script.  :)

Cheers,
Alex

-- 

GPG key fingerprint: A9348594CE31283A826FBDD8D57633D441E25BB5


OpenPGP_signature
Description: OpenPGP digital signature

Re: `make check -j` fails when building GNU Make from source

[Paul Smith]

On Sun, 2023-04-02 at 18:14 +0200, Alejandro Colomar wrote:
> I wonder if you could use the Makefile to run the tests, rather
> than calling a script from a .PHONY target that runs them all the
> time.  Why not run them only once?  You can touch empty files
> when a test succeeds to make make(1) happy (I usually call them
> *.touch files).

I don't quite understand what you mean by "why not run them only once".
They are run only once...?

Maybe you mean, if (a) the make binary hasn't been rebuilt and (b) the
test script hasn't been changed, then we wouldn't need to re-run the
test?  Perhaps, but realistically I'm not sure it's worth the effort to
make this change because almost the only time we ever run these tests
is when we've rebuilt the make binary, and it's easy enough to run just
a single test if we want to.

Re: `make check -j` fails when building GNU Make from source

[Alejandro Colomar]

On 4/2/23 16:15, Paul Smith wrote:
> On Sun, 2023-04-02 at 16:13 +0200, Alejandro Colomar wrote:
>> I can reproduce it by running `make check`, interrupting it
>> at this specific point, and then running again `make check`
>> (no -j needed):
> 
> This is a known limitation with the test suite.

I wonder if you could use the Makefile to run the tests, rather
than calling a script from a .PHONY target that runs them all the
time.  Why not run them only once?  You can touch empty files
when a test succeeds to make make(1) happy (I usually call them
*.touch files).


-- 

GPG key fingerprint: A9348594CE31283A826FBDD8D57633D441E25BB5


OpenPGP_signature
Description: OpenPGP digital signature

Re: `make check -j` fails when building GNU Make from source

[Paul Smith]

On Sun, 2023-04-02 at 16:13 +0200, Alejandro Colomar wrote:
> I can reproduce it by running `make check`, interrupting it
> at this specific point, and then running again `make check`
> (no -j needed):

This is a known limitation with the test suite.

Re: `make check -j` fails when building GNU Make from source

[Alejandro Colomar]

Hi Paul,

On 4/2/23 16:04, Paul Smith wrote:
> On Sun, 2023-04-02 at 14:52 +0200, Alejandro Colomar wrote:
>> If I build make from source and run the checks in parallel, some
>> fail.  Is this expected, or is it a bug in the Makefile?
> 
> The test suite is invoked as a single target, so there's no way that
> enabling parallelism could impact it.  Also the test suite tries to be
> careful about shielding the internal invocations of make from
> environmental factors like user settings of MAKEFLAGS and GNUMAKEFLAGS
> etc.
> 
> Indeed I tried "make check -j" and got no failures.
> 
> Based on the diffs it appears make was able to find files like "foo"
> and "bar" when it didn't expect them to exist.

Ahhh, then it was probably because I started with `make check`,
and when I saw it was lasting more than a couple of seconds, I
killed (^C) it and rerun with -j.

>  I'm not really sure how
> that would happen but if you can figure it out I would be interested to
> know.

I can reproduce it by running `make check`, interrupting it
at this specific point, and then running again `make check`
(no -j needed):

[...]
features/mult_targets ... ok (2 passed)
features/order_only . ok (10 passed)
features/output-sync  ^Cmake[2]: *** 
[Makefile:1550: check-regression] Interrupt
make[1]: *** [Makefile:1283: check-am] Interrupt
make: *** [Makefile:985: check-recursive] Interrupt


Maybe other interruption points also reproduce it; some others
don't.


>  If you run:
> 
>   cd tests
>   ./run_make_tests -make ../make features/escape
> 
> does it fail?

Nope.  And not with some other tests I tried.

Cheers,
Alex

-- 

GPG key fingerprint: A9348594CE31283A826FBDD8D57633D441E25BB5


OpenPGP_signature
Description: OpenPGP digital signature

Re: `make check -j` fails when building GNU Make from source

[Paul Smith]

On Sun, 2023-04-02 at 14:52 +0200, Alejandro Colomar wrote:
> If I build make from source and run the checks in parallel, some
> fail.  Is this expected, or is it a bug in the Makefile?

The test suite is invoked as a single target, so there's no way that
enabling parallelism could impact it.  Also the test suite tries to be
careful about shielding the internal invocations of make from
environmental factors like user settings of MAKEFLAGS and GNUMAKEFLAGS
etc.

Indeed I tried "make check -j" and got no failures.

Based on the diffs it appears make was able to find files like "foo"
and "bar" when it didn't expect them to exist.  I'm not really sure how
that would happen but if you can figure it out I would be interested to
know.  If you run:

  cd tests
  ./run_make_tests -make ../make features/escape

does it fail?

`make check -j` fails when building GNU Make from source

[Alejandro Colomar]

Hi,

If I build make from source and run the checks in parallel, some fail.
Is this expected, or is it a bug in the Makefile?

Cheers,
Alex

-- 

GPG key fingerprint: A9348594CE31283A826FBDD8D57633D441E25BB5

makeerror-4.4.1-x86_64-pc-linux-gnu-t3wj.tar.gz
Description: application/gzip


OpenPGP_signature
Description: OpenPGP digital signature