Re: "make -jN" requires mechanical changes to a Makefile [SOLVED]

[Henrik Carlqvist]

On Mon, 14 Sep 2020 12:15:58 +0200
Bruno Haible  wrote:

> Henrik Carlqvist wrote:
> > 2) Don't mention some of the extra targets:
> > ===
> > all : copy1
> >  
> > copy1: Makefile
> > install -c -m 644 Makefile copy1
> > install -c -m 644 Makefile copy2
> > install -c -m 644 Makefile copy3
> > install -c -m 644 Makefile copy4
> > ===
> 
> Fails (D) and (E). => Not a solution to the problem.
> 
> Bruno
> 

Ok, assuming that solution 1 did not meet your requirements, lets give it
another shot:

===
COPIES=copy1 copy2 copy3 copy4

.INTERMEDIATE: dummy

all: $(COPIES)

$(COPIES): %: dummy
$(RM) $<

dummy: Makefile
install -c -m 644 Makefile copy1
install -c -m 644 Makefile copy2
install -c -m 644 Makefile copy3
install -c -m 644 Makefile copy4
touch $@
===

I create an extra file "dummy" for a short time in an attempt to live up to
all your requirements.

regards Henrik

Re: "make -jN" requires mechanical changes to a Makefile [SOLVED]

[Bruno Haible]

Henrik Carlqvist wrote:
> 2) Don't mention some of the extra targets:
> ===
> all : copy1
>  
> copy1: Makefile
>   install -c -m 644 Makefile copy1
>   install -c -m 644 Makefile copy2
>   install -c -m 644 Makefile copy3
>   install -c -m 644 Makefile copy4
> ===

Fails (D) and (E). => Not a solution to the problem.

Bruno

Re: "make -jN" requires mechanical changes to a Makefile [SOLVED]

[Henrik Carlqvist]

On Sun, 13 Sep 2020 20:07:27 +0100
Bruno Haible wrote:
> Continuing this thread from May 2019
> :
> The problem was:
> 
>   How can a rule that generates multiple files be formulated so
>   that it works with parallel make?
> 
> For example, a rule that invokes bison, or a rule that invokes
> a different Makefile. For simplicity, here, use a rule that
> creates 4 files copy1, copy2, copy3, copy4.
> 
> ===
> all : copy1 copy2 copy3 copy4
> 
> copy1 copy2 copy3 copy4: Makefile
>   install -c -m 644 Makefile copy1
>   install -c -m 644 Makefile copy2
>   install -c -m 644 Makefile copy3
>   install -c -m 644 Makefile copy4
> ===

I would say there are two obvious solutions to make this work with parallel
make:


1) Allow the targets to be tuilt in paralllel:
===
all : copy1 copy2 copy3 copy4
 
copy1 copy2 copy3 copy4: Makefile
install -c -m 644 Makefile $@
===

However, this solution 1 will not work for a single command that generates
multiple files, so we might need a solution 2:

2) Don't mention some of the extra targets:
===
all : copy1
 
copy1: Makefile
install -c -m 644 Makefile copy1
install -c -m 644 Makefile copy2
install -c -m 644 Makefile copy3
install -c -m 644 Makefile copy4
===

regards Henrik

Re: "make -jN" requires mechanical changes to a Makefile [SOLVED]

[Bruno Haible]

Hi Paul,

> There is a straightforward and portable way to do this even with
> traditional make, it's just not as nice (but, nicer than changing all
> the recipes to use test IMO! :)).
> 
> If you have a rule like this:
> 
> ... : 
>   
> 
> where  generates all targets with one invocation, then your
> best bet is to modify this rule into two rules:
> 
> ... : .sentinel ;
> 
>   .sentinel: 
>   
>   @touch $@
> 
> Note, it's critical to include the semicolon here.  Or if you prefer to
> be more explicit you can write something like:
> 
> ... : .sentinel
>   : do nothing
> 
> This will work properly.

Thanks for showing this idiom.

I think this can be improved a bit: Instead of 'touch .sentinel' I would
use 'echo > .sentinel'. Because if you use 'touch .sentinel' and the build
directory is on an NFS mount and the clock of the NFS server is ahead the
clock of localhost (I have seen this quite frequently), the timestamp of
.sentinel will be behind the timestamp of the newest , and thus
another "make" run will execute the rule again.

Still, I don't like these '.sentinel' or stamp files, because they
add complexity:
  * The developer has to decide whether the stamp file should go into
the build dir or $(srcdir). The general rule should be that if
 uses only build tools that are guaranteed to be present,
then   ... and the stamp file should go into the build dir,
whereas if they use less common tools (e.g. bison), they should
all go into $(srcdir).
  * The automake dist target needs to be adjusted to include this stamp
file.
  * Version control (.gitignore) needs to be adjusted to include this
stamp file.

I'll use the rule with a phony target and 'test ... -ot ...', unless you
find a problem with it (other than the potential portability problem
redarding 'test').

Bruno

Re: "make -jN" requires mechanical changes to a Makefile [SOLVED]

[Paul Smith]

On Sun, 2020-09-13 at 15:08 -0400, Paul Smith wrote:
> ... : .sentinel ;
> 
>   .sentinel: 
>   
>   @touch $@

Just to be clear, you don't have to use ".sentinel" you can use any
target name, and obviously you must use a different name for each
"grouping" of targets that needs to be built.

Re: "make -jN" requires mechanical changes to a Makefile [SOLVED]

[Bruno Haible]

> ===
> all : copy1 copy2 copy3 copy4
> 
> copy1: Makefile
>   { test -f copy1 && test ! copy1 -ot Makefile; } || { rm -f copy4; 
> $(MAKE) copies; }
> copy2: copy1
>   { test -f copy2 && test ! copy2 -ot copy1; } || { rm -f copy4; $(MAKE) 
> copies; }
> copy3: copy2
>   { test -f copy3 && test ! copy3 -ot copy2; } || { rm -f copy4; $(MAKE) 
> copies; }
> copy4: copy3
>   { test -f copy4 && test ! copy4 -ot copy3; } || { rm -f copy4; $(MAKE) 
> copies; }
> 
> copies:
>   install -c -m 644 Makefile copy1
>   install -c -m 644 Makefile copy2
>   install -c -m 644 Makefile copy3
>   install -c -m 644 Makefile copy4
> .PHONY: copies
> ===
> 
> This solution fulfils all the requirements.

It can be simplified a bit. And remove verbosity:

===
all : copy1 copy2 copy3 copy4

copy1: Makefile
@{ test -f copy1 && test ! copy1 -ot Makefile; } || $(MAKE) copies
copy2: copy1
@{ test -f copy2 && test ! copy2 -ot copy1; } || $(MAKE) copies
copy3: copy2
@{ test -f copy3 && test ! copy3 -ot copy2; } || $(MAKE) copies
copy4: copy3
@{ test -f copy4 && test ! copy4 -ot copy3; } || $(MAKE) copies

copies:
install -c -m 644 Makefile copy1
install -c -m 644 Makefile copy2
install -c -m 644 Makefile copy3
install -c -m 644 Makefile copy4
.PHONY: copies
===

Re: "make -jN" requires mechanical changes to a Makefile [SOLVED]

[Paul Smith]

On Sun, 2020-09-13 at 20:55 +0200, Bruno Haible wrote:
>   How can a rule that generates multiple files be formulated so
>   that it works with parallel make?
> 
> For example, a rule that invokes bison, or a rule that invokes
> a different Makefile. For simplicity, here, use a rule that
> creates 4 files copy1, copy2, copy3, copy4.

Sorry, I think the last time this came up I got side-tracked by the
fact that the example was using install.

There is a straightforward and portable way to do this even with
traditional make, it's just not as nice (but, nicer than changing all
the recipes to use test IMO! :)).

If you have a rule like this:

... : 
  

where  generates all targets with one invocation, then your
best bet is to modify this rule into two rules:

... : .sentinel ;

  .sentinel: 
  
  @touch $@

Note, it's critical to include the semicolon here.  Or if you prefer to
be more explicit you can write something like:

... : .sentinel
  : do nothing

This will work properly.

Re: "make -jN" requires mechanical changes to a Makefile [SOLVED]

[Howard Chu]

Bruno Haible wrote:
> Continuing this thread from May 2019
> :
> The problem was:
> 
>   How can a rule that generates multiple files be formulated so
>   that it works with parallel make?
> 
> For example, a rule that invokes bison, or a rule that invokes
> a different Makefile. For simplicity, here, use a rule that
> creates 4 files copy1, copy2, copy3, copy4.
> 
> ===
> all : copy1 copy2 copy3 copy4
> 
> copy1 copy2 copy3 copy4: Makefile
>   install -c -m 644 Makefile copy1
>   install -c -m 644 Makefile copy2
>   install -c -m 644 Makefile copy3
>   install -c -m 644 Makefile copy4
> ===
> 
> Unfortunately, with "make -j8", it invokes the rule multiple times.
> 
> It is possible to change this Makefile so that
>   (A) "rm -f copy?; make" executes the rule once.
>   (B) "rm -f copy?; make -j8" executes the rule once as well.
>   (C) After "make", another "make" just prints "Nothing to be done for 'all'."
>   (D) After removing one of the files copy?, "make" executes the rule once.
>   (This covers also the case of pressing Ctrl-C during "make", then
>   doing "make" again.)
>   (E) After replacing one of the files copy? with a file that is older than
>   Makefile, "make" executes the rule once.
> 
> There are three possibilities:

You're thinking about this the wrong way. Your set of commands is inherently
serial, therefore you need to write serial dependencies.

===
all: copy1 copy2 copy3 copy4

copy1: Makefile
install -c -m 644 Makefile $@
copy2: copy1
install -c -m 644 Makefile $@
copy3: copy2
install -c -m 644 Makefile $@
copy4: copy3
install -c -m 644 Makefile $@
===

This satisfies all your conditions, because it is inherently correct.

More on writing proper parallel Makefiles here http://highlandsun.com/hyc/#Make

-- 
  -- Howard Chu
  CTO, Symas Corp.   http://www.symas.com
  Director, Highland Sun http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/

Re: "make -jN" requires mechanical changes to a Makefile [SOLVED]

[Bruno Haible]

Continuing this thread from May 2019
:
The problem was:

  How can a rule that generates multiple files be formulated so
  that it works with parallel make?

For example, a rule that invokes bison, or a rule that invokes
a different Makefile. For simplicity, here, use a rule that
creates 4 files copy1, copy2, copy3, copy4.

===
all : copy1 copy2 copy3 copy4

copy1 copy2 copy3 copy4: Makefile
install -c -m 644 Makefile copy1
install -c -m 644 Makefile copy2
install -c -m 644 Makefile copy3
install -c -m 644 Makefile copy4
===

Unfortunately, with "make -j8", it invokes the rule multiple times.

It is possible to change this Makefile so that
  (A) "rm -f copy?; make" executes the rule once.
  (B) "rm -f copy?; make -j8" executes the rule once as well.
  (C) After "make", another "make" just prints "Nothing to be done for 'all'."
  (D) After removing one of the files copy?, "make" executes the rule once.
  (This covers also the case of pressing Ctrl-C during "make", then
  doing "make" again.)
  (E) After replacing one of the files copy? with a file that is older than
  Makefile, "make" executes the rule once.

There are three possibilities:


(I) Assuming GNU make >= 4.3, the "Grouped explicit target" syntax does it.
Thanks to Kaz Kylheku and Paul Smith for having added this.

===
all : copy1 copy2 copy3 copy4

copy1 copy2 copy3 copy4 &: Makefile
install -c -m 644 Makefile copy1
install -c -m 644 Makefile copy2
install -c -m 644 Makefile copy3
install -c -m 644 Makefile copy4
===

But it will take a number of years until we can assume that all 'make'
programs that we care about support this syntax.


(II) It is possible to turn off parallel make. This does it:

===
all : copy1 copy2 copy3 copy4

copy1 copy2 copy3 copy4 : Makefile
install -c -m 644 Makefile copy1
install -c -m 644 Makefile copy2
install -c -m 644 Makefile copy3
install -c -m 644 Makefile copy4

# This Makefile contains rules which don't work with parallel make.
# So, turn off parallel execution in this Makefile.
.NOTPARALLEL:
===

But some people who want to minimize wall-clock execution time of
their builds may not like it.


(III) Use portable make syntax and still allow parallel make.

This is a bit harder. My solution is to first analyze in which order
the rule will generate the various files and thus what the expected
timestamp order is. In this case, it is:
  Makefile <= copy1 <= copy2 <= copy3 <= copy4

First some attempts that don't work:

===
all : copy1 copy2 copy3 copy4

copy1: Makefile
install -c -m 644 Makefile copy1
install -c -m 644 Makefile copy2
install -c -m 644 Makefile copy3
install -c -m 644 Makefile copy4

copy2 copy3 copy4: copy1
===

Fails (E).

===
all : copy1 copy2 copy3 copy4

copy4: Makefile
install -c -m 644 Makefile copy1
install -c -m 644 Makefile copy2
install -c -m 644 Makefile copy3
install -c -m 644 Makefile copy4

copy1 copy2 copy3: copy4
===

Fails (E) as well.

===
all : copy1 copy2 copy3 copy4

copy4: Makefile
install -c -m 644 Makefile copy1
install -c -m 644 Makefile copy2
install -c -m 644 Makefile copy3
install -c -m 644 Makefile copy4

copy1: copy4
test -f copy1 || { rm -f copy4; $(MAKE) copy4; }
copy2: copy4
test -f copy2 || { rm -f copy4; $(MAKE) copy4; }
copy3: copy4
test -f copy3 || { rm -f copy4; $(MAKE) copy4; }
===

Fails (E) for copy1, ..., copy3.
And fails (C), i.e. clutters the "make" output.

===
all : copy1 copy2 copy3 copy4

copy1: Makefile
{ test -f copy1 && test ! copy1 -ot Makefile; } || { rm -f copy4; 
$(MAKE) copies; }
copy2: copy1
{ test -f copy2 && test ! copy2 -ot copy1; } || { rm -f copy4; $(MAKE) 
copies; }
copy3: copy2
{ test -f copy3 && test ! copy3 -ot copy2; } || { rm -f copy4; $(MAKE) 
copies; }
copy4: copy3
{ test -f copy4 && test ! copy4 -ot copy3; } || { rm -f copy4; $(MAKE) 
copies; }

copies:
install -c -m 644 Makefile copy1
install -c -m 644 Makefile copy2
install -c -m 644 Makefile copy3
install -c -m 644 Makefile copy4
.PHONY: copies
===

This solution fulfils all the requirements.

Can we use 'test FILE1 -ot FILE2'?
- POSIX 'test' does not support this option.
 

Re: "make -jN" requires mechanical changes to a Makefile

[Edward Welbourne]

> Howard Chu wrote:
> > >> Example with one rule creating 4 files:
> > >>
> > >> all : copy1
> > >>
> > >> copy1: Makefile
> > >> install -c -m 644 Makefile copy1
> > >> install -c -m 644 Makefile copy2
> > >> install -c -m 644 Makefile copy3
> > >> install -c -m 644 Makefile copy4

For this (toy example) case,

# We want to create all 4 copies
all: copy1 copy2 copy3 copy4
# Each of the copies depends on the Makefile:
copy1 copy2 copy3 copy4: Makefile
install -c -m 644 $< $@

should do the trick.  The rule means: "To update any one of the files
copy1...copy4 (it'll be called $@), you need the Makefile (it'll be
called $<); once $< is up to date, if it's newer than $@, copy it to the
target you're after."  Note that asking for any one of the files means
asking for the rule to be run, no matter how many of them it makes.

However, that's no use to you for the case of bison generating two
files.  Your actual command doesn't just make one file; it makes two
files.  So the target whose command it is needs to be a (single) "both
files" target, not the two files.  Writing

this that .PHONY: both
both: source
COMMAND

says: if either this or that is older than source, run COMMAND (once).
Whereas

this that: source
COMMAND

says: for each of this and that (call it $@), if $@ is older than
source, run COMMAND.  If both are older than source, this claims we need
to run COMMAND twice: indeed, serial make could check both this and
that, to see if either is older than source, before running COMMAND for
either of them.  For each that is older than source, make would then
perfectly reasonably queue a copy of COMMAND, with the stale file's name
substituted for $@ anywhere it appears in COMMAND, for subsequent
execution.  It would thus run COMMAND twice, once for each of the files;
and, indeed, this would be correct behaviour if COMMAND only updated $@
(which is what make expects from its rules).

As it happens, serial make leaves the staleness check until just before
it runs the command, so it manages to optimise away the second run, if
COMMAND actually does update both (hence also the one that wasn't $@ on
the first run).  That's more or less an accident, though; and it doesn't
happen for parallel make, which duly does run COMMAND twice, in
parallel; and the two instances try to update the same files, which can
cause problems.  The error isn't in make; it is in the make file, which
misdescribed the recipe as a way to make *each* file, when it's actually
a way to make *both*.

Which is why you need that .PHONY intermediate.
It's what says "this command updates *both* files".

As Paul said, writing code to guess "what the author meant" and
implement that is usually perilous.  There are way too many ways for the
guessing code to get it wrong, at least until your guessing is being
done by a true AI that knows to recognise when it doesn't know what the
author meant, so stops the build to ask what was meant.

So, rather than asking for make to "do what I mean, not what I said",
please take the time to tell make what you really meant.  Describe,
faithfully and fully, how the commands relate to the files, then make
can do its dumb but robust thing reliably.  Making things cleverer
usually leads to them breaking; making them tediously straightforward is
good for reliability,

Eddy.
-- 
A shortcut is the longest distance between two points.

___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

Re: "make -jN" requires mechanical changes to a Makefile

[Henrik Carlqvist]

On Mon, 13 May 2019 00:05:59 +0200
Bruno Haible  wrote:

> Howard Chu wrote:
> > >> Example with one rule creating 4 files:
> > >>
> > >> all : copy1 
> > >>
> > >> copy1: Makefile
> > >> install -c -m 644 Makefile copy1
> > >> install -c -m 644 Makefile copy2
> > >> install -c -m 644 Makefile copy3
> > >> install -c -m 644 Makefile copy4
> > > 
> > > I think the "representative" file should be copy4 here, because it's
> > > the one that gets created last.
> > 
> > That sort of thing is only true in serial make, you can't rely on it
> > in parallel make.
> 
> The sequence of lines of the recipe of a rule gets executed in order,
> even in parallel make, no?

Yes, they will be run in sequence even with parallel make and copy4 might
be a better known target than copy1.

regards Henrik

___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

Re: "make -jN" requires mechanical changes to a Makefile

[Bruno Haible]

Howard Chu wrote:
> >> Example with one rule creating 4 files:
> >>
> >> all : copy1 
> >>
> >> copy1: Makefile
> >> install -c -m 644 Makefile copy1
> >> install -c -m 644 Makefile copy2
> >> install -c -m 644 Makefile copy3
> >> install -c -m 644 Makefile copy4
> > 
> > I think the "representative" file should be copy4 here, because it's the one
> > that gets created last.
> 
> That sort of thing is only true in serial make, you can't rely on it in 
> parallel make.

The sequence of lines of the recipe of a rule gets executed in order,
even in parallel make, no?

Bruno


___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

Re: "make -jN" requires mechanical changes to a Makefile

[Howard Chu]

Bruno Haible wrote:
> Henrik Carlqvist wrote:
>> Example with one rule creating 4 files:
>>
>> all : copy1 
>>
>> copy1: Makefile
>> install -c -m 644 Makefile copy1
>> install -c -m 644 Makefile copy2
>> install -c -m 644 Makefile copy3
>> install -c -m 644 Makefile copy4
> 
> I think the "representative" file should be copy4 here, because it's the one
> that gets created last.

That sort of thing is only true in serial make, you can't rely on it in 
parallel make.

-- 
  -- Howard Chu
  CTO, Symas Corp.   http://www.symas.com
  Director, Highland Sun http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/

___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

Re: "make -jN" requires mechanical changes to a Makefile

[Bruno Haible]

Henrik Carlqvist wrote:
> Example with one rule creating 4 files:
> 
> all : copy1 
> 
> copy1: Makefile
> install -c -m 644 Makefile copy1
> install -c -m 644 Makefile copy2
> install -c -m 644 Makefile copy3
> install -c -m 644 Makefile copy4

I think the "representative" file should be copy4 here, because it's the one
that gets created last.

Otherwise, when the user interrupts "make" after copy1 was created but before
copy2, copy3, copy4 are created, and then retries "make" again, the second
'make' invocation will fail somewhere when it references copy2...copy4.

Bruno


___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

Re: "make -jN" requires mechanical changes to a Makefile

[Henrik Carlqvist]

On Sun, 12 May 2019 22:23:12 +0200
Bruno Haible  wrote:
> Now, when my use-case is:
>   - one rule that produces N files (N > 1),
>   - I want "make" to execute the rule only once, not N times,
> even with parallel make.
> What is the solution?

I think that the only good solution is to make sure than only 1 of the N
created files is a known target for the Makefile. If you write single
rules that on one call creates multiple targets your Makefile will not be
compatible with parallel make.

Example with one rule creating 4 files:

all : copy1 

copy1: Makefile
install -c -m 644 Makefile copy1
install -c -m 644 Makefile copy2
install -c -m 644 Makefile copy3
install -c -m 644 Makefile copy4

A better way would be to have one rule to create multiple targets, but
only on target for each call, example:

all: copy1 copy2 copy3 copy4

copy%: Makefile
install -c -m 644 Makefile $@

The above simple Makefile would be fully compatible with parallel make.

regards Henrik

___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

Re: "make -jN" requires mechanical changes to a Makefile

[Bruno Haible]

Hi Paul,

> > The real workaround goes like this:
> > 
> > ===
> > all : copy1 copy2 copy3 copy4
> > 
> > copy1: Makefile
> > install -c -m 644 Makefile copy1
> > install -c -m 644 Makefile copy2
> > install -c -m 644 Makefile copy3
> > install -c -m 644 Makefile copy4
> > copy2 copy3 copy4: copy1
> > ===
> 
> This is not fully-correct either:
> 
>   $ make
>   install -c -m 644 Makefile copy1
>   install -c -m 644 Makefile copy2
>   install -c -m 644 Makefile copy3
>   install -c -m 644 Makefile copy4
> 
>   $ rm copy3
> 
>   $ make
>   make: Nothing to be done for 'all'.

Indeed. Thank you for having spotted this; otherwise I would have put a
broken rule into GNU gettext.

Now, when my use-case is:
  - one rule that produces N files (N > 1),
  - I want "make" to execute the rule only once, not N times,
even with parallel make.
What is the solution? The documentation page [1] explicitly does NOT mention
this use-case for multiple targets on the same rule.

You mentioned splitting the rule into one rule per file. But bison does not
work this way; it really produces two files at once. And for other rules
I mentioned Automake limitations.

Parallel make is of growing importance, because Debian now uses parallel builds
by default (quote: "debhelper >= 10 defaults to parallel build").

Bruno

[1] https://www.gnu.org/software/make/manual/html_node/Multiple-Targets.html


___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

Re: "make -jN" requires mechanical changes to a Makefile

[Paul Smith]

On Sun, 2019-05-12 at 18:07 +0200, Bruno Haible wrote:
> This workaround doesn't actually work (in the actual case of GNU gettext):
> it fails the "make distcheck" verification.
> 
> The real workaround goes like this:
> 
> ===
> all : copy1 copy2 copy3 copy4
> 
> copy1: Makefile
> install -c -m 644 Makefile copy1
> install -c -m 644 Makefile copy2
> install -c -m 644 Makefile copy3
> install -c -m 644 Makefile copy4
> copy2 copy3 copy4: copy1
> ===

This is not fully-correct either:

  $ make
  install -c -m 644 Makefile copy1
  install -c -m 644 Makefile copy2
  install -c -m 644 Makefile copy3
  install -c -m 644 Makefile copy4

  $ rm copy3

  $ make
  make: Nothing to be done for 'all'.


___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

Re: "make -jN" requires mechanical changes to a Makefile

[Bruno Haible]

Hi,

I wrote:
> The workaround is to introduce an intermediate target:
> 
> ===
> all : copy1 copy2 copy3 copy4
> 
> copy1 copy2 copy3 copy4: install-copies
> .PHONY: install-copies
> install-copies: Makefile
>   install -c -m 644 Makefile copy1
>   install -c -m 644 Makefile copy2
>   install -c -m 644 Makefile copy3
>   install -c -m 644 Makefile copy4
> ===

This workaround doesn't actually work (in the actual case of GNU gettext):
it fails the "make distcheck" verification.

The real workaround goes like this:

===
all : copy1 copy2 copy3 copy4

copy1: Makefile
install -c -m 644 Makefile copy1
install -c -m 644 Makefile copy2
install -c -m 644 Makefile copy3
install -c -m 644 Makefile copy4
copy2 copy3 copy4: copy1
===

Bruno


___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

Re: "make -jN" requires mechanical changes to a Makefile

[Bruno Haible]

Henrik Carlqvist wrote:
> If you really prefer to write rules which generates more than one target
> the "right" way to avoid parallel make would be to add the .NOTPARALLEL
> target in the Makefile.

This way allows to turn off parallel make for a single Makefile.
Indeed, this might be a better compromise, when only few Makefiles
have problems with parallel make.

Whereas the snippet I showed turns it off for an entire package (when placed
in the top-level Makefile).

Thanks for the suggestion.

Bruno


___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

Re: "make -jN" requires mechanical changes to a Makefile

[Henrik Carlqvist]

> In the current state, supporting parallel make requires extra work
> for the maintainer.
> 
> Or would you recommend that I add this snippet to the top-level
> Makefile of all my projects?
> 
> # This package does not support parallel make.
> # So, turn off parallel execution (at least in GNU make >= 4.0).
> GNUMAKEFLAGS = -j1

If you really prefer to write rules which generates more than one target
the "right" way to avoid parallel make would be to add the .NOTPARALLEL
target in the Makefile.

regards Henrik

___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

Re: "make -jN" requires mechanical changes to a Makefile

[Bruno Haible]

Hi Paul,

> This makefile should be written correctly, as:
> 
>   all : copy1 copy2 copy3 copy4
> 
>   copy1: Makefile
>   install -c -m 644 Makefile copy1
>   copy2: Makefile
>   install -c -m 644 Makefile copy2
>   copy3: Makefile
>   install -c -m 644 Makefile copy3
>   copy4: Makefile
>   install -c -m 644 Makefile copy4

The Makefile was simplified. In the real-world case [1], the STATEMENTS
is a recursive invocation of another Makefile, which is generated by
Automake, and for which I cannot create a separate targets
  install-textstyle.h
  install-textstyle_stdbool.h
  install-textstyle_version.h
  install-textstyle_woe32dll.h
(because Automake just does not offer this).

Another real-world example (where a parallel make problem surely
exists, but wasn't reported so far) is this rule from
gettext/gettext-tools/src/Makefile.am:

po-gram-gen.c po-gram-gen.h: po-gram-gen.y
$(AM_V_GEN)$(SHELL) $(YLWRAP) $(srcdir)/po-gram-gen.y \
  y.tab.c po-gram-gen.c \
  y.tab.h po-gram-gen.h \
  y.output po-gram-gen.output \
  -- $(YACC) $(YFLAGS) $(AM_YFLAGS) \
&& sed -e 's|".*/po-gram-gen.y"|"po-gram-gen.y"|' < po-gram-gen.c > 
po-gram-gen.c-tmp \
&& rm -f po-gram-gen.c \
&& mv po-gram-gen.c-tmp $(srcdir)/po-gram-gen.c \
&& { test '$(srcdir)' = . || mv po-gram-gen.h $(srcdir)/po-gram-gen.h; }

It's the same type of problem, and the same mechanical change is needed
for proper support of parallel make.

> I really don't like to try to parse recipes and modify
> behavior depending on what is found.  It leads to a lot of unexpected
> consequences and confusion.

In the current state, supporting parallel make requires extra work
for the maintainer.

Or would you recommend that I add this snippet to the top-level
Makefile of all my projects?

# This package does not support parallel make.
# So, turn off parallel execution (at least in GNU make >= 4.0).
GNUMAKEFLAGS = -j1

Bruno

[1] https://lists.gnu.org/archive/html/bug-gettext/2019-05/msg00084.html


___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

Re: "make -jN" requires mechanical changes to a Makefile

[Paul Smith]

On Fri, 2019-05-10 at 22:49 +0200, Bruno Haible wrote:
> But supporting parallel requires, in some cases, mechanical changes to a
> Makefile. How about if GNU make was improved to not require me to make these
> changes?
> 
> Namely, consider this Makefile:
> ===
> all : copy1 copy2 copy3 copy4
> 
> copy1 copy2 copy3 copy4: Makefile
> install -c -m 644 Makefile copy1
> install -c -m 644 Makefile copy2
> install -c -m 644 Makefile copy3
> install -c -m 644 Makefile copy4
> ===

Well, IMO this makefile is just wrong.  For example, in the above
makefile if you run "make copy3" it will install all the files, which
is incorrect.

This makefile should be written correctly, as:

  all : copy1 copy2 copy3 copy4

  copy1: Makefile
  install -c -m 644 Makefile copy1
  copy2: Makefile
  install -c -m 644 Makefile copy2
  copy3: Makefile
  install -c -m 644 Makefile copy3
  copy4: Makefile
  install -c -m 644 Makefile copy4

then it will work properly in both parallel and non-parallel modes, and
it will do the right thing for invocations like "make copy3".

I understand that this requires changes to the makefile... but
incorrect code needs to be changed to be correct, rather than having
the compiler try to fix it for you.  At least, that's my current
thinking about this.

Also in general I really don't like to try to parse recipes and modify
behavior depending on what is found.  It leads to a lot of unexpected
consequences and confusion.


___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

"make -jN" requires mechanical changes to a Makefile

[Bruno Haible]

Hi,

The GNU standards [1] say:
  "Try to make the build and installation targets, at least (and all their
   subtargets) work correctly with a parallel make."

But supporting parallel requires, in some cases, mechanical changes to a
Makefile. How about if GNU make was improved to not require me to make these
changes?

Namely, consider this Makefile:
===
all : copy1 copy2 copy3 copy4

copy1 copy2 copy3 copy4: Makefile
install -c -m 644 Makefile copy1
install -c -m 644 Makefile copy2
install -c -m 644 Makefile copy3
install -c -m 644 Makefile copy4
===

Observe:
* "rm -f copy?; make" works fine.
* "rm -f copy?; make -j8" occasionally fails:
$ rm -f copy? ; make -j8
install -c -m 644 Makefile copy1
install -c -m 644 Makefile copy1
install -c -m 644 Makefile copy1
install -c -m 644 Makefile copy1
install -c -m 644 Makefile copy2
install -c -m 644 Makefile copy2
install -c -m 644 Makefile copy2
install -c -m 644 Makefile copy2
install: cannot change permissions of 'copy2': No such file or directory
install -c -m 644 Makefile copy3
Makefile:4: recipe for target 'copy2' failed
make: *** [copy2] Error 1
make: *** Waiting for unfinished jobs
install -c -m 644 Makefile copy3
install -c -m 644 Makefile copy3
install -c -m 644 Makefile copy4
install -c -m 644 Makefile copy4
install -c -m 644 Makefile copy4

The workaround is to introduce an intermediate target:

===
all : copy1 copy2 copy3 copy4

copy1 copy2 copy3 copy4: install-copies
.PHONY: install-copies
install-copies: Makefile
install -c -m 644 Makefile copy1
install -c -m 644 Makefile copy2
install -c -m 644 Makefile copy3
install -c -m 644 Makefile copy4
===

This is tedious, mechanical work. Couldn't GNU make implement the workaround
by itself? Namely, when during a parallel make, it encounters a rule

target1 ... targetN : dependencies
STATEMENTS

and none of the statements depends on the precise target ($@ or similar),
it should transform that to the form

target1 ... targetN : intermediate_target
.PHONY: intermediate_target
intermediate_target: dependencies
STATEMENTS

internally.

$ make --version
GNU Make 4.1

Best regards,

  Bruno

[1] https://www.gnu.org/prep/standards/html_node/Makefile-Basics.html


___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make