Re: "make -jN" requires mechanical changes to a Makefile [SOLVED]

[Henrik Carlqvist]

On Mon, 14 Sep 2020 12:15:58 +0200
Bruno Haible  wrote:

> Henrik Carlqvist wrote:
> > 2) Don't mention some of the extra targets:
> > ===
> > all : copy1
> >  
> > copy1: Makefile
> > install -c -m 644 Makefile copy1
> > install -c -m 644 Makefile copy2
> > install -c -m 644 Makefile copy3
> > install -c -m 644 Makefile copy4
> > ===
> 
> Fails (D) and (E). => Not a solution to the problem.
> 
> Bruno
> 

Ok, assuming that solution 1 did not meet your requirements, lets give it
another shot:

===
COPIES=copy1 copy2 copy3 copy4

.INTERMEDIATE: dummy

all: $(COPIES)

$(COPIES): %: dummy
$(RM) $<

dummy: Makefile
install -c -m 644 Makefile copy1
install -c -m 644 Makefile copy2
install -c -m 644 Makefile copy3
install -c -m 644 Makefile copy4
touch $@
===

I create an extra file "dummy" for a short time in an attempt to live up to
all your requirements.

regards Henrik

Re: "make -jN" requires mechanical changes to a Makefile [SOLVED]

[Bruno Haible]

Henrik Carlqvist wrote:
> 2) Don't mention some of the extra targets:
> ===
> all : copy1
>  
> copy1: Makefile
>   install -c -m 644 Makefile copy1
>   install -c -m 644 Makefile copy2
>   install -c -m 644 Makefile copy3
>   install -c -m 644 Makefile copy4
> ===

Fails (D) and (E). => Not a solution to the problem.

Bruno

Re: "make -jN" requires mechanical changes to a Makefile [SOLVED]

[Henrik Carlqvist]

On Sun, 13 Sep 2020 20:07:27 +0100
Bruno Haible wrote:
> Continuing this thread from May 2019
> :
> The problem was:
> 
>   How can a rule that generates multiple files be formulated so
>   that it works with parallel make?
> 
> For example, a rule that invokes bison, or a rule that invokes
> a different Makefile. For simplicity, here, use a rule that
> creates 4 files copy1, copy2, copy3, copy4.
> 
> ===
> all : copy1 copy2 copy3 copy4
> 
> copy1 copy2 copy3 copy4: Makefile
>   install -c -m 644 Makefile copy1
>   install -c -m 644 Makefile copy2
>   install -c -m 644 Makefile copy3
>   install -c -m 644 Makefile copy4
> ===

I would say there are two obvious solutions to make this work with parallel
make:


1) Allow the targets to be tuilt in paralllel:
===
all : copy1 copy2 copy3 copy4
 
copy1 copy2 copy3 copy4: Makefile
install -c -m 644 Makefile $@
===

However, this solution 1 will not work for a single command that generates
multiple files, so we might need a solution 2:

2) Don't mention some of the extra targets:
===
all : copy1
 
copy1: Makefile
install -c -m 644 Makefile copy1
install -c -m 644 Makefile copy2
install -c -m 644 Makefile copy3
install -c -m 644 Makefile copy4
===

regards Henrik

Re: "make -jN" requires mechanical changes to a Makefile [SOLVED]

[Bruno Haible]

Hi Paul,

> There is a straightforward and portable way to do this even with
> traditional make, it's just not as nice (but, nicer than changing all
> the recipes to use test IMO! :)).
> 
> If you have a rule like this:
> 
> ... : 
>   
> 
> where  generates all targets with one invocation, then your
> best bet is to modify this rule into two rules:
> 
> ... : .sentinel ;
> 
>   .sentinel: 
>   
>   @touch $@
> 
> Note, it's critical to include the semicolon here.  Or if you prefer to
> be more explicit you can write something like:
> 
> ... : .sentinel
>   : do nothing
> 
> This will work properly.

Thanks for showing this idiom.

I think this can be improved a bit: Instead of 'touch .sentinel' I would
use 'echo > .sentinel'. Because if you use 'touch .sentinel' and the build
directory is on an NFS mount and the clock of the NFS server is ahead the
clock of localhost (I have seen this quite frequently), the timestamp of
.sentinel will be behind the timestamp of the newest , and thus
another "make" run will execute the rule again.

Still, I don't like these '.sentinel' or stamp files, because they
add complexity:
  * The developer has to decide whether the stamp file should go into
the build dir or $(srcdir). The general rule should be that if
 uses only build tools that are guaranteed to be present,
then   ... and the stamp file should go into the build dir,
whereas if they use less common tools (e.g. bison), they should
all go into $(srcdir).
  * The automake dist target needs to be adjusted to include this stamp
file.
  * Version control (.gitignore) needs to be adjusted to include this
stamp file.

I'll use the rule with a phony target and 'test ... -ot ...', unless you
find a problem with it (other than the potential portability problem
redarding 'test').

Bruno

Re: "make -jN" requires mechanical changes to a Makefile [SOLVED]

[Paul Smith]

On Sun, 2020-09-13 at 15:08 -0400, Paul Smith wrote:
> ... : .sentinel ;
> 
>   .sentinel: 
>   
>   @touch $@

Just to be clear, you don't have to use ".sentinel" you can use any
target name, and obviously you must use a different name for each
"grouping" of targets that needs to be built.

Re: "make -jN" requires mechanical changes to a Makefile [SOLVED]

[Bruno Haible]

> ===
> all : copy1 copy2 copy3 copy4
> 
> copy1: Makefile
>   { test -f copy1 && test ! copy1 -ot Makefile; } || { rm -f copy4; 
> $(MAKE) copies; }
> copy2: copy1
>   { test -f copy2 && test ! copy2 -ot copy1; } || { rm -f copy4; $(MAKE) 
> copies; }
> copy3: copy2
>   { test -f copy3 && test ! copy3 -ot copy2; } || { rm -f copy4; $(MAKE) 
> copies; }
> copy4: copy3
>   { test -f copy4 && test ! copy4 -ot copy3; } || { rm -f copy4; $(MAKE) 
> copies; }
> 
> copies:
>   install -c -m 644 Makefile copy1
>   install -c -m 644 Makefile copy2
>   install -c -m 644 Makefile copy3
>   install -c -m 644 Makefile copy4
> .PHONY: copies
> ===
> 
> This solution fulfils all the requirements.

It can be simplified a bit. And remove verbosity:

===
all : copy1 copy2 copy3 copy4

copy1: Makefile
@{ test -f copy1 && test ! copy1 -ot Makefile; } || $(MAKE) copies
copy2: copy1
@{ test -f copy2 && test ! copy2 -ot copy1; } || $(MAKE) copies
copy3: copy2
@{ test -f copy3 && test ! copy3 -ot copy2; } || $(MAKE) copies
copy4: copy3
@{ test -f copy4 && test ! copy4 -ot copy3; } || $(MAKE) copies

copies:
install -c -m 644 Makefile copy1
install -c -m 644 Makefile copy2
install -c -m 644 Makefile copy3
install -c -m 644 Makefile copy4
.PHONY: copies
===

Re: "make -jN" requires mechanical changes to a Makefile [SOLVED]

[Paul Smith]

On Sun, 2020-09-13 at 20:55 +0200, Bruno Haible wrote:
>   How can a rule that generates multiple files be formulated so
>   that it works with parallel make?
> 
> For example, a rule that invokes bison, or a rule that invokes
> a different Makefile. For simplicity, here, use a rule that
> creates 4 files copy1, copy2, copy3, copy4.

Sorry, I think the last time this came up I got side-tracked by the
fact that the example was using install.

There is a straightforward and portable way to do this even with
traditional make, it's just not as nice (but, nicer than changing all
the recipes to use test IMO! :)).

If you have a rule like this:

... : 
  

where  generates all targets with one invocation, then your
best bet is to modify this rule into two rules:

... : .sentinel ;

  .sentinel: 
  
  @touch $@

Note, it's critical to include the semicolon here.  Or if you prefer to
be more explicit you can write something like:

... : .sentinel
  : do nothing

This will work properly.

Re: "make -jN" requires mechanical changes to a Makefile [SOLVED]

[Howard Chu]

Bruno Haible wrote:
> Continuing this thread from May 2019
> :
> The problem was:
> 
>   How can a rule that generates multiple files be formulated so
>   that it works with parallel make?
> 
> For example, a rule that invokes bison, or a rule that invokes
> a different Makefile. For simplicity, here, use a rule that
> creates 4 files copy1, copy2, copy3, copy4.
> 
> ===
> all : copy1 copy2 copy3 copy4
> 
> copy1 copy2 copy3 copy4: Makefile
>   install -c -m 644 Makefile copy1
>   install -c -m 644 Makefile copy2
>   install -c -m 644 Makefile copy3
>   install -c -m 644 Makefile copy4
> ===
> 
> Unfortunately, with "make -j8", it invokes the rule multiple times.
> 
> It is possible to change this Makefile so that
>   (A) "rm -f copy?; make" executes the rule once.
>   (B) "rm -f copy?; make -j8" executes the rule once as well.
>   (C) After "make", another "make" just prints "Nothing to be done for 'all'."
>   (D) After removing one of the files copy?, "make" executes the rule once.
>   (This covers also the case of pressing Ctrl-C during "make", then
>   doing "make" again.)
>   (E) After replacing one of the files copy? with a file that is older than
>   Makefile, "make" executes the rule once.
> 
> There are three possibilities:

You're thinking about this the wrong way. Your set of commands is inherently
serial, therefore you need to write serial dependencies.

===
all: copy1 copy2 copy3 copy4

copy1: Makefile
install -c -m 644 Makefile $@
copy2: copy1
install -c -m 644 Makefile $@
copy3: copy2
install -c -m 644 Makefile $@
copy4: copy3
install -c -m 644 Makefile $@
===

This satisfies all your conditions, because it is inherently correct.

More on writing proper parallel Makefiles here http://highlandsun.com/hyc/#Make

-- 
  -- Howard Chu
  CTO, Symas Corp.   http://www.symas.com
  Director, Highland Sun http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/

Re: "make -jN" requires mechanical changes to a Makefile [SOLVED]

[Bruno Haible]

Continuing this thread from May 2019
:
The problem was:

  How can a rule that generates multiple files be formulated so
  that it works with parallel make?

For example, a rule that invokes bison, or a rule that invokes
a different Makefile. For simplicity, here, use a rule that
creates 4 files copy1, copy2, copy3, copy4.

===
all : copy1 copy2 copy3 copy4

copy1 copy2 copy3 copy4: Makefile
install -c -m 644 Makefile copy1
install -c -m 644 Makefile copy2
install -c -m 644 Makefile copy3
install -c -m 644 Makefile copy4
===

Unfortunately, with "make -j8", it invokes the rule multiple times.

It is possible to change this Makefile so that
  (A) "rm -f copy?; make" executes the rule once.
  (B) "rm -f copy?; make -j8" executes the rule once as well.
  (C) After "make", another "make" just prints "Nothing to be done for 'all'."
  (D) After removing one of the files copy?, "make" executes the rule once.
  (This covers also the case of pressing Ctrl-C during "make", then
  doing "make" again.)
  (E) After replacing one of the files copy? with a file that is older than
  Makefile, "make" executes the rule once.

There are three possibilities:


(I) Assuming GNU make >= 4.3, the "Grouped explicit target" syntax does it.
Thanks to Kaz Kylheku and Paul Smith for having added this.

===
all : copy1 copy2 copy3 copy4

copy1 copy2 copy3 copy4 &: Makefile
install -c -m 644 Makefile copy1
install -c -m 644 Makefile copy2
install -c -m 644 Makefile copy3
install -c -m 644 Makefile copy4
===

But it will take a number of years until we can assume that all 'make'
programs that we care about support this syntax.


(II) It is possible to turn off parallel make. This does it:

===
all : copy1 copy2 copy3 copy4

copy1 copy2 copy3 copy4 : Makefile
install -c -m 644 Makefile copy1
install -c -m 644 Makefile copy2
install -c -m 644 Makefile copy3
install -c -m 644 Makefile copy4

# This Makefile contains rules which don't work with parallel make.
# So, turn off parallel execution in this Makefile.
.NOTPARALLEL:
===

But some people who want to minimize wall-clock execution time of
their builds may not like it.


(III) Use portable make syntax and still allow parallel make.

This is a bit harder. My solution is to first analyze in which order
the rule will generate the various files and thus what the expected
timestamp order is. In this case, it is:
  Makefile <= copy1 <= copy2 <= copy3 <= copy4

First some attempts that don't work:

===
all : copy1 copy2 copy3 copy4

copy1: Makefile
install -c -m 644 Makefile copy1
install -c -m 644 Makefile copy2
install -c -m 644 Makefile copy3
install -c -m 644 Makefile copy4

copy2 copy3 copy4: copy1
===

Fails (E).

===
all : copy1 copy2 copy3 copy4

copy4: Makefile
install -c -m 644 Makefile copy1
install -c -m 644 Makefile copy2
install -c -m 644 Makefile copy3
install -c -m 644 Makefile copy4

copy1 copy2 copy3: copy4
===

Fails (E) as well.

===
all : copy1 copy2 copy3 copy4

copy4: Makefile
install -c -m 644 Makefile copy1
install -c -m 644 Makefile copy2
install -c -m 644 Makefile copy3
install -c -m 644 Makefile copy4

copy1: copy4
test -f copy1 || { rm -f copy4; $(MAKE) copy4; }
copy2: copy4
test -f copy2 || { rm -f copy4; $(MAKE) copy4; }
copy3: copy4
test -f copy3 || { rm -f copy4; $(MAKE) copy4; }
===

Fails (E) for copy1, ..., copy3.
And fails (C), i.e. clutters the "make" output.

===
all : copy1 copy2 copy3 copy4

copy1: Makefile
{ test -f copy1 && test ! copy1 -ot Makefile; } || { rm -f copy4; 
$(MAKE) copies; }
copy2: copy1
{ test -f copy2 && test ! copy2 -ot copy1; } || { rm -f copy4; $(MAKE) 
copies; }
copy3: copy2
{ test -f copy3 && test ! copy3 -ot copy2; } || { rm -f copy4; $(MAKE) 
copies; }
copy4: copy3
{ test -f copy4 && test ! copy4 -ot copy3; } || { rm -f copy4; $(MAKE) 
copies; }

copies:
install -c -m 644 Makefile copy1
install -c -m 644 Makefile copy2
install -c -m 644 Makefile copy3
install -c -m 644 Makefile copy4
.PHONY: copies
===

This solution fulfils all the requirements.

Can we use 'test FILE1 -ot FILE2'?
- POSIX 'test' does not support this option.