Re: Doubts about .WAIT and .NOTPARALLEL (was: GNU Make 4.4 released!)

2022-10-31 Thread Paul Smith
On Mon, 2022-10-31 at 11:06 +0100, Alejandro Colomar wrote:
> On 10/31/22 08:01, Paul Smith wrote:
> > * New feature: The .WAIT special target
> 
> I'm curious: what is .WAIT made for?

As mentioned in the NEWS, it's _primarly_ there because the next
release of the POSIX spec will require a POSIX-conforming make utility
to support it.

> Isn't it just a workaround for broken dependencies?

Fundamentally I agree with you, that .WAIT is a poor workaround for a
true prerequisite statement, and is hard to use in a reliable way.

I made this case to the POSIX folks.  However, the .WAIT facility with
its current behavior has been available in BSD versions of make for a
very long time and is widely used there.  It tends to be used more for
things like the FreeBSD Ports build system where you're declaring
higher-level targets to be waited on so it's simpler to reason about.

>   I mean, considering the following example:
> 
>  all: one two .WAIT three
>  one two three: ; @sleep 1; echo $@
> 
> It's the same as if it was written as
> 
>  all: one two three
>  three: one two
>  one two three: ; @sleep 1; echo $@
> 
> Isn't it?

As Eddy points out it's not identical in behavior.  Fundamentally .WAIT
does not add any new edges to the DAG that make constructs, so there is
nothing inherent to the target "three" (in your example) that connects
it to "one" or "two".  The "wait-ness" is a function of the _target_
(all), not the prerequisites at all.  And, it's specific to that
target.  So if you have:

  foo: one .WAIT two
  bar: one two

then if you run "make -j foo" make will wait for "one" to complete
before running "two", but if you run "make -j bar" then it will not.

Even more annoyingly, if you have:

  all: foo bar
  foo: one .WAIT two
  bar: one two

and you run "make -j all" then "one" and "two" will be run concurrently
because even though when make wants to build "foo" it will wait to
start "two", when make wants to build "bar" concurrently with "foo"
then "one" and "two" will run concurrently.

So like I said, IMO .WAIT is a suboptimal capability that is hard to
use reliably and probably isn't very useful compared to other ordering
capabilities that GNU make already has.

But, it is very easy to use, is required by POSIX, and there are a lot
of makefiles that want to use it.

So now they can.

> > * New feature: .NOTPARALLEL accepts prerequisites
> 
> But my useful question comes with .NOTPARALLEL, which I think can be
> useful for something I asked some time ago, and can't be done with
> pre-4.4 make.
> 
> The documentation is a bit unclear on the exact details of this
> feature, by writing it in terms of .WAIT.
> 
> Let's say I have the following:
> 
>  all: a b c
>  .NOTPARALLEL: b
>  a: b c
>  a b c: ; @sleep 1; echo $@
> 
> Is it guaranteed that b will be run either before or after c but
> never at the same time?  Please clarify the manual regarding this. 
> Maybe this example is good for the docs.

No.  In the makefile above, .NOTPARALLEL has exactly zero effect.  The
manual says:

   This implicitly adds a .WAIT between each prerequisite of the listed
   targets.

I think this is pretty clear.  You might be confusing yourself by
hoping this provides some capability that it doesn't and so reading
things into the manual beyond what it actually says.

Probably an example would be helpful in the manual, but in your snippet
above you don't list any prerequisites for "b", so ".NOTPARALLEL: b",
which adds .WAIT between each prerequisite, is a no-op.

If you had this:

  .NOTPARALLEL: b
  b: x y z

it would be identical to writing:

  b: x .WAIT y .WAIT z

and exactly that, nothing else.

> The usefulness of something like this is that b might run a command
> that asks for a password, and I want literraly _everything_ else on
> hold while a program is asking for a password.

That is not how it works.

I don't believe that such a requirement can or should be met by some
form of .NOTPARALLEL.  If we wanted to create such a thing the way I
would do it is create a new special target like ".PREMAKE" or whatever,
and all the prerequisites of that special target would be guaranteed to
be updated first, before any other target, regardless of the goal
targets.

Or, something like that.  Details would need to be considered.

> I guess shuffle doesn't mess with .NOTPARALLEL or .WAIT.  Just to
> confirm.

I'm not sure what you mean by "mess with", but they have no
relationship to each other.

> > * New feature: The --jobserver-style command line option and named
> > pipes
> 
> Another curious question:  Does this fifo jobserver allow
> .NOTPARALLEL even with recursive makes?

It has no relationship to .NOTPARALLEL which continues to be confined
to a single instance of make.  I don't even really understand what it
would mean to have .NOTPARALLEL cross recursive makes.



Re: Doubts about .WAIT and .NOTPARALLEL (was: GNU Make 4.4 released!)

2022-10-31 Thread Alejandro Colomar

Hi Eddy,

On 10/31/22 14:52, Edward Welbourne wrote:

I mean, considering the following example:

    all: one two .WAIT three
    one two three: ; @sleep 1; echo $@

It's the same as if it was written as

    all: one two three
    three: one two
    one two three: ; @sleep 1; echo $@

Isn't it?


On 10/31/22 13:42, Edward Welbourne wrote:

Only from the point of view of make all; try make three and you'll see
the difference.

If you make three with the first make file, it'll take one second and
just echo three.  Passing -j 37 will make no difference.  If you make
three with the second make file, it'll take at least two seconds (three
with -j1) and output one and two (in some order) then three.

If you make all with the either makefile, it'll behave like make three
with the second makefile, but differently from make three with the first
makefile.

If any other targets depend on three, they're likewise changed by the
second makefile, compared to the first,


Alejandro Colomar (Monday, October 31, 2022 13:46) replied:

Ahh, sure, that's right.

However, I don't see why this would be beneficial.  Why would A need
to be built before B, if there's no dependency between them?


If each of them needs to lock some resource to do its job, you don't
want them to be run in parallel.  For example, each of them appends to a
common log file, and you don't want their log entries interleaved; or
each updates a database with a new record relating to the present build.
Cases like this are also relevant to .NOT_PARALLEL; you might not care
what order the entries are processed in, just so long as no two of them
are trying to run at the same time.

That probably applies to all sorts of resources that take incremental
updates; make is a tool for doing the sorts of update that completely
rewrite a file, not the sort that add entries incrementally.

I have no doubt users of make will discover a plethora of other ways
that this is useful.


I think this is hiding some kind of missing dependency.  Maybe this is
a workaround for not having post-requisites yet?


While it might be used as a kludge-around for some such cases, it has
uses that cannot be represented faithfully be dependencies and really
just come down to "these two targets should not be built in parallel".


Thanks a lot for the detailed example!  It makes sense.

Cheers,

Alex

--



OpenPGP_signature
Description: OpenPGP digital signature


Re: Doubts about .WAIT and .NOTPARALLEL (was: GNU Make 4.4 released!)

2022-10-31 Thread Edward Welbourne
>>> I mean, considering the following example:
>>>
>>>   all: one two .WAIT three
>>>   one two three: ; @sleep 1; echo $@
>>>
>>> It's the same as if it was written as
>>>
>>>   all: one two three
>>>   three: one two
>>>   one two three: ; @sleep 1; echo $@
>>>
>>> Isn't it?

On 10/31/22 13:42, Edward Welbourne wrote:
>> Only from the point of view of make all; try make three and you'll see
>> the difference.
>> 
>> If you make three with the first make file, it'll take one second and
>> just echo three.  Passing -j 37 will make no difference.  If you make
>> three with the second make file, it'll take at least two seconds (three
>> with -j1) and output one and two (in some order) then three.
>> 
>> If you make all with the either makefile, it'll behave like make three
>> with the second makefile, but differently from make three with the first
>> makefile.
>> 
>> If any other targets depend on three, they're likewise changed by the
>> second makefile, compared to the first,

Alejandro Colomar (Monday, October 31, 2022 13:46) replied:
> Ahh, sure, that's right.
>
> However, I don't see why this would be beneficial.  Why would A need
> to be built before B, if there's no dependency between them?

If each of them needs to lock some resource to do its job, you don't
want them to be run in parallel.  For example, each of them appends to a
common log file, and you don't want their log entries interleaved; or
each updates a database with a new record relating to the present build.
Cases like this are also relevant to .NOT_PARALLEL; you might not care
what order the entries are processed in, just so long as no two of them
are trying to run at the same time.

That probably applies to all sorts of resources that take incremental
updates; make is a tool for doing the sorts of update that completely
rewrite a file, not the sort that add entries incrementally.

I have no doubt users of make will discover a plethora of other ways
that this is useful.

> I think this is hiding some kind of missing dependency.  Maybe this is
> a workaround for not having post-requisites yet?

While it might be used as a kludge-around for some such cases, it has
uses that cannot be represented faithfully be dependencies and really
just come down to "these two targets should not be built in parallel".

Eddy.



Re: Doubts about .WAIT and .NOTPARALLEL (was: GNU Make 4.4 released!)

2022-10-31 Thread Edward Welbourne
On 10/31/22 08:01, Paul Smith wrote:
>> * New feature: The .WAIT special target
>>    If the .WAIT target appears between two prerequisites of a target, then
>>    GNU Make will wait for all of the targets to the left of .WAIT in the list
>>    to complete before starting any of the targets to the right of .WAIT.
>>    This feature is available in some other versions of make, and it will be
>>    required by an upcoming version of the POSIX standard for make.
>>    Different patches were made by Alexey Neyman  
>>(2005)
>>    and Steffen Nurpmeso  (2020) that were useful but the
>>    result is a different implementation (closer to Alexey's idea).

Alejandro Colomar (Monday, October 31, 2022 11:06) replied:
> I'm curious: what is .WAIT made for?  Isn't it just a workaround for
> broken dependencies?

No.

> I mean, considering the following example:
>
>  all: one two .WAIT three
>  one two three: ; @sleep 1; echo $@
>
> It's the same as if it was written as
>
>  all: one two three
>  three: one two
>  one two three: ; @sleep 1; echo $@
>
> Isn't it?

Only from the point of view of make all; try make three and you'll see
the difference.

If you make three with the first make file, it'll take one second and
just echo three.  Passing -j 37 will make no difference.  If you make
three with the second make file, it'll take at least two seconds (three
with -j1) and output one and two (in some order) then three.

If you make all with the either makefile, it'll behave like make three
with the second makefile, but differently from make three with the first
makefile.

If any other targets depend on three, they're likewise changed by the
second makefile, compared to the first,

Eddy.



Re: Doubts about .WAIT and .NOTPARALLEL (was: GNU Make 4.4 released!)

2022-10-31 Thread Alejandro Colomar

Hi Eddy,

On 10/31/22 13:42, Edward Welbourne wrote:

Alejandro Colomar (Monday, October 31, 2022 11:06) replied:

I'm curious: what is .WAIT made for?  Isn't it just a workaround for
broken dependencies?


No.


I mean, considering the following example:

  all: one two .WAIT three
  one two three: ; @sleep 1; echo $@

It's the same as if it was written as

  all: one two three
  three: one two
  one two three: ; @sleep 1; echo $@

Isn't it?


Only from the point of view of make all; try make three and you'll see
the difference.

If you make three with the first make file, it'll take one second and
just echo three.  Passing -j 37 will make no difference.  If you make
three with the second make file, it'll take at least two seconds (three
with -j1) and output one and two (in some order) then three.

If you make all with the either makefile, it'll behave like make three
with the second makefile, but differently from make three with the first
makefile.

If any other targets depend on three, they're likewise changed by the
second makefile, compared to the first,


Ahh, sure, that's right.

However, I don't see why this would be beneficial.  Why would A need to be built 
before B, if there's no dependency between them?  I think this is hiding some 
kind of missing dependency.  Maybe this is a workaround for not having 
post-requisites yet?


Cheers,

Alex

--



OpenPGP_signature
Description: OpenPGP digital signature


Doubts about .WAIT and .NOTPARALLEL (was: GNU Make 4.4 released!)

2022-10-31 Thread Alejandro Colomar

Hi Paul,

On 10/31/22 08:01, Paul Smith wrote:



* New feature: The .WAIT special target
   If the .WAIT target appears between two prerequisites of a target, then
   GNU Make will wait for all of the targets to the left of .WAIT in the list
   to complete before starting any of the targets to the right of .WAIT.
   This feature is available in some other versions of make, and it will be
   required by an upcoming version of the POSIX standard for make.
   Different patches were made by Alexey Neyman  (2005)
   and Steffen Nurpmeso  (2020) that were useful but the
   result is a different implementation (closer to Alexey's idea).


I'm curious: what is .WAIT made for?  Isn't it just a workaround for broken 
dependencies?  I mean, considering the following example:


all: one two .WAIT three
one two three: ; @sleep 1; echo $@

It's the same as if it was written as

all: one two three
three: one two
one two three: ; @sleep 1; echo $@

Isn't it?



* New feature: .NOTPARALLEL accepts prerequisites
   If the .NOTPARALLEL special target has prerequisites then all prerequisites
   of those targets will be run serially (as if .WAIT was specified between
   each prerequisite).


But my useful question comes with .NOTPARALLEL, which I think can be useful for 
something I asked some time ago, and can't be done with pre-4.4 make.


The documentation is a bit unclear on the exact details of this feature, by 
writing it in terms of .WAIT.


Let's say I have the following:

all: a b c
.NOTPARALLEL: b
a: b c
a b c: ; @sleep 1; echo $@

Is it guaranteed that b will be run either before or after c but never at the 
same time?  Please clarify the manual regarding this.  Maybe this example is 
good for the docs.


The usefulness of something like this is that b might run a command that asks 
for a password, and I want literraly _everything_ else on hold while a program 
is asking for a password.


Of course, all this is without recursive make, I know.


* New feature: The --shuffle command line option
   This option reorders goals and prerequisites to simulate non-determinism
   that may be seen using parallel build.  Shuffle mode allows a form of "fuzz
   testing" of parallel builds to verify that all prerequisites are correctly
   described in the makefile.
   Implementation provided by Sergei Trofimovich 


I guess shuffle doesn't mess with .NOTPARALLEL or .WAIT.  Just to confirm.



* New feature: The --jobserver-style command line option and named pipes
   A new jobserver method is used on systems where mkfifo(3) is supported.
   This solves a number of obscure issues related to using the jobserver
   and recursive invocations of GNU Make.  This change means that sub-makes
   will connect to the jobserver even if they are not marked as recursive.
   It also means that other tools that want to participate in the jobserver
   will need to be enhanced as described in the GNU Make manual.
   You can force GNU Make to use the simple pipe-based jobserver (perhaps if
   you are integrating with other tools or older versions of GNU Make) by
   adding the '--jobserver-style=pipe' option to the command line of the
   top-level invocation of GNU Make, or via MAKEFLAGS or GNUMAKEFLAGS.
   To detect this change search for 'jobserver-fifo' in the .FEATURES variable.


Another curious question:  Does this fifo jobserver allow .NOTPARALLEL even with 
recursive makes?  I guess not, but I had to ask.


Cheers,

Alex
--



OpenPGP_signature
Description: OpenPGP digital signature