Re: Potential Bug: `.PHONY` targets and order-only prerequisites

[Paul Smith]

On Sat, 2022-05-21 at 14:34 -0400, Dmitry Goncharov wrote:
> On Sat, May 21, 2022 at 12:25 PM Paul Smith  wrote:
> > Maybe what you're saying is that make should throw an error or
> > warning if you try to add an order-only prerequisite to a phony
> > target, telling you that it will have no effect on your makefile?
> 
> Having a phony target depend (usually indirectly, but there are
> examples of immediate dependency) on an order-only prerequisite is
> quite common. This happens when the top target is marked phony and
> there is an order-only prerequisite down the graph.

Agreed we would never mention an indirect dependency.  That is not an
invalid use of order-only prerequisites anyway; the order-only
prerequisite will be relevant only for its immediate target not
"parents" of that target and if the immediate target is not phony then
the o-o prerequisite is entirely valid there.

I can't think of a reason to add an order-only prerequisite directly to
a phony target on purpose, but I suppose it could happen as the result
of some variable expansion or something which is annoying to avoid.

But I do not plan to add any sort of warning like this at the moment.

Re: Potential Bug: `.PHONY` targets and order-only prerequisites

[Dmitry Goncharov]

On Sat, May 21, 2022 at 12:25 PM Paul Smith  wrote:
> Maybe what you're saying is that make should throw an error or warning
> if you try to add an order-only prerequisite to a phony target, telling
> you that it will have no effect on your makefile?

Having a phony target depend (usually indirectly, but there are
examples of immediate dependency) on an order-only prerequisite is
quite common. This happens when the top target is marked phony and
there is an order-only prerequisite down the graph.

regards, Dmitry

Re: Potential Bug: `.PHONY` targets and order-only prerequisites

[Alejandro Colomar]

Hi Paul and Jacob,

On 5/21/22 18:24, Paul Smith wrote:

On Wed, 2022-05-18 at 14:36 -0700, Jacob Kopczynski wrote:

The thing that the docs refer to as "impose order" is not a single
thing, but two. I would characterize a normal prerequisite as doing
three things rather than two:
- update-marking: cause a target to be marked out of date if the
prereq is marked out of date
- require-existence: require the prereq to be built successfully at
least once before the target is built
- imposed-order: require the prereq to be built before the target, if
both are being built


I guess I've just been using make for too long because I don't
understand the distinction you're trying to make between the last two.
There's no difference here, and the extra comment at the end "if both
are being built" is not meaningful (or anyway I don't understand what
it means).


I kind of get what he means.  Let me try to explain below.



Let's review how make works: it reads the makefile(s) and builds a
directed acyclic graph where every target is a node and every
prerequisite relationship is an edge between two nodes.

Then starting with each node representing a goal target (either the
first target in the makefile, or the target(s) given on the command
line) make performs a depth-first walk of the graph starting at that
node.  "Walking the graph" consists of processing each child node
recursively, in order, and once all children are complete make compares
the timestamp of each (not order-only) child node to the timestamp of
the current node.  If the current node is out of date, then it is
updated (by running its recipe), else we do nothing.  Then this node is
complete and we return to the processing of the parent node, or if
we're the goal target then we're done.


I think knowing so much how make(1) is implemented is hiding the 
intuitive idea of make(1) for someone not so familiar with it.  By "once 
all children are complete" you are implying the "existence" of the 
children (which make(1) doesn't really check, but one can think of it as 
if it did).


Still, since make(1) doesn't really check for existence of files, I 
think the current term (order-only) is the correct one.


See some example that will probably show why existence is really never 
tested:



$ cat Makefile
a: b
ls b

# Oh, I accidentally forgot to actually create b in its recipe
#
# To make(1), 'b' "exists", as it's recipe has succeeded.
#
# But if someone thinks of make as something that makes sure that
# 'b' exists before building 'a', this is for sure unintuitive.
# Can't blame them, as in the common case make(1) is used so that
# one can think of it as making sure the prerequisites exist.
b:
touch a

$ make --silent
ls: cannot access 'b': No such file or directory
make: *** [Makefile:4: a] Error 2


Jacob, as you can see, make(1) doesn't fail here due to the 
non-existence of b.  It simply assumes that it exists, but never really 
tests if it exists.  It is ls(1) that is failing, as make(1) run it 
assuming that 'b' existed.


An implementation that checked that a non-.PHONY target really existed 
after running its recipe wouldn't seem insane to me; just a bit 
inefficient, and maybe in some corner case one would make good use of 
the behavior represented by my broken Makefile above, which wouldn't be 
possible if make(1) really checked for existence.


Cheers,

Alex

Re: Potential Bug: `.PHONY` targets and order-only prerequisites

[Paul Smith]

On Sat, 2022-05-21 at 19:06 +0200, Alejandro Colomar wrote:
> By "once all children are complete" you are implying the "existence"
> of the children (which make(1) doesn't really check, but one can
> think of it as if it did).

Perhaps that's the confusion.  Make doesn't care about files at all per
se: it treats the filesystem as a key/value store where the key is the
pathname and the value is the mod time.  It will look up each target in
that store (filesystem) and if it doesn't exist it gets a special
"oldest" timestamp so that the recipe is run.  After the recipe is run
make gives the node a special "newest" timestamp so targets that list
it as a prerequisite are out of date.  It's not an error (to make) if
the key (pathname) still doesn't exist after the recipe runs.

Make really only cares about its internal graph.  The filesystem is
just a place to keep target state in between invocations of make (and
of course, a way for other tools to let make know that something
changed), since make doesn't have its own database.

Having make complain for any non-phony target that didn't actually
create its target would violate POSIX, and probably cause hundreds of
thousands of makefiles out there to throw complaints where people
forgot or didn't want to bother with using .PHONY (not all makefiles
were written for GNU make, and .PHONY is a GNU make feature).

Re: Potential Bug: `.PHONY` targets and order-only prerequisites

[Paul Smith]

On Wed, 2022-05-18 at 14:36 -0700, Jacob Kopczynski wrote:
> The thing that the docs refer to as "impose order" is not a single
> thing, but two. I would characterize a normal prerequisite as doing
> three things rather than two:
> - update-marking: cause a target to be marked out of date if the
> prereq is marked out of date
> - require-existence: require the prereq to be built successfully at
> least once before the target is built
> - imposed-order: require the prereq to be built before the target, if
> both are being built

I guess I've just been using make for too long because I don't
understand the distinction you're trying to make between the last two.
There's no difference here, and the extra comment at the end "if both
are being built" is not meaningful (or anyway I don't understand what
it means).

Let's review how make works: it reads the makefile(s) and builds a
directed acyclic graph where every target is a node and every
prerequisite relationship is an edge between two nodes.

Then starting with each node representing a goal target (either the
first target in the makefile, or the target(s) given on the command
line) make performs a depth-first walk of the graph starting at that
node.  "Walking the graph" consists of processing each child node
recursively, in order, and once all children are complete make compares
the timestamp of each (not order-only) child node to the timestamp of
the current node.  If the current node is out of date, then it is
updated (by running its recipe), else we do nothing.  Then this node is
complete and we return to the processing of the parent node, or if
we're the goal target then we're done.

I don't see any way that you can talk about "require-existence"
separately from "imposed-order", and the attempt to do so is (it seems
to me) just creating confusion.

> For .PHONY targets, require-existence always mandates update-marking,
> which is why the overlap between those two cases is particularly
> confusing - a phony target which drops only update-marking is a
> contradiction in terms, so the natural assumption to make is that the
> effect of "order-only" prereqs is not that, and is instead something
> non-paradoxical.

Order-only prerequisites weren't created because people wanted to use
them with .PHONY targets.  They were created for use with non-phony
targets.

You can add them into phony targets, but they make no difference there
as we described.

Maybe what you're saying is that make should throw an error or warning
if you try to add an order-only prerequisite to a phony target, telling
you that it will have no effect on your makefile?

Perhaps.  I'm generally not inclined to do that sort of thing but maybe
it would be helpful.  Many people have asked for an equivalent to a GCC
-Wall flag for GNU make that gives warnings about things that are
probably not doing what you expected.

Re: Potential Bug: `.PHONY` targets and order-only prerequisites

[Dmitry Goncharov]

On Wed, May 18, 2022 at 5:37 PM Jacob Kopczynski
 wrote:
>>   - it might make sense to state explicitly that order-only prereqs
>> which are phony behave in all ways like standard prerequisites.

This statement is simply incorrect.
Order-only prerequisites which are phony do not behave like regular
prerequisites.
An order-only prerequisite which is phony is rebuilt every time, but
still does not cause those targets, which depend on this order-only
phony prerequisite, to be rebuilt.

regards, Dmitry

Re: Potential Bug: `.PHONY` targets and order-only prerequisites

[Jacob Kopczynski]

The thing that the docs refer to as "impose order" is not a single thing,
but two. I would characterize a normal prerequisite as doing three things
rather than two:
- *update-marking: *cause a target to be marked out of date if the prereq
is marked out of date
- *require-existence: *require the prereq to be built successfully at least
once before the target is built
- *imposed-order: *require the prereq to be built before the target, if
both are being built
A prereq which truly was order-only would *exclusively* do the last one,
imposing order. The "order-only" prereqs as they stand do both
*imposed-order* and *require-existence*. For .PHONY targets,
*require-existence* always mandates *update-marking*, which is why the
overlap between those two cases is particularly confusing - a phony target
which drops only update-marking is a contradiction in terms, so the natural
assumption to make is that the effect of "order-only" prereqs is *not* that,
and is instead something non-paradoxical.

I looked deeper into the list archives and saw that this discussion has
come up before; Stefano Lattarini experienced the same problem in 2012
 (thread
starts 2012/06/11). Phillip Guenther said
 that
this pure-order effect is not achievable with the structure of make, which
presumably remains true.

I agree with Lattarini's recommendations, which I'll reproduce below for
convenience. The primary one was the suggestion of alternate names that
might be used for what are currently called order-only prerequisites -
"existence-only prerequisites" seems like the clearest one to me.

Thanks,
Jacob

Lattarini:

>   - the name "order-only prerequisites" is IMHO poorly chosen, and don't
> really correspond to their actual semantics; a more appropriate might
> be "weak prerequisites" or "existence-only prerequisites";
>   - the explanation given in the manual is imprecise and confusing; since
> the explanation you've given here is OTOH very clear and concise, I
> suggest you report it in the manual;
>   - it might make sense to state explicitly that order-only prereqs
> which are phony behave in all ways like standard prerequisites.

The referenced 'explanation you've given here' is from Paul Smith -
presumably the same one that has responded to me? It was

> No, that's not how order-only prerequisites work.  An order-only
> prerequisite is treated identically to a normal prerequisite except for
> exactly one thing: when make checks to see if a target needs to be
> remade, the time-last-modified values for any order-only prerequisites
> are not considered.


On Wed, May 18, 2022 at 11:14 AM Paul Smith  wrote:

> On Wed, 2022-05-18 at 10:22 -0700, Jacob Kopczynski wrote:
> > I believe I understand. The name "order-only" is highly misleading
> > and should be changed - it does considerably more than "only"
> > "order"; the only thing it does not do is check the timestamp.
>
> As described in the docs there are only two things a prerequisite can
> do: impose order and effect the out-of-date decision of the target.
> Order-only prerequisites do one of them (impose order) and don't do the
> other (effect out-of-date decisions).
>
> What are the "considerably more" things that order-only prerequisites
> do, besides impose order?
>
> I do agree that the docs need clarification though.
>

Re: Potential Bug: `.PHONY` targets and order-only prerequisites

[Paul Smith]

On Wed, 2022-05-18 at 10:22 -0700, Jacob Kopczynski wrote:
> I believe I understand. The name "order-only" is highly misleading
> and should be changed - it does considerably more than "only"
> "order"; the only thing it does not do is check the timestamp.

As described in the docs there are only two things a prerequisite can
do: impose order and effect the out-of-date decision of the target. 
Order-only prerequisites do one of them (impose order) and don't do the
other (effect out-of-date decisions).

What are the "considerably more" things that order-only prerequisites
do, besides impose order?

I do agree that the docs need clarification though.

Re: Potential Bug: `.PHONY` targets and order-only prerequisites

[Jacob Kopczynski]

I believe I understand. The name "order-only" is highly misleading and
should be changed - it does considerably more than "only" "order"; the only
thing it does not do is check the timestamp. This portion of the
documentation *particularly* needs changing:
> Occasionally, however, you have a situation where you want to impose
> a specific ordering on the rules to be invoked without forcing the
> target to be updated if one of those rules is executed. In that case,
> you want to define order-only prerequisites.
The natural reading of this strongly implies that an order-only
prerequisite literally only affects the order of things. I would suggest
> Occasionally, however, you have a situation where you do not want to
> force the target to be updated if a rule is executed, but you do want to
> ensure that rule is executed before the target every time the target is
> built. In that case, you want to define order-only prerequisites.

Thanks,
Jacob

On Tue, May 17, 2022 at 3:40 PM Paul Smith  wrote:

> On Tue, 2022-05-17 at 22:32 +, Martin Dorey wrote:
> > >  all your targets are .PHONY, and thus are always rebuilt anyway
> >
> > If you "make down", the rule for "down-clean" doesn't run.  They're
> > only rebuilt if something causes them to be considered.
> >
> > >  order-only prerequisites are totally irrelevant and have no impact
> > > on
> > > your makefile.
> >
> > If you comment-out the order-only prerequisite that says that the up
> > target depends on down-clean, then the recipe for down-clean doesn't
> > get run when you "make up".
>
> Sure, of course.
>
> What I was trying to say was, any rule that would normally be run will
> be run regardless of order-only or not, because all the targets in the
> makefile are phony.  So adding or removing the order-only operator in a
> prerequisites list makes no difference to how the targets will be
> processed, in this specific makefile.
>

Re: Potential Bug: `.PHONY` targets and order-only prerequisites

[Paul Smith]

On Tue, 2022-05-17 at 22:32 +, Martin Dorey wrote:
> >  all your targets are .PHONY, and thus are always rebuilt anyway
> 
> If you "make down", the rule for "down-clean" doesn't run.  They're
> only rebuilt if something causes them to be considered.
> 
> >  order-only prerequisites are totally irrelevant and have no impact
> > on
> > your makefile.
> 
> If you comment-out the order-only prerequisite that says that the up
> target depends on down-clean, then the recipe for down-clean doesn't
> get run when you "make up".

Sure, of course.

What I was trying to say was, any rule that would normally be run will
be run regardless of order-only or not, because all the targets in the
makefile are phony.  So adding or removing the order-only operator in a
prerequisites list makes no difference to how the targets will be
processed, in this specific makefile.

Re: Potential Bug: `.PHONY` targets and order-only prerequisites

[Martin Dorey]

> all your targets are .PHONY, and thus are always rebuilt anyway

If you "make down", the rule for "down-clean" doesn't run.  They're only 
rebuilt if something causes them to be considered.

> order-only prerequisites are totally irrelevant and have no impact on
> your makefile.

If you comment-out the order-only prerequisite that says that the up target 
depends on down-clean, then the recipe for down-clean doesn't get run when you 
"make up".


From: Bug-make  on behalf of 
Paul Smith 
Sent: Tuesday, May 17, 2022 14:20
To: Jacob Kopczynski ; bug-make@gnu.org 

Subject: Re: Potential Bug: `.PHONY` targets and order-only prerequisites

* EXTERNAL EMAIL *

On Tue, 2022-05-17 at 14:00 -0700, Jacob Kopczynski wrote:
> I'm unsure whether this is a bug or just undocumented, but I found a
> confusing interaction in a simple Makefile:

You are misreading the documentation.  I will quote:

> A normal prerequisite makes two statements: first, it imposes an
> order in which recipes will be invoked: the recipes for all
> prerequisites of a target will be completed before the recipe for the
> target is run. Second, it imposes a dependency relationship: if any
> prerequisite is newer than the target, then the target is considered
> out-of-date and must be rebuilt.

Hopefully that is clear.  To give an example, if you have:

  a: b c

this says two things: first, that b and c will both be rebuilt (if
necessary) before a's recipe is started, and second, that if either b
or c are newer than a (if a is out of date with respect to b or c),
then a will be considered out of date and rebuilt.

The docs go on to say:

> Occasionally, however, you have a situation where you want to impose
> a specific ordering on the rules to be invoked without forcing the
> target to be updated if one of those rules is executed. In that case,
> you want to define order-only prerequisites.

What this means is that if you have:

  a: b | c

then the ordering statement (that both b and c will be rebuilt (if
necessary) before a's recipe is started) is still in place for both
prerequisites b and c.

However, the second statement, that a is considered out of date if it's
older than the prerequisite, only is in effect for b, and not for c.

"c" is an "order-only" prerequisite in that it is ordered before "a",
but if it's updated then that doesn't mean "a" will be updated.

Since all your targets are .PHONY, and thus are always rebuilt anyway,
order-only prerequisites are totally irrelevant and have no impact on
your makefile.

Re: Potential Bug: `.PHONY` targets and order-only prerequisites

[Paul Smith]

On Tue, 2022-05-17 at 17:20 -0400, Paul Smith wrote:
> this says two things: first, that b and c will both be rebuilt (if
> necessary) before a's recipe is started,

I guess I should be more clear about the "(if necessary)".  What I mean
is the same as if you had run "make b" or "make c" and make had decided
it needed to rebuild those targets.

In other words, the "if necessary" has nothing to do with "a" or
whether "a" needs to be rebuilt.  It only relates to "b" and "c" and
whether they need to be rebuilt, on their own.  If those targets don't
exist, or are phony, or they have prerequisites which cause them to be
out of date, they will be rebuilt.  If they exist, are not phony, and
have no prerequisites that cause them to be out of date, they will not
be rebuilt.

Once both "b" and "c" are completed, then make will decide whether "a"
needs to be rebuilt.  For that decision make will ignore any order-only
prerequisites and only consider the normal prerequisites.

Re: Potential Bug: `.PHONY` targets and order-only prerequisites

[Paul Smith]

On Tue, 2022-05-17 at 14:00 -0700, Jacob Kopczynski wrote:
> I'm unsure whether this is a bug or just undocumented, but I found a
> confusing interaction in a simple Makefile:

You are misreading the documentation.  I will quote:

> A normal prerequisite makes two statements: first, it imposes an
> order in which recipes will be invoked: the recipes for all
> prerequisites of a target will be completed before the recipe for the
> target is run. Second, it imposes a dependency relationship: if any
> prerequisite is newer than the target, then the target is considered
> out-of-date and must be rebuilt.

Hopefully that is clear.  To give an example, if you have:

  a: b c

this says two things: first, that b and c will both be rebuilt (if
necessary) before a's recipe is started, and second, that if either b
or c are newer than a (if a is out of date with respect to b or c),
then a will be considered out of date and rebuilt.

The docs go on to say:

> Occasionally, however, you have a situation where you want to impose
> a specific ordering on the rules to be invoked without forcing the
> target to be updated if one of those rules is executed. In that case,
> you want to define order-only prerequisites.

What this means is that if you have:

  a: b | c

then the ordering statement (that both b and c will be rebuilt (if
necessary) before a's recipe is started) is still in place for both
prerequisites b and c.

However, the second statement, that a is considered out of date if it's
older than the prerequisite, only is in effect for b, and not for c.

"c" is an "order-only" prerequisite in that it is ordered before "a",
but if it's updated then that doesn't mean "a" will be updated.

Since all your targets are .PHONY, and thus are always rebuilt anyway,
order-only prerequisites are totally irrelevant and have no impact on
your makefile.