Re: New feature: Post-requisites

[Alejandro Colomar (man-pages)]

Hi Paul

On 7/31/22 14:36, Paul Smith wrote:

On Sun, 2022-07-31 at 11:45 +0200, Alejandro Colomar (man-pages) wrote:

If some new post-requisite "baz" was found while building either
"foo" or "bar", it would be built after those and be as if the user
had invoked "make foo bar baz".


What if one runs `make -j foo foo2` and both foo and foo2 have a
post-requisite of bar?  `make -j foo foo2 bar` doesn't guarantee that
bar will be started after foo2 ends, but this feature would need such
a guarantee.


Well, what I was really thinking was that the postrequisites would go
into their own list.  So the set of "current" goals would be in one
list and the postrequisites would be added to a new list.  Once all the
current goals were completed, then make would start working on the new
list, and any further postrequisites would be added to a new new list.
Etc. until the "new list" was empty (no more postrequisites were added
during the last set of builds).


That makes sense to me.  The performance is suboptimal, I think, since 
it needs to wait for all "current" goals even if there was no actual 
dependency to some of them.  But if the implementation is much simpler, 
I think it's a good tradeoff.  Looks good to me.


The ideal solution would be to extend the DAG in some way that includes 
the post-requisites, but that would need an architectural change.


Cheers,

Alex

--
Alejandro Colomar
Linux man-pages comaintainer; http://www.kernel.org/doc/man-pages/
http://www.alejandro-colomar.es/

Re: New feature: Post-requisites

[Paul Smith]

On Sun, 2022-07-31 at 11:45 +0200, Alejandro Colomar (man-pages) wrote:
> > If some new post-requisite "baz" was found while building either
> > "foo" or "bar", it would be built after those and be as if the user
> > had invoked "make foo bar baz".
> 
> What if one runs `make -j foo foo2` and both foo and foo2 have a 
> post-requisite of bar?  `make -j foo foo2 bar` doesn't guarantee that
> bar will be started after foo2 ends, but this feature would need such
> a guarantee.

Well, what I was really thinking was that the postrequisites would go
into their own list.  So the set of "current" goals would be in one
list and the postrequisites would be added to a new list.  Once all the
current goals were completed, then make would start working on the new
list, and any further postrequisites would be added to a new new list.
Etc. until the "new list" was empty (no more postrequisites were added
during the last set of builds).

Re: New feature: Post-requisites

[Alejandro Colomar (man-pages)]

Hi Paul

On 7/30/22 19:01, Paul Smith wrote:

On Mon, 2022-07-25 at 09:06 +, Edward Welbourne wrote:

On 7/22/22 16:20, Paul Smith wrote:

So, after all the "normal" goal targets were completed make would
proceed to run any "extra" goal targets that were added as part
of the processing of the "normal" goal targets.  Then after the
those "extra" goal targets were complete, make would proceed to
run any "new extra" goal targets that were added as part of THAT
processing.


How would any new targets be added ?  Surely make's dependency graph
would ensure that you know what you'll be building before you start,
regardless of whether it's a prerequisite or a postrequisite.


I think I wasn't clear.  I wasn't suggesting that "post-requisites"
would be represented in the dependency graph in any way at all.

When make runs it basically does this: it first generates the DAG.
Then it has a list of one or more goal targets (either the default
target or one or more targets from the command line) and make starts
walking the DAG using each of these goal targets in turn as the
starting node.  This behavior is already well-understood.

What I'm suggesting is that "post-requisites" simply be added to the
end of the list of goal targets, as they are encountered while walking
the DAG.

So if you ran "make foo" and some target that was processed during the
walk of the DAG to build "foo" had a post-requisite of "bar", then
after "foo" was built make would try to build "bar".  It would be
essentially exactly the same as if the user had invoked "make foo bar".

If some new post-requisite "baz" was found while building either "foo"
or "bar", it would be built after those and be as if the user had
invoked "make foo bar baz".


What if one runs `make -j foo foo2` and both foo and foo2 have a 
post-requisite of bar?  `make -j foo foo2 bar` doesn't guarantee that 
bar will be started after foo2 ends, but this feature would need such a 
guarantee.




Etc.  If a target was not encountered during the walk, then of course
its post-requisites would not be added to the set of goal targets.


Indeed - for example, if a target is specified *both* as a
prerequisite of some target that does get built *and* as a
postrequisite of some rule we do exercise, it has to be built both
before and after, which violates the "only build once" rule above.


No.  It would be built before as a prerequisite, then when it was
attempted to be built after it would be skipped because it was already
built.


Makes sense for now.  It would be rare to make a post-requisite target 
be also a pre-requisite.  I'm not sure I can think of a use case for that.




That is _exactly_ the kind situation I was referring to by my statement
about "only build things once".

In other words, a target "foo" with a post-requisite "bar" does NOT say
that make will guarantee that "bar" is always built after "foo".  What
it ensures is that if "foo" is built, then "bar" will also be tried
afterwards.

It's possible that make could be taught to detect a situation like
this:

   foo: bar  baz

   bar: baz

and give an error.  But I'd have to play with it to determine how much
work it would be to catch that.


Cheers,

Alex

--
Alejandro Colomar
Linux man-pages comaintainer; http://www.kernel.org/doc/man-pages/
http://www.alejandro-colomar.es/

Re: New feature: Post-requisites

[Paul Smith]

On Mon, 2022-07-25 at 09:06 +, Edward Welbourne wrote:
> On 7/22/22 16:20, Paul Smith wrote:
> > > So, after all the "normal" goal targets were completed make would
> > > proceed to run any "extra" goal targets that were added as part
> > > of the processing of the "normal" goal targets.  Then after the
> > > those "extra" goal targets were complete, make would proceed to
> > > run any "new extra" goal targets that were added as part of THAT
> > > processing.
> 
> How would any new targets be added ?  Surely make's dependency graph
> would ensure that you know what you'll be building before you start,
> regardless of whether it's a prerequisite or a postrequisite.

I think I wasn't clear.  I wasn't suggesting that "post-requisites"
would be represented in the dependency graph in any way at all.

When make runs it basically does this: it first generates the DAG. 
Then it has a list of one or more goal targets (either the default
target or one or more targets from the command line) and make starts
walking the DAG using each of these goal targets in turn as the
starting node.  This behavior is already well-understood.

What I'm suggesting is that "post-requisites" simply be added to the
end of the list of goal targets, as they are encountered while walking
the DAG.

So if you ran "make foo" and some target that was processed during the
walk of the DAG to build "foo" had a post-requisite of "bar", then
after "foo" was built make would try to build "bar".  It would be
essentially exactly the same as if the user had invoked "make foo bar".

If some new post-requisite "baz" was found while building either "foo"
or "bar", it would be built after those and be as if the user had
invoked "make foo bar baz".

Etc.  If a target was not encountered during the walk, then of course
its post-requisites would not be added to the set of goal targets.

> Indeed - for example, if a target is specified *both* as a
> prerequisite of some target that does get built *and* as a
> postrequisite of some rule we do exercise, it has to be built both
> before and after, which violates the "only build once" rule above.

No.  It would be built before as a prerequisite, then when it was
attempted to be built after it would be skipped because it was already
built.

That is _exactly_ the kind situation I was referring to by my statement
about "only build things once".

In other words, a target "foo" with a post-requisite "bar" does NOT say
that make will guarantee that "bar" is always built after "foo".  What
it ensures is that if "foo" is built, then "bar" will also be tried
afterwards.

It's possible that make could be taught to detect a situation like
this:

  foo: bar  baz

  bar: baz

and give an error.  But I'd have to play with it to determine how much
work it would be to catch that.

Re: New feature: Post-requisites

[Edward Welbourne]

On 7/25/22 11:06, Edward Welbourne wrote:
>> Also, this new feature smells like it may open up some interesting
>> corner-cases in dependency-loop analysis, although I can't think of
>> any immediately.

Alejandro Colomar (Monday, July 25, 2022 13:54) replied:
> The same way that having dep loops right now is an error, I'd just
> throw an error if a post-requisite creates a dependency loop.

oh, absolutely - I wasn't suggesting this was a case against the
proposal - only pointing it out as one more area that's going to need a
bit more thought, to look for new and interesting ways that dependency
loops can arise.  How those get handled once detected is, of course,
business as usual.

Eddy.

Re: New feature: Post-requisites

[Alejandro Colomar]

Hi Edward,

On 7/25/22 11:06, Edward Welbourne wrote:

On 7/22/22 16:20, Paul Smith wrote:

So, after all the "normal" goal targets were completed make would
proceed to run any "extra" goal targets that were added as part of
the processing of the "normal" goal targets.  Then after the those
"extra" goal targets were complete, make would proceed to run any
"new extra" goal targets that were added as part of THAT processing.
Etc.


How would any new targets be added ?  Surely make's dependency graph
would ensure that you know what you'll be building before you start,
regardless of whether it's a prerequisite or a postrequisite.


Since of course we never build the same target twice in a single
instance of make, this cannot result in an infinite loop.


But for the dependency graph, this would be a problem: if it were
possible to discover a need for a postrequisite during processing of
postrequisites, and that postrequisite had already been built (because
somehow make was unaware that it would be needed later), then we'd
either built it twice (or more) or fail to build it after a target
(triggered by a postrequisite, possibly itself) queued it to be built
because we need its command to be run after the one that asked for it.


However this would need a more detailed specification.


Indeed - for example, if a target is specified *both* as a prerequisite
of some target that does get built *and* as a postrequisite of some rule
we do exercise, it has to be built both before and after, which violates
the "only build once" rule above.  However, as long as the dependency
tree does in fact enable you to know before you start what you're going
to build, such repeat building should not be a problem.  It is built
before the first target that has it as a prerequisite and after the last
that has it as a postrequisite.

Also, this new feature smells like it may open up some interesting
corner-cases in dependency-loop analysis, although I can't think of any
immediately.


The same way that having dep loops right now is an error, I'd just throw 
an error if a post-requisite creates a dependency loop.


Cheers,

Alex

--
Alejandro Colomar



OpenPGP_signature
Description: OpenPGP digital signature

Re: New feature: Post-requisites

[Edward Welbourne]

On 7/22/22 16:20, Paul Smith wrote:
>> So, after all the "normal" goal targets were completed make would
>> proceed to run any "extra" goal targets that were added as part of
>> the processing of the "normal" goal targets.  Then after the those
>> "extra" goal targets were complete, make would proceed to run any
>> "new extra" goal targets that were added as part of THAT processing. 
>> Etc.

How would any new targets be added ?  Surely make's dependency graph
would ensure that you know what you'll be building before you start,
regardless of whether it's a prerequisite or a postrequisite.

>> Since of course we never build the same target twice in a single
>> instance of make, this cannot result in an infinite loop.

But for the dependency graph, this would be a problem: if it were
possible to discover a need for a postrequisite during processing of
postrequisites, and that postrequisite had already been built (because
somehow make was unaware that it would be needed later), then we'd
either built it twice (or more) or fail to build it after a target
(triggered by a postrequisite, possibly itself) queued it to be built
because we need its command to be run after the one that asked for it.

>> However this would need a more detailed specification.

Indeed - for example, if a target is specified *both* as a prerequisite
of some target that does get built *and* as a postrequisite of some rule
we do exercise, it has to be built both before and after, which violates
the "only build once" rule above.  However, as long as the dependency
tree does in fact enable you to know before you start what you're going
to build, such repeat building should not be a problem.  It is built
before the first target that has it as a prerequisite and after the last
that has it as a postrequisite.

Also, this new feature smells like it may open up some interesting
corner-cases in dependency-loop analysis, although I can't think of any
immediately.

>> For example does a "post-requisite" get built only if its target is
>> considered out of date and its recipe invoked? Or does it get built
>> if the target is considered, regardless of whether it's up-to-date?

The logic that motivates the feature, at least, argues for the
postrequisite to only be queued for refresh if make runs the command to
rebuild the target that has it as post-requisite.  If we don't install
the new library, makefile, , we don't need to run the "update my
system's knowledge of that" command.

The alternative, in any case, amounts to saying that whenever make even
checks that a target is up to date (and even if it is) we must run all
the post-conditions of anything it depends on.  That sounds like a lot
of wanton re-building on a no-op; and better handled by having a script
that runs those commands, that you can run independently of make, or
that might wrap make to check things are up to date and then
unconditionally run a bunch of post-make commands.

>> Also, is it up to make to FORCE the post-requisite to be built, if
>> its target was built?  Or is it only rebuilt if it's considered out
>> of date in its own right as a normal target?

Alejandro Colomar (Friday, July 22, 2022 16:38) replied:
> I'd say that post-requisites need only be build if they are not
> up-to-date.  Let the user tell make(1) that a post-requisite needs to
> be build regardless of its up-to-date status by marking it .PHONY or
> FORCE.

+1 to that.

Eddy.

Re: New feature: Post-requisites

[Alejandro Colomar]

Hi Paul,

On 7/22/22 16:20, Paul Smith wrote:

On Fri, 2022-07-22 at 14:05 +0200, Alejandro Colomar wrote:

As the word suggests, as pre-requisites are satisfied before a given
target, post-requisites would be satisfied _after_ a given target.


The way I can see this being implemented without a very significant
amount of work, would be for make to dynamically add goal targets based
on some internal syntax such as you describe (I'm not sure I like the
">" token however... needs thought).


I chose '>' because it doesn't seem to be in use by GNU make.  Also, 
there's no $> automatic variable yet in GNU make (I ignore other make(1) 
implementations), which could be added to contain the post-requisites of 
a given target

.


So, after all the "normal" goal targets were completed make would
proceed to run any "extra" goal targets that were added as part of the
processing of the "normal" goal targets.  Then after the those "extra"
goal targets were complete, make would proceed to run any "new extra"
goal targets that were added as part of THAT processing.  Etc.

Since of course we never build the same target twice in a single
instance of make, this cannot result in an infinite loop.

However this would need a more detailed specification.  For example
does a "post-requisite" get built only if its target is considered out
of date and its recipe invoked?  Or does it get built if the target is
considered, regardless of whether it's up-to-date?

Also, is it up to make to FORCE the post-requisite to be built, if its
target was built?  Or is it only rebuilt if it's considered out of date
in its own right as a normal target?


I'd say that post-requisites need only be build if they are not 
up-to-date.  Let the user tell make(1) that a post-requisite needs to be 
build regardless of its up-to-date status by marking it .PHONY or FORCE.


Cheers,

Alex

--
Alejandro Colomar



OpenPGP_signature
Description: OpenPGP digital signature

Re: New feature: Post-requisites

[Dmitry Goncharov]

On Fri, Jul 22, 2022 at 9:34 AM  wrote:
> In your example, wouldn't the following accomplish the same thing?
>
>
> update-mandb:  $(DESTDIR)$(man3dir)/foo.3 $(DESTDIR)$(man3dir)/bar.3

i'd also be interested in the answer.
Either this traditional syntax specified above, or

$(DESTDIR)$(man3dir)/bar.3: man3/bar.3 | $(DESTDIR)$(man3dir)
$(info INSTALL  $@)
@$(INSTALL_DATA) -T $< $@
@mandb

regards, Dmitry

Re: New feature: Post-requisites

[Paul Smith]

On Fri, 2022-07-22 at 14:05 +0200, Alejandro Colomar wrote:
> As the word suggests, as pre-requisites are satisfied before a given 
> target, post-requisites would be satisfied _after_ a given target.

The way I can see this being implemented without a very significant
amount of work, would be for make to dynamically add goal targets based
on some internal syntax such as you describe (I'm not sure I like the
">" token however... needs thought).

So, after all the "normal" goal targets were completed make would
proceed to run any "extra" goal targets that were added as part of the
processing of the "normal" goal targets.  Then after the those "extra"
goal targets were complete, make would proceed to run any "new extra"
goal targets that were added as part of THAT processing.  Etc.

Since of course we never build the same target twice in a single
instance of make, this cannot result in an infinite loop.

However this would need a more detailed specification.  For example
does a "post-requisite" get built only if its target is considered out
of date and its recipe invoked?  Or does it get built if the target is
considered, regardless of whether it's up-to-date?

Also, is it up to make to FORCE the post-requisite to be built, if its
target was built?  Or is it only rebuilt if it's considered out of date
in its own right as a normal target?

Re: New feature: Post-requisites

[Edward Welbourne]

th...@vmware.com (22 July 2022 15:28) wrote:
> In your example, wouldn't the following accomplish the same thing?
>
> update-mandb:  $(DESTDIR)$(man3dir)/foo.3 $(DESTDIR)$(man3dir)/bar.3

I presume that'd have a rule that runs mandb.  And it works as long as
you're happy to update all man pages whenever you want to update any man
page.

However, if I only want to update perlsyntax.1 and not the rest of the
perl man pages, and I'm doing it this way, I end up with a mess whose
simplified form, for your two pages, looks like:

install: install-other-stuff update-mandb

install-mandb: $(DESTDIR)$(man3dir)/foo.3 $(DESTDIR)$(man3dir)/bar.3
mandb

install-man-foo: $(DESTDIR)$(man3dir)/foo.3
mandb

install-man-bar: $(DESTDIR)$(man3dir)/bar.3
mandb

with rules to build $(DESTDIR) things taking care of actual installing.
and, in the case of a big library of man pages, I'm probably going to
want a separate rule for each of various subsets of the pages, that
installs the subset and runs mandb, because if I try to

$ make install-man-perl-toc install-man-perl-unicode install-man-perlvar

I'm going to run mandb three times, when I'd rather have run it only
once; and my only alternative will be to make install-mandb and install
*all* of my library of man pages, which "may take some time".

So the traditional approach you're proposing does not scale well,

Eddy.

Re: New feature: Post-requisites

[Alejandro Colomar]

Hi Edward, thutt,

On 7/22/22 15:36, Edward Welbourne wrote:

Alejandro Colomar (Friday, July 22, 2022 14:05)
 > I'd like to suggest a new feature: post-requisites.
 >
 > As the word suggests, as pre-requisites are satisfied before a given
 > target, post-requisites would be satisfied _after_ a given target.

and just as a prerequisite can be built at any time before, but must be
built before, the post-requisite can be run at any time after, but must
be built after.  So if several targets that are being built require some
given post-requisite, it waits until all of them have been built.


Exactly.  I didn't word it, but I meant it :)



 > Why would someone ever need that in a Makefile:
 >
 > In an install target for a library, you may want to run ldconfig(8)
 > after any install targets have executed.
 >
 > In an install target for the linux man-pages, you may want to run
 > mandb(8)  after any pages have been installed in the system.

I think the traditional solution to this would be to make the install
depend on rules that get individual files installed and have its command
run the post-install steps, or have the install rule depend on a set of
post-install rules (each of which does one post-install task), that in
turn depend on all the files whose updates would trigger them.  However,
the problem with that is that if I want to just update the one man page,
the mandb(8) run that's needed following it won't be run unless I build
the mandb-run rule, that depends on *all* my man pages.  So I can't do
fine-grained targeted "build this one thing" parts of my build process.


Yeah, those are the current workarounds to this missing feature.  As you 
explained, they are very suboptimal.




Post-requisites would take care of that; if I build and install an
update to any man page, then mandb gets run, regardless of what else I
am also building, aside from mandb's run happening after any of those
that need it to be done later.


thutt, this answers your question.



 > The proposed syntax would be similar to that of order only
 > prerequisites, but using '>' instead of '|':
 >
 > [
 > [...]
 >
 > $(DESTDIR)$(man3dir)/foo.3: man3/foo.3 | $(DESTDIR)$(man3dir) > 
update-mandb

 >     $(info INSTALL  $@)
 >     @$(INSTALL_DATA) -T $< $@
 >
 > $(DESTDIR)$(man3dir)/bar.3: man3/bar.3 | $(DESTDIR)$(man3dir) > 
update-mandb

 >     $(info INSTALL  $@)
 >     @$(INSTALL_DATA) -T $< $@
 >
 > .PHONY: update-mandb
 > update-mandb:
 >     $(info MANDB)
 >     @mandb
 >
 > .PHONY: install
 > install: $(DESTDIR)$(man3dir)/foo.3 $(DESTDIR)$(man3dir)/bar.3
 > ]
 >
 > So that `make install` would install foo.3 and bar.3, and after all of
 > the pages have been installed, mandb(8) would be run.
 > And `make /usr/local/share/man/man3/foo.3` would install foo.3, and
 > after it, it would run mandb(8).
 >
 > Does that make sense to you?

and if my package (e.g. perl) has a whole library of man pages, I'd be
able to do updates to just one (or a few) of them at a time and get
mandb run after that update, regardless of which subset of pages I
update, rather than only being able to get that run when I install the
whole library.

I guess it would also be useful for tidying away scratch directories and
the like, used by rules for their transient or intermediate files.

It sounds like a neat idea.  I wonder how practical it would be to
implement ...


Yep, that's the idea.  I hope it's as easy to implement as it's to describe.

Cheers,

Alex



  Eddy.



--
Alejandro Colomar



OpenPGP_signature
Description: OpenPGP digital signature

Re: New feature: Post-requisites

[Edward Welbourne]

Alejandro Colomar (Friday, July 22, 2022 14:05)
> I'd like to suggest a new feature: post-requisites.
>
> As the word suggests, as pre-requisites are satisfied before a given
> target, post-requisites would be satisfied _after_ a given target.

and just as a prerequisite can be built at any time before, but must be
built before, the post-requisite can be run at any time after, but must
be built after.  So if several targets that are being built require some
given post-requisite, it waits until all of them have been built.

> Why would someone ever need that in a Makefile:
>
> In an install target for a library, you may want to run ldconfig(8)
> after any install targets have executed.
>
> In an install target for the linux man-pages, you may want to run
> mandb(8)  after any pages have been installed in the system.

I think the traditional solution to this would be to make the install
depend on rules that get individual files installed and have its command
run the post-install steps, or have the install rule depend on a set of
post-install rules (each of which does one post-install task), that in
turn depend on all the files whose updates would trigger them.  However,
the problem with that is that if I want to just update the one man page,
the mandb(8) run that's needed following it won't be run unless I build
the mandb-run rule, that depends on *all* my man pages.  So I can't do
fine-grained targeted "build this one thing" parts of my build process.

Post-requisites would take care of that; if I build and install an
update to any man page, then mandb gets run, regardless of what else I
am also building, aside from mandb's run happening after any of those
that need it to be done later.

> The proposed syntax would be similar to that of order only
> prerequisites, but using '>' instead of '|':
>
> [
> [...]
>
> $(DESTDIR)$(man3dir)/foo.3: man3/foo.3 | $(DESTDIR)$(man3dir) > update-mandb
> $(info INSTALL  $@)
> @$(INSTALL_DATA) -T $< $@
>
> $(DESTDIR)$(man3dir)/bar.3: man3/bar.3 | $(DESTDIR)$(man3dir) > update-mandb
> $(info INSTALL  $@)
> @$(INSTALL_DATA) -T $< $@
>
> .PHONY: update-mandb
> update-mandb:
> $(info MANDB)
> @mandb
>
> .PHONY: install
> install: $(DESTDIR)$(man3dir)/foo.3 $(DESTDIR)$(man3dir)/bar.3
> ]
>
> So that `make install` would install foo.3 and bar.3, and after all of
> the pages have been installed, mandb(8) would be run.
> And `make /usr/local/share/man/man3/foo.3` would install foo.3, and
> after it, it would run mandb(8).
>
> Does that make sense to you?

and if my package (e.g. perl) has a whole library of man pages, I'd be
able to do updates to just one (or a few) of them at a time and get
mandb run after that update, regardless of which subset of pages I
update, rather than only being able to get that run when I install the
whole library.

I guess it would also be useful for tidying away scratch directories and
the like, used by rules for their transient or intermediate files.

It sounds like a neat idea.  I wonder how practical it would be to
implement ...

  Eddy.