Re: make pattern rules (was: Re: [Issue 8 drafts 0001325]: Allow make to remake an included file)
Joerg Schilling wrote, on 09 Nov 2020: > > Geoff Clare via austin-group-l at The Open Group > wrote: > > > Paul Smith wrote, on 07 Nov 2020: > > > > > > Unless we are proposing pattern rules for inclusion in POSIX, [...] > > > > This was proposed in 2011 in bug 513, which is still open. > [...] > > Do you believe we have a chance to get to a text for Issue-8? As with any enhancement request that doesn't need a sponsor, all it needs is for someone to propose some suitable, detailed wording changes and for those changes to be accepted (perhaps after some modifications) in a teleconference. -- Geoff Clare The Open Group, Apex Plaza, Forbury Road, Reading, RG1 1AX, England
Re: make pattern rules (was: Re: [Issue 8 drafts 0001325]: Allow make to remake an included file)
On Mon, 2020-11-09 at 14:16 +0100, Joerg Schilling wrote: > Paul Smith via austin-group-l at The Open Group < > austin-group-l@opengroup.org> wrote: > My idea is to start with a subset of the features of the original > implementation and to add some commonly *agreed* new features. Sure. > > I don't understand the distinction you're trying to draw between > > "disabled" and "deleted". Regardless, if you define a pattern rule > > with no commands then the previous pattern is gone and cannot be > > "re- > > enabled", so I think "deleted" is the correct term. > > Well, the fact that gmake still lists a "deleted" rule with gmake -p > is confusing users. I don't agree: it tells users that someone deleted that rule which I think is a useful thing to know. It's clear from the fact that no commands are shown, that this pattern cannot be used. > > Yes, that could be done. > > > > It's different than MAKEFLAGS += -r, however, because the latter is > > guaranteed to only delete the _built-in_ rules. This is almost > > always what the user wants. > > As mentioned before: internal rules need to be parsed before reading > the users Makefile, so a line in the form > > MAKEFLAGS += -r > > in the users Makefile is inefficient unless you ignore the rules for > the precedence of definitions that are in POSIX. I'm afraid I don't see the issue. It's not hard to keep a flag in any default rule specifying that it was created as a default, and either delete them all or just ignore them when searching rules, if they are disabled. Or, depending on the implementation, you can keep the default rules in a separate list and not search that list if the default rules are disabled. There are multiple implementation possibilities, which are not inefficient. > > Having "%:%" delete ALL the existing pattern rules is problematic > > because it relies on a specific ordering of appearance in > > makefiles. I'm sure such a feature would lead to a lot of head- > > scratching followed by cursing once a stray "%:%" pattern was > > discovered down in the bowels of the makefile system. > > Given that pattern rules _are_ based on the concept of a specific > ordering, this is a perfect enhancement. I disagree. The thing that users want is to remove the built-in rules. I see no reason why we should force them to add this special line as the first thing in their makefile to be sure they don't accidentally remove their own pattern rules. We don't require people to set .SUFFIXES: as the first thing in their makefile, before they define any suffix rules, for example. > > I would prefer to use a special target which forced all built-in / > > system rules (only) to be dropped, something like: > > > > .NOBUILTINS: > > This may be a useful potential enhancement, but this would still > differ from > > $make -r Since we are making it up there's no reason that it needs to differ from "make -r": we can just say that it doesn't :). > since .NOBUILTINS: would only affect rules from the internal > makefile, but not disable macro defitions. "make -r" does not disable macro definitions either. The standard says: > -r > Clear the suffix list and do not use the built-in rules. That does not mention macros anywhere, and in fact GNU make does not remove default macro definitions when given -r. This is the expected behavior because often makefiles will use the default macro definitions when defining their own rules, even if they are invoked with "make -r".
Re: make pattern rules (was: Re: [Issue 8 drafts 0001325]: Allow make to remake an included file)
Paul Smith via austin-group-l at The Open Group wrote: > Unless we are proposing pattern rules for inclusion in POSIX, which I > personally would be in favor of since they are much more useful than > suffix rules, perhaps we should move the discussion to a different > list? I would like to do this, but I am of course not willing to have any potential incompatible changes from GNU make. > On Fri, 2020-11-06 at 13:25 +0100, Joerg Schilling wrote: > > > I'm not sure where the idea that pattern rules are immutable comes > > > from: in GNU make any pattern rule can be overwritten at any time, > > > including the default pattern rules. > > > > This is how pattern rules have been defined when they appeared in > > SunPro Make in January 1986. > > Well, it's not how pattern rules in GNU make work. > > > The fact that gmake overwrites existing pattern rules is not > > sufficient (as it does not permit to remove them) > > That is not the case. You can absolutely remove them. The accepted way on UNIX for copying ideas from other implementations is to create a compatible clone with no "incompatible enhancements". My idea is to start with a subset of the features of the original implementation and to add some commonly *agreed* new features. This could be new features to remove single pattern rules or all pattern rules. > I don't understand the distinction you're trying to draw between > "disabled" and "deleted". Regardless, if you define a pattern rule > with no commands then the previous pattern is gone and cannot be "re- > enabled", so I think "deleted" is the correct term. Well, the fact that gmake still lists a "deleted" rule with gmake -p is confusing users. I just verified that gmake in fact correctly adds a new pattern rule with the same patterns of a deleted one to the end of the list. > Yes, that could be done. > > It's different than MAKEFLAGS += -r, however, because the latter is > guaranteed to only delete the _built-in_ rules. This is almost always > what the user wants. As mentioned before: internal rules need to be parsed before reading the users Makefile, so a line in the form MAKEFLAGS += -r in the users Makefile is inefficient unless you ignore the rules for the precedence of definitions that are in POSIX. > Having "%:%" delete ALL the existing pattern rules is problematic > because it relies on a specific ordering of appearance in makefiles. > I'm sure such a feature would lead to a lot of head-scratching followed > by cursing once a stray "%:%" pattern was discovered down in the bowels > of the makefile system. Given that pattern rules _are_ based on the concept of a specific ordering, this is a perfect enhancement. People who may get problems with that, are obviously using unplanned Makefiles. > I would prefer to use a special target which forced all built-in / > system rules (only) to be dropped, something like: > > .NOBUILTINS: This may be a useful potential enhancement, but this would still differ from $make -r since .NOBUILTINS: would only affect rules from the internal makefile, but not disable macro defitions. Jörg -- EMail:jo...@schily.net(home) Jörg Schilling D-13353 Berlin Blog: http://schily.blogspot.com/ URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/'
Re: make pattern rules (was: Re: [Issue 8 drafts 0001325]: Allow make to remake an included file)
Geoff Clare via austin-group-l at The Open Group wrote: > Paul Smith wrote, on 07 Nov 2020: > > > > Unless we are proposing pattern rules for inclusion in POSIX, [...] > > This was proposed in 2011 in bug 513, which is still open. the problem with this bug report is that I got the impression, it started with the wrong intention. The important point is that you need pattern rules if you like to put compilation results into platform specific sub-directories. I like to get pattern rules into the standard by only requiring a subset of what has been available since January 1986. The documentation is currently on page 30 of the man page: http://schilytools.sourceforge.net/man/man1/make.1s.html I would only standardize the form: tp%ts: dp%ds rule and mention tp%ts: [dependency ...] dp%ds [dependency ...] rule only as a potential enhancement that is not part of the standard. Smake e.g. currently only supports: tp%ts: dp%ds rule and tp%ts: rule BTW: this reminds me that I need to add documentation for: tp%ts: rule Note that the behavior of SunPro Make was not changed since 1986 with regards to pattern matching rules. Do you believe we have a chance to get to a text for Issue-8? Jörg -- EMail:jo...@schily.net(home) Jörg Schilling D-13353 Berlin Blog: http://schily.blogspot.com/ URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/'
Re: make pattern rules (was: Re: [Issue 8 drafts 0001325]: Allow make to remake an included file)
Paul Smith wrote, on 07 Nov 2020: > > Unless we are proposing pattern rules for inclusion in POSIX, [...] This was proposed in 2011 in bug 513, which is still open. -- Geoff Clare The Open Group, Apex Plaza, Forbury Road, Reading, RG1 1AX, England
make pattern rules (was: Re: [Issue 8 drafts 0001325]: Allow make to remake an included file)
Unless we are proposing pattern rules for inclusion in POSIX, which I personally would be in favor of since they are much more useful than suffix rules, perhaps we should move the discussion to a different list? On Fri, 2020-11-06 at 13:25 +0100, Joerg Schilling wrote: > > I'm not sure where the idea that pattern rules are immutable comes > > from: in GNU make any pattern rule can be overwritten at any time, > > including the default pattern rules. > > This is how pattern rules have been defined when they appeared in > SunPro Make in January 1986. Well, it's not how pattern rules in GNU make work. > The fact that gmake overwrites existing pattern rules is not > sufficient (as it does not permit to remove them) That is not the case. You can absolutely remove them. > and it is a non-compliance related to the original definition so > this modification from gmake causes unpredictable behavior. There is non-compliance with the POSIX standard. There are differences of implementation in areas that are not covered by the standard. There is no such thing as non-compliance with areas that are not covered by the standard. > Well after some searching (documentation could be better), I > discovered that gmake seems to disable (not to delete) pattern rules > if a rule with the same pattern but no command is defined. This is > not compliant to the definition of how pattern rules work. I don't understand the distinction you're trying to draw between "disabled" and "deleted". Regardless, if you define a pattern rule with no commands then the previous pattern is gone and cannot be "re- enabled", so I think "deleted" is the correct term. I think the docs are pretty clear: https://www.gnu.org/software/make/manual/html_node/Canceling-Rules.html > This is non-compliant behavior Pattern rules are not part of the POSIX standard, so "non-compliant behavior" is not an appropriate term. I'm happy to discuss incompatibilities between SunPro make/smake and GNU make as long as the discussion proceeds from a position that they are two co-equal implementations each with their own long history and (certainly in the case of GNU make) vast user bases that rely on the existing behaviors. However, I don't have the energy for discussions that begin with the premise that everything that exists in SunPro make/smake is canonical and any behavior in GNU make that differs from that is "non-compliant". > The series: > > %.o: %.c > echo some command > > %.o: %.c > > could be defined to require to completely remove the first definition > from the chain of pattern rules. That is already the way GNU make works. Although I don't know what you mean by the "first definition"; there can only be one instance of a given pattern rule (note, not one instance of a given pattern _target_; obviously many pattern rules may be defined with the same target). > As a result, the series: > > %.o: %.c > echo some command > > %.o: %.c > > %.o: %.c > echo some new command > > could be defined as to first define a pattern rule, then to > completely remove it from the chain of pattern rules and to finaly > add a new rule with a different command to the current end of the > pattern rule chain, without being completely incompatible to the > existing pattern rule specs. There is no single "spec" for pattern rules, there is only how SunPro make handles pattern rules and how GNU make handles pattern rules. In GNU make there's no need to delete the pattern rule before you define a new one. You just define the new one and it replaces the existing one. I don't see the point in requiring people to delete the pattern rule first. What does that buy you? > To fix this problem, the nonsense pattern with no command definition: > > %: % > > could be defined as the method to completely remove all existing > pattern rules from the current chain of pattern rules. Yes, that could be done. It's different than MAKEFLAGS += -r, however, because the latter is guaranteed to only delete the _built-in_ rules. This is almost always what the user wants. Having "%:%" delete ALL the existing pattern rules is problematic because it relies on a specific ordering of appearance in makefiles. I'm sure such a feature would lead to a lot of head-scratching followed by cursing once a stray "%:%" pattern was discovered down in the bowels of the makefile system. Possibly even accidentally as the result of a variable expanding to empty unexpectedly... A number of the enhancements to GNU make are intended to reduce the reliance on order of definition as much as possible. Users do want and rely on these enhancements. I would prefer to use a special target which forced all built-in / system rules (only) to be dropped, something like: .NOBUILTINS: This would provide the identical behavior as "make -r", but without modifying MAKEFLAGS. This seems like something that could easily be added to the standard as well, without even considering pattern
Re: [Issue 8 drafts 0001325]: Allow make to remake an included file
Paul Smith via austin-group-l at The Open Group
wrote:
> On Thu, 2020-11-05 at 19:36 +0100, Joerg Schilling wrote:
> > .SUFFIXES:
> > .SUFFIXES: .o .c
> > .c.o:
> > echo foo
> >
> > In other words, a users makefile has full control over the "builtin"
> > default rules even without a need to use "make -r".
>
> Sure, it's easy to redefine a suffix rule. The issue in question was
> that Geoff wanted to not use the .c.o rule at all. He wanted to use
> other rules to build .o files in other ways, but the .c.o rule was
> interfering with it. There's no way to DELETE that .c.o rule other
> than -r.
Well, this indeed is a problem with the suffix rules. You need to be careful to
understand their working method correctly. But I expect people to either not
touch these rules at all or to learn how to use them first.
smake in former times had simple suffix rules this way:
.o: .c .s .l
$(CC) -c $(CFLAGS) $(CPPFLAGS) $<
$(AS) -o $*.o $(ASFLAGS) $<
$(LEX) $(LFLAGS) $<;$(CC) -c $(CFLAGS) lex.yy.c;rm lex.yy.c;mv lex.yy.o
$@
which is easier to understand, but I stopped to define them 21 years ago
because three methods for implicit rules (simple suffix, suffix and pattern)
most
likely are too much for users.
> In fact, GNU make allows users to disable built-in rules completely
> WITHOUT having to require users to add -r to the make command line,
> because it allows users to add this to their makefile:
>
> MAKEFLAGS += -r
I would not call this an advantage, since it is in conflict with all other
make implementations and people that may have seen this to work (how ever) in
gmake, will realize that this cannot work if you follow the lines of POSIX and
first read the internal rules and then the external makefiles. If a make
implementation reads the external makefile, it is thus too late to disable the
internal rules unless you introduce dirty tricks that are not compatible to the
POSIX definitions.
> and it does the right thing. Of course this is problematic as well
> because that is now passed to sub-make invocations which may not be
> what you want. However, in practice it is rarely a problem.
This is true, if you pass this to sub-makes, it is not too late...
> In fact, most of the built-in rules in GNU make ARE defined as suffix
> rules; see this from default.c:
>
> ".c.o",
> "$(COMPILE.c) $(OUTPUT_OPTION) $<",
I cannot confirm this, gmake at the same time defines
%.o: %.c
# recipe to execute (built-in):
$(COMPILE.c) $(OUTPUT_OPTION) $<
whis is a rule with precedence before suffix rules.
> However, GNU make also provides a number of built-in rules that cannot
> be expressed as suffix rules, such as these:
>
> /* RCS. */
> { "%", "%,v",
> "$(CHECKOUT,v)" },
> { "%", "RCS/%,v",
> "$(CHECKOUT,v)" },
> { "%", "RCS/%",
> "$(CHECKOUT,v)" },
Classical make implementation (and in this special case, this also applies to
SunPro Make) implement this hardcoded in C. This is why I so far have been
hesitant to define similar rules in smake. RCS is non-standard, dead and slower
than SCCS anyway, so this does not seem to be a problem ;-)
> > - pattern matching rules have precedence over suffix rules and may
> > themselves not be overwritten
> >
> > - pattern matching rules match in the order of their apperance to the
> > parser. The first pattern rule that matches a specific pattern is
> > used. Later specifying a replacement rule for a specific pattern is
> > impossible.
> >
> > - The built in default rules must to be "read" by make before the
> > users makefiles are parsed in order to allow them (the suffix
> > rules) to be overwritten. Any pattern matching rule that appears
> > in the builtin default rules thus would have precedence over
> > anything else and cannot be overwritten.
>
> Maybe these are problems in SunOS make / smake, but they are certainly
> not problems in GNU make.
>
> I'm not sure where the idea that pattern rules are immutable comes
> from: in GNU make any pattern rule can be overwritten at any time,
> including the default pattern rules.
This is how pattern rules have been defined when they appeared in SunPro Make
in January 1986.
If this was a problem, people could have made a bug report at that time but
this did not happen Given that pattern rules are not (neither in SunPro
Make, nor in smake) used in the built in rules, there is no need to have a
method to remove them later, since they are only used by the user makefiles.
Since I tend to avoid a hard coded implementation for SCCS support in smake, I
tend to introduce SCCS checkout support in a future version of smake using
pattern rules. This would introduce the need to be able to withdraw existing
pattern rules selectively.
The fact that gmake overwrites existing pattern rules is not sufficient (as it
does not permit to remove them) and it is a non-compliance related to the
original definition so
Re: [Issue 8 drafts 0001325]: Allow make to remake an included file
Paul Smith wrote, on 05 Nov 2020: > > On Thu, 2020-11-05 at 09:25 +, Geoff Clare via austin-group-l at > The Open Group wrote: > > > .SUFFIXES: > > > .SUFFIXES: .c .i .o > The order in which suffix rules are found depends on the order in which > the suffixes appear in .SUFFIXES. To make your example work you need > the order to be this: > > .SUFFIXES: > .SUFFIXES: .o .i .c Oh, I see. Thanks! Don't know how I had missed that the order is important. I'm used to putting things in alphabetical order when editing POSIX/SUS text, so that's what I naturally did here. -- Geoff Clare The Open Group, Apex Plaza, Forbury Road, Reading, RG1 1AX, England
Re: [Issue 8 drafts 0001325]: Allow make to remake an included file
On Thu, 2020-11-05 at 15:42 +, Geoff Clare via austin-group-l at The Open Group wrote: > > I agree we can't over-simplify but I don't see a problem with the > > specific case you mention. > > If the first target_name operand is the name of an include file that > needs to be created, your statement "All include file regeneration is > complete before make attempts to bring [up-to-date] the > first target_name operand" creates a paradox. I don't really see a paradox here, sorry. Maybe I'm just not good enough at detailed reading. All include file generation is performed, then make attempts to bring the operands up to date. Why is there a paradox if an operand is also an include file? But I don't see a problem per se with the current wording so I'm happy to let this thread die as an academic discussion. Thanks Geoff!
Re: [Issue 8 drafts 0001325]: Allow make to remake an included file
On Thu, 2020-11-05 at 19:36 +0100, Joerg Schilling wrote:
> .SUFFIXES:
> .SUFFIXES: .o .c
> .c.o:
> echo foo
>
> In other words, a users makefile has full control over the "builtin"
> default rules even without a need to use "make -r".
Sure, it's easy to redefine a suffix rule. The issue in question was
that Geoff wanted to not use the .c.o rule at all. He wanted to use
other rules to build .o files in other ways, but the .c.o rule was
interfering with it. There's no way to DELETE that .c.o rule other
than -r.
However as I mentioned in my previous message you can still do it
without requiring -r, if you're very careful to order your suffixes
correctly so that the built-in .c.o rule is not considered.
> > In my opinion the -r option is wrong.
>
> Well GNU make makes the mistake to define pattern matching implicit
> rules in its default rules and this causes the following problems:
This is irrelevant to the topic at hand.
In fact, GNU make allows users to disable built-in rules completely
WITHOUT having to require users to add -r to the make command line,
because it allows users to add this to their makefile:
MAKEFLAGS += -r
and it does the right thing. Of course this is problematic as well
because that is now passed to sub-make invocations which may not be
what you want. However, in practice it is rarely a problem.
In fact, most of the built-in rules in GNU make ARE defined as suffix
rules; see this from default.c:
".c.o",
"$(COMPILE.c) $(OUTPUT_OPTION) $<",
".cc.o",
"$(COMPILE.cc) $(OUTPUT_OPTION) $<",
".C.o",
"$(COMPILE.C) $(OUTPUT_OPTION) $<",
".cpp.o",
etc.
However, GNU make also provides a number of built-in rules that cannot
be expressed as suffix rules, such as these:
/* RCS. */
{ "%", "%,v",
"$(CHECKOUT,v)" },
{ "%", "RCS/%,v",
"$(CHECKOUT,v)" },
{ "%", "RCS/%",
"$(CHECKOUT,v)" },
> - pattern matching rules have precedence over suffix rules and may
> themselves not be overwritten
>
> - pattern matching rules match in the order of their apperance to the
> parser. The first pattern rule that matches a specific pattern is
> used. Later specifying a replacement rule for a specific pattern is
> impossible.
>
> - The built in default rules must to be "read" by make before the
> users makefiles are parsed in order to allow them (the suffix
> rules) to be overwritten. Any pattern matching rule that appears
> in the builtin default rules thus would have precedence over
> anything else and cannot be overwritten.
Maybe these are problems in SunOS make / smake, but they are certainly
not problems in GNU make.
I'm not sure where the idea that pattern rules are immutable comes
from: in GNU make any pattern rule can be overwritten at any time,
including the default pattern rules.
Also, pattern rules can be completely canceled without overwriting
them, including as well the default pattern rules. It's just annoying
to have to do it by hand, and make sure you get them all, so something
like MAKEFLAGS += -r is much more robust.
Also, user-defined pattern rules are always added to be searched before
built-in pattern rules so the first one will always be a user-defined
pattern, if such exists.
And finally, it's not actually true that the first pattern rule that
matches is used. That used to be the case but quite a while ago the
pattern matcher was changed so that the rule with the shortest matching
stem is chosen first. If multiple pattern rules have stems with the
same length then the first one is chosen.
This algorithm change is a little controversial and I'm not sure about
it, but it seems to be acceptable in practice and it does avoid
worrying about the order in which patterns are defined.
Re: [Issue 8 drafts 0001325]: Allow make to remake an included file
On Thu, 2020-11-05 at 09:25 +, Geoff Clare via austin-group-l at The Open Group wrote: > > .SUFFIXES: > > .SUFFIXES: .c .i .o > > > That doesn't make any difference, since .c and .o are both in the > specified suffixes, so that brings the default .c.o rule into play. Sorry, I didn't read your example closely enough. Not only do you have to remove pre-existing suffixes, but you have to be very careful to order the suffixes correctly when you redefine them. I find the text in the standard somewhat confusing (particularly how when first introducing inference rules they are shown as ".s1.s2" but then a few paragraphs later this is changed to ".s2.s1"; if you miss that switch then the rest of the text doesn't make any sense). The order in which suffix rules are found depends on the order in which the suffixes appear in .SUFFIXES. To make your example work you need the order to be this: .SUFFIXES: .SUFFIXES: .o .i .c rather than ".c .i .o"... in particular you want the .c suffix to be after the .i suffix so that rules with .i are preferred.
Re: [Issue 8 drafts 0001325]: Allow make to remake an included file
Paul Smith via austin-group-l at The Open Group wrote: > On Thu, 2020-11-05 at 09:25 +, Geoff Clare via austin-group-l at > The Open Group wrote: > > That doesn't make any difference, since .c and .o are both in the > > specified suffixes, so that brings the default .c.o rule into play. > > Hm. Right, I forgot that clearing the suffixes doesn't also delete the > rules. It's annoying that there's no POSIX way to delete the default > rules from within the makefile. There is no need to do that. In former times, there was no way to clear .SUFFIXES, but at that time, SunOS defined default rules that look this way: SUFFIXES= .o .c ... .SUFFIXES: $(SUFFIXES) so you could clear it by overwriting the SUFFIXES variable from your user makefile. Now POSIX requires that writing: .SUFFIXES: into a Makefile clears the suffix list and thus effectively disables all default implicit rules. If you only like the .c.o rule but are not happy with the builtin default rule for e.g. .c.o:, you can overwrite it by e.g. writing: .SUFFIXES: .SUFFIXES: .o .c .c.o: echo foo In other words, a users makefile has full control over the "builtin" default rules even without a need to use "make -r". > In my opinion the -r option is wrong. Whether or not the default rules > should be used is a function of the _makefile_, not a function of the > _user_ of the makefile (the person who runs make). If the makefile is > written so that it doesn't want or need the default rules then the > makefile should be able to delete them. It shouldn't be up to the > invoker of make to do that. Well GNU make makes the mistake to define pattern matching implicit rules in its default rules and this causes the following problems: - pattern matching rules have precedence over suffix rules and may themselves not be overwritten - pattern matching rules match in the order of their apperance to the parser. The first pattern rule that matches a specific pattern is used. Later specifying a replacement rule for a specific pattern is impossible. - The built in default rules must to be "read" by make before the users makefiles are parsed in order to allow them (the suffix rules) to be overwritten. Any pattern matching rule that appears in the builtin default rules thus would have precedence over anything else and cannot be overwritten. For this reason, SunPro Make intentionally does not make use of pattern matching rules in the builtin default rules in order to permit the user to have full control over the behavior of make without a need to call "make -r". Jörg -- EMail:jo...@schily.net(home) Jörg Schilling D-13353 Berlin Blog: http://schily.blogspot.com/ URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/'
Re: [Issue 8 drafts 0001325]: Allow make to remake an included file
Paul Smith wrote, on 05 Nov 2020: > > On Thu, 2020-11-05 at 09:57 +, Geoff Clare via austin-group-l at > The Open Group wrote: > > > > The aim here was to describe the cut-off-point where all include > > > > file generation has been completed and after which the new > > > > contents of the files is used. This cut-off-point needs to be > > > > before make starts the "real work", i.e. starts the work to bring > > > > the first target operand, or the first target make encounters if > > > > there are no operands, up-to-date. > > > Can we just say that? All include file regeneration is complete > > > before make attempts to bring the first target_name operand, or the > > > first target make encounters if no target is specified? > > > > If we simplify it too much, it won't be correct. For example, > > consider the case where the first target_name operand is the name of > > an include file that needs to be created. > > I agree we can't over-simplify but I don't see a problem with the > specific case you mention. If the first target_name operand is the name of an include file that needs to be created, your statement "All include file regeneration is complete before make attempts to bring [up-to-date] the first target_name operand" creates a paradox. -- Geoff Clare The Open Group, Apex Plaza, Forbury Road, Reading, RG1 1AX, England
Re: [Issue 8 drafts 0001325]: Allow make to remake an included file
On Thu, 2020-11-05 at 09:57 +, Geoff Clare via austin-group-l at The Open Group wrote: > > > The aim here was to describe the cut-off-point where all include > > > file generation has been completed and after which the new > > > contents of the files is used. This cut-off-point needs to be > > > before make starts the "real work", i.e. starts the work to bring > > > the first target operand, or the first target make encounters if > > > there are no operands, up-to-date. > > Can we just say that? All include file regeneration is complete > > before make attempts to bring the first target_name operand, or the > > first target make encounters if no target is specified? > > If we simplify it too much, it won't be correct. For example, > consider the case where the first target_name operand is the name of > an include file that needs to be created. I agree we can't over-simplify but I don't see a problem with the specific case you mention. For GNU make, at any rate, nothing special will be done if a target_name operand is also an include file. The include file will be rebuilt (if needed) as usual, then make will attempt to build the target_name operand and realize it's already up to date, and go on to the next one (or finish).
Re: [Issue 8 drafts 0001325]: Allow make to remake an included file
On Thu, 2020-11-05 at 09:25 +, Geoff Clare via austin-group-l at The Open Group wrote: > That doesn't make any difference, since .c and .o are both in the > specified suffixes, so that brings the default .c.o rule into play. Hm. Right, I forgot that clearing the suffixes doesn't also delete the rules. It's annoying that there's no POSIX way to delete the default rules from within the makefile. In my opinion the -r option is wrong. Whether or not the default rules should be used is a function of the _makefile_, not a function of the _user_ of the makefile (the person who runs make). If the makefile is written so that it doesn't want or need the default rules then the makefile should be able to delete them. It shouldn't be up to the invoker of make to do that. And if the makefile is written so that it _does_ need the default rules, it shouldn't be up to the user to delete them with -r. Of course that's a completely separate topic :).
Re: [Issue 8 drafts 0001325]: Allow make to remake an included file
Paul Smith wrote, on 04 Nov 2020: > > On Wed, 2020-11-04 at 10:48 +, Geoff Clare via austin-group-l at > The Open Group wrote: > > The aim here was to describe the cut-off-point where all include file > > generation has been completed and after which the new contents of the > > files is used. This cut-off-point needs to be before make starts the > > "real work", i.e. starts the work to bring the first target operand, > > or the first target make encounters if there are no operands, up-to- > > date. > > Can we just say that? All include file regeneration is complete before > make attempts to bring the first target_name operand, or the first > target make encounters if no target is specified? If we simplify it too much, it won't be correct. For example, consider the case where the first target_name operand is the name of an include file that needs to be created. -- Geoff Clare The Open Group, Apex Plaza, Forbury Road, Reading, RG1 1AX, England
Re: [Issue 8 drafts 0001325]: Allow make to remake an included file
Joerg Schilling wrote, on 05 Nov 2020: > > Geoff Clare via austin-group-l at The Open Group > wrote: > > > The make utility shall use one of the following two methods > > to attempt to create the file or bring it up-to-date: > > > > 1. The "immediate remaking" method > > > > If make uses this method, any target rules or inference > > rules for the pathname that were parsed before the include line > > was parsed shall be used to attempt to create the file or to > > bring it up-to-date before opening the file. > > > > 2. The "delayed remaking" method > > > > If make uses this method, no attempt shall be made to > > create the file or bring it up-to-date until after the > > makefile(s) have been read. During processing of the include > > line, make shall read the current contents of the file, > > if it exists, or treat it as an empty file if it does not exist. > > Once the makefile(s) have been read, make shall use any > > applicable target rule or inference rule for the pathname, > > regardless of whether it is parsed before or after the include > > line, when creating the file or bringing it up-to-date. > > Additionally in this case, [...] > > > > If the pathname is relative, [...] > > Sorry, but this would result in non-portable makefiles and it is not > acceptable. > > In order to permit portable makefiles, all rules to make/remake include files > need to be before the include statement. That's a constraint on applications, not on impelementations. It is covered by the first bullet point in the proposed APPLICATION USAGE addition. -- Geoff Clare The Open Group, Apex Plaza, Forbury Road, Reading, RG1 1AX, England
Re: [Issue 8 drafts 0001325]: Allow make to remake an included file
Geoff Clare via austin-group-l at The Open Group wrote: > The make utility shall use one of the following two methods > to attempt to create the file or bring it up-to-date: > > 1. The "immediate remaking" method > > If make uses this method, any target rules or inference > rules for the pathname that were parsed before the include line > was parsed shall be used to attempt to create the file or to > bring it up-to-date before opening the file. > > 2. The "delayed remaking" method > > If make uses this method, no attempt shall be made to > create the file or bring it up-to-date until after the > makefile(s) have been read. During processing of the include > line, make shall read the current contents of the file, > if it exists, or treat it as an empty file if it does not exist. > Once the makefile(s) have been read, make shall use any > applicable target rule or inference rule for the pathname, > regardless of whether it is parsed before or after the include > line, when creating the file or bringing it up-to-date. > Additionally in this case, [...] > > If the pathname is relative, [...] Sorry, but this would result in non-portable makefiles and it is not acceptable. In order to permit portable makefiles, all rules to make/remake include files need to be before the include statement. Jörg -- EMail:jo...@schily.net(home) Jörg Schilling D-13353 Berlin Blog: http://schily.blogspot.com/ URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/'
Re: [Issue 8 drafts 0001325]: Allow make to remake an included file
Paul Smith wrote, on 04 Nov 2020: > > On Wed, 2020-11-04 at 16:47 +, Geoff Clare via austin-group-l at > The Open Group wrote: > > Here's a modified version of the proposed example that uses this new > > technique. Note that you have to use "make -r" otherwise it uses the > > default .c.o rule instead! The rules for a.o and b.o seem to be > > needed by SunPro make but not by GNU make. > > > > .POSIX: > > .SUFFIXES: .c .i .o > > To avoid using make -r you can clear the suffixes before defining new > ones: > >.SUFFIXES: >.SUFFIXES: .c .i .o That doesn't make any difference, since .c and .o are both in the specified suffixes, so that brings the default .c.o rule into play. One way of handling the issue is to alert the user to the need to use -r if they forget: CC = echo "Please use make -r"; false (Of course, this means you can't use $(CC) in the specified rules, but my example uses c99 so it works there.) -- Geoff Clare The Open Group, Apex Plaza, Forbury Road, Reading, RG1 1AX, England
Re: [Issue 8 drafts 0001325]: Allow make to remake an included file
On Wed, 2020-11-04 at 18:05 +0100, Joerg Schilling via austin-group-l at The Open Group wrote: > I am not sure whether I missed something, but I cannot see a method > that may work in a portable way and unless I missed something, I see > no new idea that makes things really better than what we are > currently discussing. In the paper I discuss the problems with the generated output and the advantages to avoiding it. > The problem I see is that the advanced method available with SunPro > Make does not seem to be supported by GNU make and the options > mentioned in the paper from above seem to be GCC specific and still > require a separate compiler call. They are technically not GCC-specific: clang also provides these options. But that support is not necessary: in the paper I go over a few alternatives to use if your compiler isn't capable of generating dependencies as a side-effect of compilation. Whatever process you use to generate the .d files is simply added to the compiler invocation rule. Yes, it means you may need to run the compiler twice but it still works and you have to run the compiler twice when using generated include files anyway. The built-in compiler support is simply an efficiency improvement, it's not a fundamental requirement.
Re: [Issue 8 drafts 0001325]: Allow make to remake an included file
On Wed, 2020-11-04 at 16:47 +, Geoff Clare via austin-group-l at The Open Group wrote: > Having read this, I'm now wondering why we are bothering to add > requirements for generating include files, if it is no longer the > recommended way of doing things. There are other uses for auto-generating included files: it does give you a bit of "meta-programming" capability and some flexibility to auto-generate rules that plain makefiles cannot. Implicit rules, and particularly POSIX suffix rules, are really not powerful enough to provide all the different types of templating you might want. However I don't know if this is sufficient justification for a change to POSIX. I wrote a post about this too: http://make.mad-scientist.net/constructed-include-files/ Looks like this post needs to be updated to note you don't need to use -include anymore as well :) > > This post obviously makes use of GNU make-specific features but I > > don't believe there's anything there that couldn't be also > > implemented in standard make. > > To improve efficiency it relies on the compiler having an option to > request creation of a dependency list at the same time as creating > the .o file. However, something similar is possible with standard c99 > by creating a .i file and then creating the .o and .d from that. Yes. But in any event, even if you do the simple thing and just run it twice you're no worse off since regenerated include files require running the compiler twice as well. > Here's a modified version of the proposed example that uses this new > technique. Note that you have to use "make -r" otherwise it uses the > default .c.o rule instead! The rules for a.o and b.o seem to be > needed by SunPro make but not by GNU make. > > .POSIX: > .SUFFIXES: .c .i .o To avoid using make -r you can clear the suffixes before defining new ones: .SUFFIXES: .SUFFIXES: .c .i .o
Re: [Issue 8 drafts 0001325]: Allow make to remake an included file
On Wed, 2020-11-04 at 10:48 +, Geoff Clare via austin-group-l at The Open Group wrote: > > - > > (0005094) psmith (developer) - 2020-11-03 15:23 > > https://austingroupbugs.net/view.php?id=1325#c5094 > > > > Regarding the change to page 2891 lines 97033-97035: > > It's there because GNU make uses re-execution. The re-executed make > doesn't know what targets the previous invocation updated, so it > doesn't know to treat them as already up-to-date. I understand. I need to stop thinking of the re-exec'd make and simply consider the entirety of the process. The real process GNU make undertakes is something like: 1. Read all makefiles. Don't error if they do not exist. 2. Consider all makefiles as goal targets and try to rebuild them. 3. If any rebuild, clear out all knowledge of all parsed makefiles and go to step 1 (GNU make does this via re-exec but that's not required of course). 4. If any required makefiles are missing, fail. 5. Build the normal goal targets. So, it's never the case that by the time you get to step #5, any makefile will be reconsidered: at that point all makefiles will be known to be up to date. > The aim here was to describe the cut-off-point where all include file > generation has been completed and after which the new contents of the > files is used. This cut-off-point needs to be before make starts the > "real work", i.e. starts the work to bring the first target operand, > or the first target make encounters if there are no operands, up-to- > date. Can we just say that? All include file regeneration is complete before make attempts to bring the first target_name operand, or the first target make encounters if no target is specified? > > In the portability text (second bullet), a portable workaround to > > using a macro defined in one included file in the include line of > > another, is to put the second include _inside_ the first included > > file instead of in the outer including makefile. > > Okay, that sounds like it is worth adding. Maybe something like: > > Include lines and rules for creating dynamic include files do not > depend on the contents of any earlier dynamic include file. For > example, defining a macro in a dynamic include file and using > that > macro in a later include line should be avoided (unless the later > include line is itself inside the dynamic include file). OK. > If adding "ddir" to the search path is achieved by means of an > extension (such as -I), then the requirement in the standard does not > apply to "ddir"; it only applies to the default search path. > > It is only the default search path that I see as a problem. If a > user explicitly wants a directory searched, then that implies they > know what's in that directory and will choose include file names with > that in mind. If there's a clash which causes their makefile to > behave differently than intended, that's the user's fault. Yes, exactly. > Sounds like we are in agreement with how GNU make should handle this > issue. It can prioritise specified search paths over remaking, but > not the default search path. I believe the currently proposed text > allows that, because specifying a search path involves using an > extension. OK, I'm OK with making this change to GNU make to meet the standard if needed. Thanks!
Re: [Issue 8 drafts 0001325]: Allow make to remake an included file
On Wed, 2020-11-04 at 12:01 +, Geoff Clare via austin-group-l at The Open Group wrote: > I haven't been able to come up with a common initial statement that > is preferable to just describing the two methods separately. So > unless someone else wants to give it a go, I suggest that I should > restructure that part along these lines: > > The make utility shall use one of the following two methods > to attempt to create the file or bring it up-to-date: > > 1. The "immediate remaking" method > > If make uses this method, any target rules or inference > rules for the pathname that were parsed before the include line > was parsed shall be used to attempt to create the file or to > bring it up-to-date before opening the file. > > 2. The "delayed remaking" method > > If make uses this method, no attempt shall be made to > create the file or bring it up-to-date until after the > makefile(s) have been read. During processing of the include > line, make shall read the current contents of the file, > if it exists, or treat it as an empty file if it does not exist. > Once the makefile(s) have been read, make shall use any > applicable target rule or inference rule for the pathname, > regardless of whether it is parsed before or after the include > line, when creating the file or bringing it up-to-date. This seems OK to me. I like the idea of using two separate sections; the process seems different enough that it's confusing to shoe-horn it into a single one.
Re: [Issue 8 drafts 0001325]: Allow make to remake an included file
Geoff Clare via austin-group-l at The Open Group wrote: > > I wrote a blog post about this which may be interesting: > > http://make.mad-scientist.net/papers/advanced-auto-dependency-generation/ > > Having read this, I'm now wondering why we are bothering to add > requirements for generating include files, if it is no longer the > recommended way of doing things. I am not sure whether I missed something, but I cannot see a method that may work in a portable way and unless I missed something, I see no new idea that makes things really better than what we are currently discussing. The problem I see is that the advanced method available with SunPro Make does not seem to be supported by GNU make and the options mentioned in the paper from above seem to be GCC specific and still require a separate compiler call. The intention for POSIX however is to standardize portable solutions. This is why I believe that the current solution from bug 1325 is the right way to go in the close future. The method based on cc -E as distributed by the schily makefilesystem has been verified to be portable to virtually any platform - even to the Microsoft compiler, see the shell script conf/mkdep-msc.sh Jörg -- EMail:jo...@schily.net(home) Jörg Schilling D-13353 Berlin Blog: http://schily.blogspot.com/ URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/'
Re: [Issue 8 drafts 0001325]: Allow make to remake an included file
> --
> (0005096) psmith (developer) - 2020-11-03 15:31
> https://austingroupbugs.net/view.php?id=1325#c5096
> --
> Apropos of the example given here: I just want to say that in the GNU
> project we have completely moved away from using rebuilt included makefiles
> as part of automatic prerequisite generation, as being (a) tricky, (b)
> impossible to make completely seamless, (c) inefficient, and (d) wholly
> unnecessary.
>
> I wrote a blog post about this which may be interesting:
> http://make.mad-scientist.net/papers/advanced-auto-dependency-generation/
Having read this, I'm now wondering why we are bothering to add
requirements for generating include files, if it is no longer the
recommended way of doing things.
> This post obviously makes use of GNU make-specific features but I don't
> believe there's anything there that couldn't be also implemented in
> standard make.
To improve efficiency it relies on the compiler having an option to
request creation of a dependency list at the same time as creating
the .o file. However, something similar is possible with standard c99
by creating a .i file and then creating the .o and .d from that.
At first sight it also looked like it depended on $(wildcard ...) but
then I realised it could just use:
-include $(DEPFILES)
instead of:
include $(wildcard $(DEPFILES))
Here's a modified version of the proposed example that uses this new
technique. Note that you have to use "make -r" otherwise it uses the
default .c.o rule instead! The rules for a.o and b.o seem to be
needed by SunPro make but not by GNU make.
.POSIX:
.SUFFIXES: .c .i .o
OFILES = a.o b.o
pgm: $(OFILES)
c99 $(OFILES) -o pgm
a.o: a.i
b.o: b.i
.c.i:
c99 -E $< > $@
cfile=$<; dfile=$${cfile%.c}.d; \
{ \
set -o pipefail; \
printf '%s: %s ' $@ $<; \
LC_ALL=C sed -n \
'/^#[[:blank:]]*[[:digit:]]/s/.*"\([^"]*\.h\)".*/\1/p' $@ | \
LC_ALL=C sort -u | tr '\n' ' '; \
echo; \
} > "$$dfile"
.i.o:
c99 -c $<
-include $(OFILES:.o=.d)
--
Geoff Clare
The Open Group, Apex Plaza, Forbury Road, Reading, RG1 1AX, England
Re: [Issue 8 drafts 0001325]: Allow make to remake an included file
Geoff Clare wrote, on 04 Nov 2020: > > > > If a target rule or inference rule for the pathname has been parsed > > > before the include line is parsed, make shall use the rule to attempt > > > to create the file or to bring it up-to-date. > > > > I don't think this is quite correct. In GNU make, anyway, it's not > > required that an inference rule parsed before the include line is parsed > > will always be used. If a target rule, or a different inference rule, is > > defined after the include line then that will be used instead. > > I see your point. I was trying to find a way to make an initial single > statement that was true for both methods, but maybe that's not possible. > The text may need to separate the two descriptions entirely. I'll give > it some more thought. I haven't been able to come up with a common initial statement that is preferable to just describing the two methods separately. So unless someone else wants to give it a go, I suggest that I should restructure that part along these lines: The make utility shall use one of the following two methods to attempt to create the file or bring it up-to-date: 1. The "immediate remaking" method If make uses this method, any target rules or inference rules for the pathname that were parsed before the include line was parsed shall be used to attempt to create the file or to bring it up-to-date before opening the file. 2. The "delayed remaking" method If make uses this method, no attempt shall be made to create the file or bring it up-to-date until after the makefile(s) have been read. During processing of the include line, make shall read the current contents of the file, if it exists, or treat it as an empty file if it does not exist. Once the makefile(s) have been read, make shall use any applicable target rule or inference rule for the pathname, regardless of whether it is parsed before or after the include line, when creating the file or bringing it up-to-date. Additionally in this case, [...] If the pathname is relative, [...] -- Geoff Clare The Open Group, Apex Plaza, Forbury Road, Reading, RG1 1AX, England
Re: [Issue 8 drafts 0001325]: Allow make to remake an included file
> -- > (0005094) psmith (developer) - 2020-11-03 15:23 > https://austingroupbugs.net/view.php?id=1325#c5094 > -- > Thank you for all your effort Geoff! I like this version very much. A few > notes / clarifying questions: > > Regarding the change to page 2891 lines 97033-97035: is this because smake > / SunPro make will re-consider and re-build included makefiles if they > appear as prerequisites to another target? GNU make never reconsiders a > target after it's been considered, regardless of whether or not it's an > included file. It's there because GNU make uses re-execution. The re-executed make doesn't know what targets the previous invocation updated, so it doesn't know to treat them as already up-to-date. > Regarding this: > > > If a target rule or inference rule for the pathname has been parsed > before the include line is parsed, make shall use the rule to attempt to > create the file or to bring it up-to-date. > > I don't think this is quite correct. In GNU make, anyway, it's not > required that an inference rule parsed before the include line is parsed > will always be used. If a target rule, or a different inference rule, is > defined after the include line then that will be used instead. I see your point. I was trying to find a way to make an initial single statement that was true for both methods, but maybe that's not possible. The text may need to separate the two descriptions entirely. I'll give it some more thought. > For the section "Additionally in this case, the new contents of the file, > if it is successfully created or updated, shall be used when processing > rules for the following targets after the makefile(s) have been read:" > followed by the bullet points, can you give me a hint as to what we are > trying to ensure or prevent by this text? By listing these specific things > I worry that I will overlook something that we do today that will become > invalid as a result of this text. Is there a difference we are trying to > surface between how included files normally work, and these rebuilt > included files work? The aim here was to describe the cut-off-point where all include file generation has been completed and after which the new contents of the files is used. This cut-off-point needs to be before make starts the "real work", i.e. starts the work to bring the first target operand, or the first target make encounters if there are no operands, up-to-date. The third bullet is because make has to (recursively) bring the prerequisites of this first target up-to-date before the target itself. Perhaps there is a better way to achieve this aim - I'm open to suggestions. > Is it not sufficient to say something like, the new > contents of the file will be used when processing rules that appear after > the included file as if the text had appeared in the makefile? Surely that's not true for GNU make. I thought it would use old include file contents when processing a later rule that is used to generate another include file. > In the portability text (second bullet), a portable workaround to using a > macro defined in one included file in the include line of another, is to > put the second include _inside_ the first included file instead of in the > outer including makefile. This can be less convenient in some situations > but it will work in all situations. Okay, that sounds like it is worth adding. Maybe something like: Include lines and rules for creating dynamic include files do not depend on the contents of any earlier dynamic include file. For example, defining a macro in a dynamic include file and using that macro in a later include line should be avoided (unless the later include line is itself inside the dynamic include file). (the only difference is the added part in parentheses). > > In the portability text I don't understand the need for the third > bullet...? It was an attempt at guidance on how to prevent re-execution-ad-infinitum. > In the make examples text I am not sure what the point of the initial > -include line is... why is that needed in this situation? So that SunPro make knows the prerequisites for the include file (if it already exists) when it processes the second include line. > It also seems > that each target will list all its prerequisites twice, although of course > that's not a problem. You might also consider using the make variable > $(CC) in these examples rather than hardcoding the c99 value. It's intended as an updated version of the previous example, which uses c99. I would not object to changing both examples to use $(CC). > Regarding the issue of searching of include files, I don't agree with this > as written. I do understand the problem you're addressing, and I have no > problems making some types of changes to GNU make to meet the standard, but > the
