Re: [Issue 8 drafts 0001505]: Make doesn't seem to specify unset macro expansion behaviour
Quentin Rameau wrote, on 17 Jan 2022: > > > A NOTE has been added to this issue. > > == > > https://austingroupbugs.net/view.php?id=1505 > > Maybe we should use a different wording for qualifying the macro state, > like “being set” and “unset” (or “not set”) instead of existing and > not existing). > > What do you think? I used the "exist" phrasing in the proposal because it matches the wording used for other additions to the make page in Issue 8 that have already been applied, e.g. from bug 330. -- Geoff Clare The Open Group, Apex Plaza, Forbury Road, Reading, RG1 1AX, England
Re: [Issue 8 drafts 0001505]: Make doesn't seem to specify unset macro expansion behaviour
Hello, > A NOTE has been added to this issue. > == > https://austingroupbugs.net/view.php?id=1505 Maybe we should use a different wording for qualifying the macro state, like “being set” and “unset” (or “not set”) instead of existing and not existing). What do you think?
Re: [Issue 8 drafts 0001505]: Make doesn't seem to specify unset macro expansion behaviour
Geoff Clare wrote in
<20211221103814.GB12295@localhost>:
|Steffen Nurpmeso wrote, on 20 Dec 2021:
|>
|> For example CRUX-Linux has a /etc/pkgmk.conf where people can
|> define $CFLAGS, $CXXFLAGS, etc., also things like
|>
|> export JOBS=$(nproc)
|> export MAKEFLAGS="-j $JOBS"
...
|So use:
|
|CFLAGS?=
|CFLAGS+=-Weven-more-noise
|
|to get the same behaviour as before, if you are willing to keep taking
|the risk of it misbehaving.
I fail to find a real life makefile example which would be bitten
by your proposal. Most programs use make file generators, for
which it does not matter, and those which still support straight
make files usually do something similar to git(1) here
# Guard against environment variables
...
SCRIPT_SH =
...
SCRIPT_SH += git-bisect.sh
Only busybox uses
CFLAGS := $(CFLAGS)
but that is ok since CFLAGS is made available via make(1) anyway.
While i am looking around, users of GNU make(1) seem to make heavy
use of (un)?export statements in make files, which addresses
a deficiency in POSIX make(1), one may say, i usually generate
shell variable assignment files that then are included before the
sub-make is executed, that is
build:
@$(_prestop); LC_ALL=C $${MAKE} -f mk-config.mk all
[.]
_prestop = $(__prestop); cd "$(OBJDIR)" && . ./mk-config.env
--steffen
|
|Der Kragenbaer,The moon bear,
|der holt sich munter he cheerfully and one by one
|einen nach dem anderen runter wa.ks himself off
|(By Robert Gernhardt)
Re: [Issue 8 drafts 0001505]: Make doesn't seem to specify unset macro expansion behaviour
Steffen Nurpmeso wrote, on 20 Dec 2021: > > For example CRUX-Linux has a /etc/pkgmk.conf where people can > define $CFLAGS, $CXXFLAGS, etc., also things like > > export JOBS=$(nproc) > export MAKEFLAGS="-j $JOBS" > > It is processed by the shell before the actual package is build. > I personally set things in my environment, $CFLAGS for example. > These things are picked up by most build processes automatically > ever since i consciously look at that. > > all: > echo CFLAGS=$(CFLAGS) > -> > echo CFLAGS=-O1 -g > CFLAGS=-O1 -g > > or even > > CFLAGS+=-Weven-more-noise > all: > echo CFLAGS=$(CFLAGS) > -> > echo CFLAGS=-O1 -g -Weven-more-noise > CFLAGS=-O1 -g -Weven-more-noise > > This would vanish with the proposal: > > CFLAGS= > CFLAGS+=-Weven-more-noise > all: > echo CFLAGS=$(CFLAGS) > -> > echo CFLAGS=-Weven-more-noise > CFLAGS=-Weven-more-noise So use: CFLAGS?= CFLAGS+=-Weven-more-noise to get the same behaviour as before, if you are willing to keep taking the risk of it misbehaving. -- Geoff Clare The Open Group, Apex Plaza, Forbury Road, Reading, RG1 1AX, England
Re: [Issue 8 drafts 0001505]: Make doesn't seem to specify unset macro expansion behaviour
Gabriel Soldani wrote, on 20 Dec 2021: > > On Mon, 20 Dec 2021 at 11:55, Geoff Clare wrote: > > The idea is that if make gives an error for unset macros, then the > > *author* of the makefile will get the error during development, and > > will not ship a makefile that depends on expanding unset macros, thus > > preventing that scenario happening at all. > > The author won't be able to rely on these errors being complete, because > their implementation or their environment may set any macro, and a > conforming implementation would not be allowed to warn the author in > neither case. This is exactly the "opposite problem" that I brought into the discussion later (where I talked about utilities reading from standard input and the behaviour of cd and GNU find). As I explained then, it is also beneficial for make to report an error when a user has a macro unset that the makefile author had non-empty during development. > > Environment variables do not override macros set in the makefile > > (unless you use make -e). > > You're right, but consider the "CPPFLAGS ?=" case. > > This is how a makefile author currently expresses that they want the > user's environment to override CPPFLAGS only, but not every macro in the > file. Perhaps CPPFLAGS isn't the best example, but there are many cases > where deferring to the user's environment while providing a sensible > default is useful. > > With this change, there would be no way for an author to selectively > choose which macros they want to defer to the user's environment, > they'll either get all (with -e) or nothing. I don't see why ?= couldn't still be used for this purpose, if the makefile author wishes to take the risk of doing that. To have the environment override a non-empty default you would still be able to use: CPPFLAGS ?= defaults Although for an empty default you would now need to add: CPPFLAGS ?= whereas before it would work without that. -- Geoff Clare The Open Group, Apex Plaza, Forbury Road, Reading, RG1 1AX, England
Re: [Issue 8 drafts 0001505]: Make doesn't seem to specify unset macro expansion behaviour
I'm not sure whether continuing to go back and forth on this is fruitful. I certainly understand the issue under discussion but I absolutely do not agree that it's the place of the POSIX standard to try to remediate it by allowing implementations to fail. As the only (as far as I'm aware) representative of make implementers in this conversation, I'm opposed to the change in wording as currently described in #1505 and it should be rejected. It appears that the underlying objection is to the whole idea of importing environment variables as make variables. This proposal is a back-door way to make such usage untenable in portable makefiles, without actually changing the standard to disallow it. The arguments for this change boil down to "we don't think people should do this because it's not reliable" and the arguments against it boil down to "every implementation does this and most every makefile relies on it working this way". Both can be true, but the latter must outweigh the former when deciding what behavior to standardize. If makefile authors want to ensure that their makefiles are not subject to this kind of unreliability they are free to write their makefiles to avoid it. Whether an implementation assists them in that or not is a QoI issue. On Mon, 2021-12-20 at 12:41 +, Geoff Clare via austin-group-l at The Open Group wrote: > > Also, as far as I can tell there's nothing in the standard that > > says that make can't show a warning message even as default > > behavior, if some implementation wanted to do that. > > Then you didn't look far enough. Such as message is not allowed to > standard error because STDERR says the usual "The standard error > shall be used only for diagnostic messages" (which require non-zero > exit). And it's not allowed to standard output because there is > nothing in the STDOUT section that explicitly allows it, and 1.4 > Utility Description Defaults says that STDOUT "completely describes > the standard output of the utility". Fine, then I would be OK with a change to the standard that allowed diagnostics for this situation to be printed to stdout or stderr as a warning, but requiring that make expand the variable to the empty string and continuing to process the makefile without failing. Just to note, I'm not aware of any implementation of make that strictly obeys these rules regarding stdout and stderr. They all either may generate extra output to stdout, or may generate output to stderr while still exiting with a success/0 exit code, or both.
Re: [Issue 8 drafts 0001505]: Make doesn't seem to specify unset macro expansion behaviour
Hello.
Geoff Clare wrote in
<20211220133200.GA25606@localhost>:
|Robert Elz wrote, on 18 Dec 2021:
|> bsd make (bmake) has a couple of macros (the doc calls what are
|> macros here variables, but they are the same thing), which default to
|> being unset, which can be set in the environment or command line,
|> but which cannot be set in any Makefile, as they alter the interpretation
|> of makefiles (slightly). That is, if they are going to be defined, that
|> needs to have happened before the first makefile line is read.
|> They are used (expanded) in makefiles quite commonly.
|>
|> For that to work, no rule requiring variables to be always set in the
|> Makefile before being expanded is possible
|
|No such rule is being proposed. The error is only allowed if
|the macro is expanded while unset. If it is set, it makes no
|difference how it was set.
|
|> The += is effectively shorthand for (not possible in make I think)
|> MACRO= ${MACRO} string
|> an so is effectively expanding the otherwise potentially unset MACRO
|> before altering it. Any implementation which generated an error on
|> an expansion of an unset macro would need to generate an error for
|> this usage (assuming MACRO was unset previously) as well.
|
|Thank you for pointing that out. We should revisit the proposed
|changes to allow this error as well.
Now that it starts to affect me.. :)
I do not think the proposal reflects practice.
For example CRUX-Linux has a /etc/pkgmk.conf where people can
define $CFLAGS, $CXXFLAGS, etc., also things like
export JOBS=$(nproc)
export MAKEFLAGS="-j $JOBS"
It is processed by the shell before the actual package is build.
I personally set things in my environment, $CFLAGS for example.
These things are picked up by most build processes automatically
ever since i consciously look at that.
all:
echo CFLAGS=$(CFLAGS)
->
echo CFLAGS=-O1 -g
CFLAGS=-O1 -g
or even
CFLAGS+=-Weven-more-noise
all:
echo CFLAGS=$(CFLAGS)
->
echo CFLAGS=-O1 -g -Weven-more-noise
CFLAGS=-O1 -g -Weven-more-noise
This would vanish with the proposal:
CFLAGS=
CFLAGS+=-Weven-more-noise
all:
echo CFLAGS=$(CFLAGS)
->
echo CFLAGS=-Weven-more-noise
CFLAGS=-Weven-more-noise
It even becomes impossible to get the same effect that is used in
the wild, as Robert Elz already said, specifying 'make
CFLAGS="$CFLAGS"' on the command line hard-sets CFLAGS in the
makefile, and the += never comes into play.
--steffen
|
|Der Kragenbaer,The moon bear,
|der holt sich munter he cheerfully and one by one
|einen nach dem anderen runter wa.ks himself off
|(By Robert Gernhardt)
Re: [Issue 8 drafts 0001505]: Make doesn't seem to specify unset macro expansion behaviour
On Mon, 20 Dec 2021 at 11:55, Geoff Clare wrote: > The idea is that if make gives an error for unset macros, then the > *author* of the makefile will get the error during development, and > will not ship a makefile that depends on expanding unset macros, thus > preventing that scenario happening at all. The author won't be able to rely on these errors being complete, because their implementation or their environment may set any macro, and a conforming implementation would not be allowed to warn the author in neither case. > Environment variables do not override macros set in the makefile > (unless you use make -e). You're right, but consider the "CPPFLAGS ?=" case. This is how a makefile author currently expresses that they want the user's environment to override CPPFLAGS only, but not every macro in the file. Perhaps CPPFLAGS isn't the best example, but there are many cases where deferring to the user's environment while providing a sensible default is useful. With this change, there would be no way for an author to selectively choose which macros they want to defer to the user's environment, they'll either get all (with -e) or nothing. -- Gabriel Soldani
Re: [Issue 8 drafts 0001505]: Make doesn't seem to specify unset macro expansion behaviour
Gabriel Soldani wrote, on 20 Dec 2021: > > On Mon, 20 Dec 2021 at 09:16, Geoff Clare wrote: > > Okay, I'll give you that. So the problem we are trying to prevent > > occurs if the macro happens to be set to something unexpected instead > > of something useful. > > [...] > > There is nothing to prevent a make implementation providing a > > non-empty default for a CPPFLAGS macro that is completely unrelated > > to C compilation. Or a user could have an environment variable > > called CPPFLAGS that they have to set for some "Corporate Pension > > Processing" application that they use. > > > > In either of these cases, the .c.o rule will do something unexpected > > In the first scenario you propose, even if the author is using a make > implementation that warns about unset macros, and that implementation > also happens to set CPPFLAGS to something unexpected, it won't warn the > author about CPPFLAGS being unset and the .c.o rule will do something > unexpected anyway. The idea is that if make gives an error for unset macros, then the *author* of the makefile will get the error during development, and will not ship a makefile that depends on expanding unset macros, thus preventing that scenario happening at all. > In the second scenario, even if the author adds "CPPFLAGS =" or > "CPPFLAGS ?=" the user's environment will override CPPFLAGS with the > value set for the "Corporate Pension Processing" application and the > .c.o rule will again do something unexpected. Environment variables do not override macros set in the makefile (unless you use make -e). -- Geoff Clare The Open Group, Apex Plaza, Forbury Road, Reading, RG1 1AX, England
Re: [Issue 8 drafts 0001505]: Make doesn't seem to specify unset macro expansion behaviour
On Mon, 20 Dec 2021 at 09:16, Geoff Clare wrote: > Okay, I'll give you that. So the problem we are trying to prevent > occurs if the macro happens to be set to something unexpected instead > of something useful. > [...] > There is nothing to prevent a make implementation providing a > non-empty default for a CPPFLAGS macro that is completely unrelated > to C compilation. Or a user could have an environment variable > called CPPFLAGS that they have to set for some "Corporate Pension > Processing" application that they use. > > In either of these cases, the .c.o rule will do something unexpected In the first scenario you propose, even if the author is using a make implementation that warns about unset macros, and that implementation also happens to set CPPFLAGS to something unexpected, it won't warn the author about CPPFLAGS being unset and the .c.o rule will do something unexpected anyway. In the second scenario, even if the author adds "CPPFLAGS =" or "CPPFLAGS ?=" the user's environment will override CPPFLAGS with the value set for the "Corporate Pension Processing" application and the .c.o rule will again do something unexpected. I agree that if this committee were designing new software, certainly make should be redesigned to avoid _any_ unexpected behavior such as those discussed here. But this is certainly not the case: the proposed changes don't really reduce the number of unexpected cases, and we've yet to see real-world examples of unexpected make behaviors with unintended consequences to warrant considering which unexpected cases are more likely than others. As such, please consider standardizing the de facto standard behavior of expanding unset macros to the empty string. This edition standardizes many behaviors which makefile authors already rely on and it is going to be easier than ever to write fully conforming makefiles. Don't make them settle for almost conforming.
Re: [Issue 8 drafts 0001505]: Make doesn't seem to specify unset macro expansion behaviour
Robert Elz wrote, on 18 Dec 2021:
>
> bsd make (bmake) has a couple of macros (the doc calls what are
> macros here variables, but they are the same thing), which default to
> being unset, which can be set in the environment or command line,
> but which cannot be set in any Makefile, as they alter the interpretation
> of makefiles (slightly). That is, if they are going to be defined, that
> needs to have happened before the first makefile line is read.
> They are used (expanded) in makefiles quite commonly.
>
> For that to work, no rule requiring variables to be always set in the
> Makefile before being expanded is possible
No such rule is being proposed. The error is only allowed if
the macro is expanded while unset. If it is set, it makes no
difference how it was set.
> The += is effectively shorthand for (not possible in make I think)
> MACRO= ${MACRO} string
> an so is effectively expanding the otherwise potentially unset MACRO
> before altering it. Any implementation which generated an error on
> an expansion of an unset macro would need to generate an error for
> this usage (assuming MACRO was unset previously) as well.
Thank you for pointing that out. We should revisit the proposed
changes to allow this error as well.
--
Geoff Clare
The Open Group, Apex Plaza, Forbury Road, Reading, RG1 1AX, England
Re: [Issue 8 drafts 0001505]: Make doesn't seem to specify unset macro expansion behaviour
Paul Smith wrote, on 17 Dec 2021: > > On Fri, 2021-12-17 at 10:11 +, Geoff Clare via austin-group-l at > The Open Group wrote: > > Paul Smith wrote, on 16 Dec 2021: > > > I'm somewhat nonplussed by this conversation. > > > > > > We have a POSIX tool, that's been in use for 45 years. All known > > > implementations of that tool provide a specific behavior. No known > > > implementations of that tool provide some different behavior. Users > > > of the tool have written hundreds of thousands of files that expect > > > and rely on the provided behavior and feel that it's valuable. > > > > They rely not only on the behaviour of make but also on macros being > > unset by default. They are wrong to rely on the latter, because it > > can easily not be the case depending on circumstances. > > They are not wrong, because that's exactly the behavior they want: they > want the variable to expand to empty when it's not set, and expand to > something else when it is set. You missed (or ignored) my point. They want the macro to expand to empty when it has not been explicitly set for the purposes of using that makefile. It is a matter of luck whether the macro is actually unset when it has not been explicitly set for the purposes of using that makefile. It may have a non-empty default value in the version of make they are using, or they may have forgotten that they have an environment variable set that happens to have the same name but is set for a completely different purpose. > And they're also not wrong because up > until now (with the advent of ?=) there has been no other way for them > to get the behavior they want in POSIX standard makefiles. There are alternatives methods of achieving much the same thing. As Quentin pointed out, you can set the macro to an empty string in the makefile and then override that from the command line. (Someone said that this could have problems if the make invocation is "buried" and not readily alterable, but if the makefile is designed from the outset to be used this way, then that would not be the case.) You can also have use the include line mechanism to have all the makefiles include a configuration file containing default macro values that the user can edit. > GNU make has had an option that will warn when expanding unset > variables for almost 30 years. This is one reason I'm sure that > throwing an error is a complete non-starter. > > If we wanted to add such an option to the standard, or even an option > that forced an error, not just a warning, in this situation, I would be > OK with that. > > Also, as far as I can tell there's nothing in the standard that says > that make can't show a warning message even as default behavior, if > some implementation wanted to do that. Then you didn't look far enough. Such as message is not allowed to standard error because STDERR says the usual "The standard error shall be used only for diagnostic messages" (which require non-zero exit). And it's not allowed to standard output because there is nothing in the STDOUT section that explicitly allows it, and 1.4 Utility Description Defaults says that STDOUT "completely describes the standard output of the utility". -- Geoff Clare The Open Group, Apex Plaza, Forbury Road, Reading, RG1 1AX, England
Re: [Issue 8 drafts 0001505]: Make doesn't seem to specify unset macro expansion behaviour
lenn...@getcoding.de wrote, on 17 Dec 2021:
>
> Quoth Geoff Clare via austin-group-l at The Open Group:
> > They rely not only on the behaviour of make but also on macros being
> > unset by default. They are wrong to rely on the latter, because it
> > can easily not be the case depending on circumstances.
>
> They don’t. They rely on the Makefile working both if the macro happens to
> be unset and if it happens to be set to something useful.
Okay, I'll give you that. So the problem we are trying to prevent
occurs if the macro happens to be set to something unexpected instead
of something useful.
> consider a rule like
>
> .c.o:
> ${CC} -c ${CFLAGS} ${CPPFLAGS} $<
>
> CFLAGS is set, but CPPFLAGS might not be. If User wants extra options for
> the preprocessor, he sets it in the environment or on the command line. If
> he doesn’t want them, he doesn’t set it. We certainly don’t override User’s
> settings.
>
> This is common (and useful). We should standardize this.
No, because it is relying on something that is unreliable, and it
would be a good thing for the makefile author to be warned of this.
There is nothing to prevent a make implementation providing a
non-empty default for a CPPFLAGS macro that is completely unrelated
to C compilation. Or a user could have an environment variable
called CPPFLAGS that they have to set for some "Corporate Pension
Processing" application that they use.
In either of these cases, the .c.o rule will do something unexpected.
> > Many utilities that process files will read from standard input if no
> > files are specified. To a make user it would seem like the system is
> > very slow or has frozen, and the problem could be hard to debug
> > depending on their level of experience.
> >
> > If cd is given no argument it will change to $HOME instead of the
> > intended directory, potentially resulting in subsequent commands
> > wreaking havoc with the user's files.
> >
> > If GNU find is given no pathnames it traverses the current directory,
> > again potentially causing the wrong files to be operated on.
> >
> > In all of these cases it would be greatly preferable for the user to
> > get an error message from make about an unset macro than to experience
> > these misbehaviours.
>
> And in those cases, the author doesn’t depend on macros being set or not.
The author depends on the macros being set to a non-empty value.
This could easily happen with a makefile that is designed to take a
macro value from the command line or the environment. The author would
do this in all of their testing. But if they forget to document that
it needs to be done (or a user doesn't bother to read the documentation)
then a user might run make without the necessary value supplied on the
command line or via the environment, and unexpected things will happen.
--
Geoff Clare
The Open Group, Apex Plaza, Forbury Road, Reading, RG1 1AX, England
Re: [Issue 8 drafts 0001505]: Make doesn't seem to specify unset macro expansion behaviour
Hi Robert,
> bsd make (bmake) has a couple of macros (the doc calls what are
> macros here variables, but they are the same thing), which default to
> being unset, which can be set in the environment or command line,
> but which cannot be set in any Makefile, as they alter the interpretation
> of makefiles (slightly). That is, if they are going to be defined, that
> needs to have happened before the first makefile line is read.
> They are used (expanded) in makefiles quite commonly.
>
> For that to work, no rule requiring variables to be always set in the
> Makefile before being expanded is possible - unless those vars were
> made an exception (which would be ugly). Note, I have no idea if
> bmake tries to enforce the "must not be set in the makefile" rule
> or not, but I know the overall system cannot work if that happens.
That sounds problematic indeed.
A questions then, would requiring that non-reserved macros *can't* be
set within a makefile be conformant to POSIX make?
If not, either POSIX should be fixed to allow this, or bsdmake to allow
that.
> Second, a common practice, particularly in makefile fragments
> which are included in other makefiles is to do something like
>
> MACRO+= string
>
> eg: CFLAGS+= -O2
>
> Such a fragment cannot do
> MACRO=
> for the (I hope) obvious reasons.
>
> The += is effectively shorthand for (not possible in make I think)
> MACRO= ${MACRO} string
> an so is effectively expanding the otherwise potentially unset MACRO
> before altering it. Any implementation which generated an error on
> an expansion of an unset macro would need to generate an error for
> this usage (assuming MACRO was unset previously) as well.
>
> To me this seems to be a very common idiom with no easy fix,
> other than making this an exception in the standard.
I don't think that this case is different from classical case.
MACRO =
MACRO += string
The final MACRO will still be “string”.
> Let's just do the only sane thing, implement the request in the
> desired action, which is clearly the right thing to do, and specify
> that if a macro happens to not have been defined when being expanded,
> it shall expand to nothing. No alternatives allowed in a fully
> conforming (unextended) implementation.
I think that both proposed solutions have benefits and drawbacks, it's
not “the only sane thing” vs. “the rest of those crazies”.
> By all means add an application usage note warning users about assuming
> macros will be unset initially, and advising that makefiles explicitly
> set macros they use, if that seems like wise advice (those needing to
> do otherwise will simply ignore that advice.)
Yes, I agree after those exchanges that this would be the minimal thing
to do in either case.
Re: [Issue 8 drafts 0001505]: Make doesn't seem to specify unset macro expansion behaviour
Hi Ukko, > Standardising that unset variables expand to nothing doesn't mean that > makefile authors will now be able to depend on some variables always > expanding to the empty string. It however standardises that all unset > variables will expand to nothing unless set in environment, builtin in the > make implementation, command line, or some non-standard way (which is what > "unset" means). This is true. However, from a specification perspective, how is that useful if you can't rely on the first part of what you said? > There is a whole bunch of perfectly fine working makefiles > relying on that. And, frankly, make is quite far from a rapidly evolving > software/language, nobody will introduce breaking changes in their > implementations. > > Someone mentioned new user frustration earlier which ends with "I guess > this is just how make works". Having a make implementation that errors on > unset variables would be so much more frustrating than forgetting to > initialise, eg, CC, to a sane default value. That's not a very good example, as CC is specified to be set by the implementation, the user don't have to set it. Unless the implementation doesn't support C development utilities, but then I suppose that's actually something you'd want, as a user, to be notified about instead of running blind commands. This is the kind of arguments that make me wonder how much the arguments are made from a “I just don't want to be bothered” perspective and “I prefer to have a more robust system”. I think that both perspectives are valid, but maybe not necessarily for specifying software behaviour. > In any case, I think two different users should be considered: the ones > writing makefiles (authors) and the ones who just run `make install` (end > users). While errors/warnings about unset variables might be helpful to > the authors, I, as an end user, would absolutely hate having to patch every > single makefile to get rid of errors like "attempting to expand DESTDIR > which is unset" for every thing I'd like to install. And not all makefiles > are human written and human readable. Well, that's something put upon the author, not the user. A broken makefile is a makefile broken by the author, whatever the cause of that breakage. > I, personally, think the standard should require unset variables to expand > to nothing or empty string if there's any difference in make (or allow > both). A standardised option enabling warnings might be a nice addition > but I'm not sure that is required as authors writing makefiles would > probably be using non-standard stuff locally anyways. The point, I suppose, of being in favor to warn or error on the use of unset macros, is that while it is being used in the wild (and that's true it's used a lot) it can't be relied upon from a portability and robustness perspective. I might or might not work. Albeit it should work most of the time. As you said earlier, it's a “trick”. Now the role of a standard like POSIX is indeed to describe the different already existing usages. It's also trying to make a consistent and robust specification. Enabling expanding unset macros to empty would just make it not undefinied behaviour. It would not enable authors to rely on an unset macro being empty, as you said it yourself in the introduction of this message. And yet most people would assume that it's safe because that's a specified behaviour. It think that if POSIX chose this (I'm not personnaly against, I don't have a set opinion on this), it would be good to have alongside a warning by default and/or an option to enable to error on cases like this. This would help authors realize this trick cannot be relied upon and then make the decision in full knowledge. > Hope I managed to make my point clear and that gmail doesn't mess up > sending the email. Clear, thanks for your input!
Re: [Issue 8 drafts 0001505]: Make doesn't seem to specify unset macro expansion behaviour
To hopefully finally let this discussion end, let me point out two
more issues.
bsd make (bmake) has a couple of macros (the doc calls what are
macros here variables, but they are the same thing), which default to
being unset, which can be set in the environment or command line,
but which cannot be set in any Makefile, as they alter the interpretation
of makefiles (slightly). That is, if they are going to be defined, that
needs to have happened before the first makefile line is read.
They are used (expanded) in makefiles quite commonly.
For that to work, no rule requiring variables to be always set in the
Makefile before being expanded is possible - unless those vars were
made an exception (which would be ugly). Note, I have no idea if
bmake tries to enforce the "must not be set in the makefile" rule
or not, but I know the overall system cannot work if that happens.
Second, a common practice, particularly in makefile fragments
which are included in other makefiles is to do something like
MACRO+= string
eg: CFLAGS+= -O2
Such a fragment cannot do
MACRO=
for the (I hope) obvious reasons.
The += is effectively shorthand for (not possible in make I think)
MACRO= ${MACRO} string
an so is effectively expanding the otherwise potentially unset MACRO
before altering it. Any implementation which generated an error on
an expansion of an unset macro would need to generate an error for
this usage (assuming MACRO was unset previously) as well.
To me this seems to be a very common idiom with no easy fix,
other than making this an exception in the standard.
Let's just do the only sane thing, implement the request in the
desired action, which is clearly the right thing to do, and specify
that if a macro happens to not have been defined when being expanded,
it shall expand to nothing. No alternatives allowed in a fully
conforming (unextended) implementation.
By all means add an application usage note warning users about assuming
macros will be unset initially, and advising that makefiles explicitly
set macros they use, if that seems like wise advice (those needing to
do otherwise will simply ignore that advice.)
kre
Re: [Issue 8 drafts 0001505]: Make doesn't seem to specify unset macro expansion behaviour
17 Aralık 2021 Cuma tarihinde Quentin Rameau via austin-group-l at The Open Group yazdı: > > The standard could state that the result of trying to expand an unset > macro is implementation-defined, > But that's not the case; the widely-adopted behavior is that unset macros expand to empty string. And if there is no make implementation that does otherwise, what is there to argue? -- Oğuz
Re: [Issue 8 drafts 0001505]: Make doesn't seem to specify unset macro expansion behaviour
Quentin Rameau wrote: > > That was the actual standard at the time, and has been ever since. > > That POSIX doesn't contain a sentence like that is a bug in POSIX, > > but it doesn't change the actual standard. > > I'm not sure about what standard you're talking, the behaviour isn't > specified in the different make implementation manuals I could read. It might not be explicitly worded in a standard but it's a de facto / living standard which is what POSIX should be trying to describe. It might not be mentioned in all make manuals but it definitely is mentioned in various "Advanced Make Tips & Tricks" and the like. Standardising that unset variables expand to nothing doesn't mean that makefile authors will now be able to depend on some variables always expanding to the empty string. It however standardises that all unset variables will expand to nothing unless set in environment, builtin in the make implementation, command line, or some non-standard way (which is what "unset" means). There is a whole bunch of perfectly fine working makefiles relying on that. And, frankly, make is quite far from a rapidly evolving software/language, nobody will introduce breaking changes in their implementations. Someone mentioned new user frustration earlier which ends with "I guess this is just how make works". Having a make implementation that errors on unset variables would be so much more frustrating than forgetting to initialise, eg, CC, to a sane default value. In any case, I think two different users should be considered: the ones writing makefiles (authors) and the ones who just run `make install` (end users). While errors/warnings about unset variables might be helpful to the authors, I, as an end user, would absolutely hate having to patch every single makefile to get rid of errors like "attempting to expand DESTDIR which is unset" for every thing I'd like to install. And not all makefiles are human written and human readable. I, personally, think the standard should require unset variables to expand to nothing or empty string if there's any difference in make (or allow both). A standardised option enabling warnings might be a nice addition but I'm not sure that is required as authors writing makefiles would probably be using non-standard stuff locally anyways. Hope I managed to make my point clear and that gmail doesn't mess up sending the email. -- ukko
Re: [Issue 8 drafts 0001505]: Make doesn't seem to specify unset macro expansion behaviour
On Fri, 2021-12-17 at 21:34 +0100, Quentin Rameau via austin-group-l at The Open Group wrote: > Why not allowing in the end both behaviours? I feel like we've gone over the problem with this. While the current standard may not guarantee that unset variables expand to the empty string, virtually all makefiles--including intending-to-be conforming makefiles--assume that it does. If you allow a conforming make implementation to fail if it expands an unset variable, you're effectively saying that the only way to have a makefile that will work on all POSIX systems (which is, after all, the entire point of writing a makefile to conform to standard) is to write makefiles that never reference any unset variables. If you write a makefile which does reference an unset variable then your perfectly conforming makefile can fail with a perfectly conforming implementation of make which is clearly a bad situation for the standard. We have two choices: we can add in the text that every makefile author and implementer has assumed was there since the standard was created and require unset variables to expand to an empty string, retroactively blessing all otherwise-conforming makefiles as actually conforming. Or we can allow conforming make implementations to fail, consigning almost every makefile currently existing as allowed to fail on a conforming system and likely rendering the standard irrelevant, at least in this particular way, as authors ignore it. I don't see any way in which the latter is a preferred option. > The standard could state that the result of trying to expand an unset > macro is implementation-defined, this way implementations that want > to expand thoses as an empty string can, and also provide an option > to error out on such behaviour (-u for example, a bit like sh set > -u), for helping the people who want so making their makefiles safer. The standard needs to require that in a conforming make implementation unset variables will expand to the empty string, full stop, for the reasons I've outlined above. There should be no "implementation defined" behavior here. Beyond that, other proposals, such as a new make option that would enable a warning or even an error if unset variables are expanded, would be OK with me (although I don't see why the standard needs to get out in front of implementations on what is basically a quality-of- implementation issue).
Re: [Issue 8 drafts 0001505]: Make doesn't seem to specify unset macro expansion behaviour
Hi Paul, Geoff, > > The more I think about it, the more I am convinced that an error > > is the right thing for make to do, and I would urge you and the > > other implementors to provide a means (e.g. a command line option) > > to turn it on, and have manuals encourage its use. > > GNU make has had an option that will warn when expanding unset > variables for almost 30 years. This is one reason I'm sure that > throwing an error is a complete non-starter. > > If we wanted to add such an option to the standard, or even an option > that forced an error, not just a warning, in this situation, I would be > OK with that. Why not allowing in the end both behaviours? The standard could state that the result of trying to expand an unset macro is implementation-defined, this way implementations that want to expand thoses as an empty string can, and also provide an option to error out on such behaviour (-u for example, a bit like sh set -u), for helping the people who want so making their makefiles safer.
Re: [Issue 8 drafts 0001505]: Make doesn't seem to specify unset macro expansion behaviour
On Fri, 2021-12-17 at 21:16 +0100, Quentin Rameau via austin-group-l at The Open Group wrote: > I'm not sure about what standard you're talking, the behaviour isn't > specified in the different make implementation manuals I could read. GNU make has an section that discusses empty variables versus undefined variables, which explains that both expand to the empty string.
Re: [Issue 8 drafts 0001505]: Make doesn't seem to specify unset macro expansion behaviour
On Fri, 2021-12-17 at 21:04 +0100, Quentin Rameau via austin-group-l at The Open Group wrote: > Again, relying on a variable being unset was never 100% safe. This same argument could be made about the shell. Why don't we, while we're at it, make a change to POSIX allowing implementations of sh to either expand an unset variable to the empty string, like every shell does today, or else throw an error and fail, at the discretion of the implementor? Both implementations are equally acceptable and both would be considered standards-conforming. Clearly the latter behavior would result in fewer accidental errors, and clearly it doesn't force any existing shell to change its behavior, so why not do it?
Re: [Issue 8 drafts 0001505]: Make doesn't seem to specify unset macro expansion behaviour
Robert, > Date:Fri, 17 Dec 2021 19:52:37 +0100 > From:"Quentin Rameau via austin-group-l at The Open Group" > > Message-ID: <20211217195237.7780b91e.quinq@fifth.space> > > | That was just one example out of two, then you can simply use > | make VERBOSE=set and be done with it :) > > One can, but the question here was never how to overcome a meaningless > proposed change to the standard, but why anyone would have to. > > The whole issue was that, years ago, no-one bothered to write in the > POSIX standard (or whichever of those which came before it this > actually first appeared in) "if a macro is not defined, expanding it > produces a null string" or words to that effect. Yes, that's why I asked the question. > That was the actual standard at the time, and has been ever since. > That POSIX doesn't contain a sentence like that is a bug in POSIX, > but it doesn't change the actual standard. I'm not sure about what standard you're talking, the behaviour isn't specified in the different make implementation manuals I could read. > The proposal is to make such a change - which (from what little I understand > of what the process is) would have required sending this off somewhere > for an "interpretation" whatever that actually means, but in this case > this would, rather than the traditional > > The standard says xxx but implementations have always done yyy, > so we propose to change the standard to match the real world > > (not those words, but that basic meaning) > > (or something similar when there is no actual real world standard, and > POSIX is to be changed to make something unspecified) > > instead say something like > > The real world standard is clear, all implementations implement > it, but we didn't ever write that down, so now we want to take > the opportunity to break everything, because it seems better to > a few of us > > Good luck getting that approved. I'm not sure this kind of tone is constructive on this mailing list, but I doubt that the intention of the resolution proposal was to “break everything”. It's simple to raise concerns in a courteous manner, and for as long as I've read the exchanges on the austin group mailing list, I've seen some reasonable discussions in order to solve problems, which is the goal of it. I'm sure that people will take your proposal to come to a consensus, if you make any, for the better of the standard!
Re: [Issue 8 drafts 0001505]: Make doesn't seem to specify unset macro expansion behaviour
Hello Robert, I'm sorry I didn't see your messages earlier, your emails were not properly linked to the POSIX mailing list locally. > Date:Thu, 16 Dec 2021 08:36:11 +0100 > From:"Quentin Rameau via austin-group-l at The Open Group" > > Message-ID: <20211216083611.209af10a.quinq@fifth.space> > > Sorry, this combination of statements: > > | It's not safe to rely on a macro being unset, as a make implementation > | is free to provide its own internal macro that might conflict with the > | macro the user hope to be left unset. > > and > > | > Of course it would be possible to require all these makefiles to be > | > rewritten as: > | > > | >VERBOSE ?= > | >$(VERBOSE).SILENT: > > | That would indeed be the correct way to do it, you found it yourself so > | anybody else can do it to! > > makes absolutely no sense. > > | Point is if one expects an empty variable by default, it should be set > | explicitely as empty by default. > > What difference does that make to the supposed reason that it is unsafe > to assume it will be unset by default. > > If "a make implementation is free to provide its own internal macro that > might conflict with the macro the user hope to be left unset" (VERBOSE > in the case of the example) then how would that method make any difference > at all? > > VERBOSE would still be set, regardless of where that comes from, and the > explicit setting to empty would not happen. Yes, it would be set to an empty string, by default, as what was the use-case that was given as an example by somebody here. > The only way to avoid that would be to simply do > > VERBOSE= > > instead of VERBOSE?= > > | As a note, it's unneeded to use the ?= operator, classical = operator > | is enough, macros specified on the command line overtake the one > | defined in make files, and environment ones do that with -e option if > | that's what the user wants. > > And then the user needs to know to use the -e option, and after all of > this, how exactly does any of this justify make generating an error when > expanding undefined variables. Yes, make users are expected to know make options. I think that the given justification was to help users being aware that relying on a variable being unset isn't portable by nature, although it should work “most of the time”. Now, should this be made the default behaviour or an option, like the shell having set -u for example, I think remains debatable. > To me this smells far too much like like a bad experience someone had - a > typo in a macro name in a makefile or something, thus generating nothing > where some other value was expected, and an "I wish make had generated an > error so I would have found my typo earlier" moment - that is now being > foisted upon every makefile author on the planet (it won't affect make > implementations, none of those will change to actually generate that error > message, that would break backward compatibility far too badly to contemplate) > but makefile authors will no longer be able to 100% safely rely upon what > has always been true. Again, relying on a variable being unset was never 100% safe. > | I hope those explanations helped understanding the decision proposal, > | as I understood them. > > No. They don't at all. > > Furthermore, and once again, it is this groups responsibility to document > the standard, not to invent a utopia (even if that were agreed) and attempt > to mandate that.Doing more like this will just reinforce the "Isn't > POSIX irrelevant these days?" question that I hear more and more often. The goals of POSIX, as I understand them, isn't to blindly apply observable behaviour, it's also to make a strong standard that leaves ambiguity in behaviour, and it shouldn't just apply a rule because some might use a feature, is said feature cannot be speficied safely, or if it conflicts with existing standard. But that's a bit on a more general topic, let's keep on the matter at hand.
Re: [Issue 8 drafts 0001505]: Make doesn't seem to specify unset macro expansion behaviour
On Fri, 2021-12-17 at 19:52 +0100, Quentin Rameau via austin-group-l at The Open Group wrote: > > On Fri, 2021-12-17 at 18:19 +0100, Quentin Rameau via austin-group- > > l atThe Open Group wrote: > > > Then one can just call make VERBOSE=set, or VERBOSE=set make -e > > I already explained in a previous post why 'make -e' is not > > acceptable. It's a much, much more dangerous than allowing unset > > variables. > > That was just one example out of two, then you can simply use make > VERBOSE=set and be done with it :) There are a number of types of build environments where it's straightforward to add a new environment variable to the build's configuration, but it's not so simple (or even not possible) to add a new command line argument to the invocation of make, which might be buried inside lots of scripting or invoked by other tools. In short, people have evolved their systems and processes to utilize the features provided to them. If you remove those features you'll break things. We should not break things.
Re: [Issue 8 drafts 0001505]: Make doesn't seem to specify unset macro expansion behaviour
Hello Paul, > On Fri, 2021-12-17 at 18:19 +0100, Quentin Rameau via austin-group-l at > The Open Group wrote: > > Then one can just call make VERBOSE=set, or VERBOSE=set make -e > > I already explained in a previous post why 'make -e' is not acceptable. > It's a much, much more dangerous than allowing unset variables. That was just one example out of two, then you can simply use make VERBOSE=set and be done with it :)
Re: [Issue 8 drafts 0001505]: Make doesn't seem to specify unset macro expansion behaviour
On Fri, 2021-12-17 at 18:19 +0100, Quentin Rameau via austin-group-l at The Open Group wrote: > Then one can just call make VERBOSE=set, or VERBOSE=set make -e I already explained in a previous post why 'make -e' is not acceptable. It's a much, much more dangerous than allowing unset variables. If it were possible to do something like "make -e VERBOSE" so that the override only impacted that one variable I guess it would be a way to go, but -e forces ALL environment variables to take precedence over ALL makefile-set variables.
Re: [Issue 8 drafts 0001505]: Make doesn't seem to specify unset macro expansion behaviour
As a note, > until now (with the advent of ?=) there has been no other way for them > to get the behavior they want in POSIX standard makefiles. .POSIX: VERBOSE = $(VERBOSE).SILENT: That's the vay to do it in standard POSIX makefiles, isn't it? Then one can just call make VERBOSE=set, or VERBOSE=set make -e
Re: [Issue 8 drafts 0001505]: Make doesn't seem to specify unset macro expansion behaviour
On 12/17/21 5:11 AM, Geoff Clare via austin-group-l at The Open Group wrote: The more I think about it, the more I am convinced that an error is the right thing for make to do, The world is an imperfect place. It seems that few, if any, make implementations agree. We can't start standardizing behavior that no one implements because of a desire for possible future improvement. That's what gives standards bodies a bad name. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: [Issue 8 drafts 0001505]: Make doesn't seem to specify unset macro expansion behaviour
On Fri, 2021-12-17 at 10:11 +, Geoff Clare via austin-group-l at The Open Group wrote: > Paul Smith wrote, on 16 Dec 2021: > > I'm somewhat nonplussed by this conversation. > > > > We have a POSIX tool, that's been in use for 45 years. All known > > implementations of that tool provide a specific behavior. No known > > implementations of that tool provide some different behavior. Users > > of the tool have written hundreds of thousands of files that expect > > and rely on the provided behavior and feel that it's valuable. > > They rely not only on the behaviour of make but also on macros being > unset by default. They are wrong to rely on the latter, because it > can easily not be the case depending on circumstances. They are not wrong, because that's exactly the behavior they want: they want the variable to expand to empty when it's not set, and expand to something else when it is set. And they're also not wrong because up until now (with the advent of ?=) there has been no other way for them to get the behavior they want in POSIX standard makefiles. It may be that they were relying on behavior that is not explicitly stated. But I can't understand why the right answer to that problem is not to standardize the behavior that everyone gets today and expects to get, and instead allow behaviors which would break all those assumptions. Whether we wish that some other behavior had been chosen 45 years ago, is irrelevant. POSIX standardizes what is, not what might have been. > I would not be surprised if there have been many cases over those 45 > years where this very problem has arisen and the makefile author has > simply fixed it either by changing the macro name or adding an > assignment. Users would tend to think "that's how make works, so > it's my problem and anyway it's simple to fix" rather than trying to > get make changed so as to prevent the same problem arising in future. make cannot ever change in this way, practically, because that would break almost every makefile in existence. There are many annoying things required by the POSIX standard because "that's how it's always worked" and since all implementations have that behavior, it's added to the standard. Such is life. > > Which by the way, no implementation will ever do > > because it would break the previously-mentioned hundreds of > > thousands of files. > > If you are so sure that no implementation will ever do that, then I > don't see that you have any reason to complain about the proposed > change to the standard. I'm not speaking as the maintainer of GNU make here. This change makes no difference to me as the maintainer because GNU make will certainly never change this behavior. I'm speaking on behalf of all the users who maintain makefiles. This has a big impact on them, unless they simply want to say "we won't bother to care about ensuring that our makefiles are POSIX compliant" which benefits no one, most especially POSX itself. > The more I think about it, the more I am convinced that an error > is the right thing for make to do, and I would urge you and the > other implementors to provide a means (e.g. a command line option) > to turn it on, and have manuals encourage its use. That is a QOI issue, not a standards issue. GNU make has had an option that will warn when expanding unset variables for almost 30 years. This is one reason I'm sure that throwing an error is a complete non-starter. If we wanted to add such an option to the standard, or even an option that forced an error, not just a warning, in this situation, I would be OK with that. Also, as far as I can tell there's nothing in the standard that says that make can't show a warning message even as default behavior, if some implementation wanted to do that. But, it must absolutely not be the case that the standard allows implementations to throw an error when expanding an unset variable as the default behavior.
Re: [Issue 8 drafts 0001505]: Make doesn't seem to specify unset macro expansion behaviour
On 12/17/21 5:11 AM, Geoff Clare via austin-group-l at The Open Group wrote: Currently POSIX does not require unset macros to expand to an empty string. The standard is silent on the matter, so the behaviour is implicitly unspecified. It seems like this is an opportunity to standardize behavior that is common across multiple (all?) make implementations. The proposed change *reduces* the allowed behaviours from many to just two. If all make implementations have the same behavior, why not standardize that? Is there evidence that the "typo in the makefile" problem is widespread enough to devise and require a hypothetical fix in make? -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: [Issue 8 drafts 0001505]: Make doesn't seem to specify unset macro expansion behaviour
Date:Fri, 17 Dec 2021 10:11:44 + From:"Geoff Clare via austin-group-l at The Open Group" Message-ID: <20211217101144.GA11681@localhost> | They rely not only on the behaviour of make but also on macros being | unset by default. They are wrong to rely on the latter, because it | can easily not be the case depending on circumstances. That might ir might not be a bug in the makefiles, like many things whether considered as a bug or feature turns very much upon the impact it has on the individual concerned. It is not, however, a bug in make. | I would not be surprised if there have been many cases over those 45 | years where this very problem has arisen I don't know that I can speak to many, but I am certain it has happened. So? | and the makefile author has simply fixed it either by changing the | macro name or adding an assignment. If the latter fixes anything we are takking about a different kind of problem. My experience is that the usual solution is to remove the offending assignment. | Users would tend to think "that's how make works, so | it's my problem and anyway it's simple to fix" rather than trying to | get make changed so as to prevent the same problem arising in future. It is how it works, and it is their problem, and there is no simple way to change make to avoid the problem. Certainly the disputed change here is not doing that. | Currently POSIX does not require unset macros to expand to an empty | string. The standard is silent on the matter, so the behaviour is | implicitly unspecified. That is a problem, and should be fixed. | The proposed change *reduces* the allowed behaviours from many to | just two. The correct change would do even better and reduce that to one. | If you are so sure that no implementation will ever do that, then I | don't see that you have any reason to complain about the proposed | change to the standard. Because Makefile authors do not like to rely upon unspecified behaviour, and would be reluctant to rely upon no new make imolementatiin ever (really ever) defaulting to generating errors if an unset macro us expanded. There would need to be lots of work, which no-one woukd ever know was done correctly, as no implementations actually generate the error they would be protecting against. That means lots of untestable code, and inevitably, many bugs. | If you are right, then the reduction in | allowed behaviours will affect nobody. If that was correct no-one would care. But many people do care. Can you draw the obvious conclusion? | If you are wrong, and at some | point in the future implementors and users of make start to realise | that it would be a good idea for make to report the use of unset | macros as an error, then it will be harder for implementors to make | that change if the next revision of the standard disallows it. The only conceivable way such a change could ever be made would be as a non-standard option, which if set be the user would take them outside the standard, so this would not be an issue. Should such an option ever become popular, it could then be standardised. | and I would urge you and the | other implementors to provide a means (e.g. a command line option) | to turn it on, Try submitting a bug report to the maintainers of any versions of make you use and request that chwnge. It will be better received if accompanied by well tested code. I see no reason such an option could not be added, but using ut will have zero impact upon the issue yiu seem most focussed upon as a bug. | There is also the opposite problem to consider. If | the makefile uses a macro that it does not set, Yes, that is the typo in the makefile problem. Generating an error would help catch those, but tgey are generally obvious (and this one is always a makefile bug, not user contributed, so can easily be reproduced and fixed) Unfortunately the cost to provide the means to trivially fix that is simply too great to consider. kre
Re: [Issue 8 drafts 0001505]: Make doesn't seem to specify unset macro expansion behaviour
Quoth Geoff Clare via austin-group-l at The Open Group:
They rely not only on the behaviour of make but also on macros being
unset by default. They are wrong to rely on the latter, because it
can easily not be the case depending on circumstances.
They don’t. They rely on the Makefile working both if the macro
happens to be unset and if it happens to be set to something useful.
I would not be surprised if there have been many cases over those 45
years where this very problem has arisen and the makefile author has
simply fixed it either by changing the macro name or adding an
assignment. Users would tend to think "that's how make works, so
it's my problem and anyway it's simple to fix" rather than trying to
get make changed so as to prevent the same problem arising in future.
I doubt that. But even if so, those cases would be far less than
cases where having unset macros expand to nothing is expected.
There's no role for POSIX, IMO, to say that this behavior can no longer
be relied on and some future implementation may instead substitute
another behavior.
Currently POSIX does not require unset macros to expand to an empty
string. The standard is silent on the matter, so the behaviour is
implicitly unspecified.
So we should fix it and require unset macros to expand to nothing.
So far the discussion has been about makefiles where a macro is
unset for the author but might be non-empty on other systems or for
other users. There is also the opposite problem to consider. If
the makefile uses a macro that it does not set, and for the author
it is non-empty and works fine, there is the possibility that on other
systems or for other users the macro will be unset and will expand to
an empty string, and this will cause commands to misbehave.
To me, the discussion seems to be about Makefiles where a macro may be
set or unset, both on the author’s and other users’ systems, Makefiles
where a macro being set or not decides on which expected behavior is
to be taken.
In Paul Smith’s example, it decides on whether make is .SILENT:. Or,
consider a rule like
.c.o:
${CC} -c ${CFLAGS} ${CPPFLAGS} $<
CFLAGS is set, but CPPFLAGS might not be. If User wants extra options
for the preprocessor, he sets it in the environment or on the command
line. If he doesn’t want them, he doesn’t set it. We certainly don’t
override User’s settings.
This is common (and useful). We should standardize this.
Many utilities that process files will read from standard input if no
files are specified. To a make user it would seem like the system is
very slow or has frozen, and the problem could be hard to debug
depending on their level of experience.
If cd is given no argument it will change to $HOME instead of the
intended directory, potentially resulting in subsequent commands
wreaking havoc with the user's files.
If GNU find is given no pathnames it traverses the current directory,
again potentially causing the wrong files to be operated on.
In all of these cases it would be greatly preferable for the user to
get an error message from make about an unset macro than to experience
these misbehaviours.
And in those cases, the author doesn’t depend on macros being set or
not.
--
Lennart Jablonka
Re: [Issue 8 drafts 0001505]: Make doesn't seem to specify unset macro expansion behaviour
Paul Smith wrote, on 16 Dec 2021: > > I'm somewhat nonplussed by this conversation. > > We have a POSIX tool, that's been in use for 45 years. All known > implementations of that tool provide a specific behavior. No known > implementations of that tool provide some different behavior. Users of > the tool have written hundreds of thousands of files that expect and > rely on the provided behavior and feel that it's valuable. They rely not only on the behaviour of make but also on macros being unset by default. They are wrong to rely on the latter, because it can easily not be the case depending on circumstances. I would not be surprised if there have been many cases over those 45 years where this very problem has arisen and the makefile author has simply fixed it either by changing the macro name or adding an assignment. Users would tend to think "that's how make works, so it's my problem and anyway it's simple to fix" rather than trying to get make changed so as to prevent the same problem arising in future. > There's no role for POSIX, IMO, to say that this behavior can no longer > be relied on and some future implementation may instead substitute > another behavior. Currently POSIX does not require unset macros to expand to an empty string. The standard is silent on the matter, so the behaviour is implicitly unspecified. The proposed change *reduces* the allowed behaviours from many to just two. > Which by the way, no implementation will ever do > because it would break the previously-mentioned hundreds of thousands > of files. If you are so sure that no implementation will ever do that, then I don't see that you have any reason to complain about the proposed change to the standard. If you are right, then the reduction in allowed behaviours will affect nobody. If you are wrong, and at some point in the future implementors and users of make start to realise that it would be a good idea for make to report the use of unset macros as an error, then it will be harder for implementors to make that change if the next revision of the standard disallows it. The more I think about it, the more I am convinced that an error is the right thing for make to do, and I would urge you and the other implementors to provide a means (e.g. a command line option) to turn it on, and have manuals encourage its use. At some point in the future it could become the default and an option provided to turn it off, so as to ease the transition. So far the discussion has been about makefiles where a macro is unset for the author but might be non-empty on other systems or for other users. There is also the opposite problem to consider. If the makefile uses a macro that it does not set, and for the author it is non-empty and works fine, there is the possibility that on other systems or for other users the macro will be unset and will expand to an empty string, and this will cause commands to misbehave. Many utilities that process files will read from standard input if no files are specified. To a make user it would seem like the system is very slow or has frozen, and the problem could be hard to debug depending on their level of experience. If cd is given no argument it will change to $HOME instead of the intended directory, potentially resulting in subsequent commands wreaking havoc with the user's files. If GNU find is given no pathnames it traverses the current directory, again potentially causing the wrong files to be operated on. In all of these cases it would be greatly preferable for the user to get an error message from make about an unset macro than to experience these misbehaviours. -- Geoff Clare The Open Group, Apex Plaza, Forbury Road, Reading, RG1 1AX, England
Re: [Issue 8 drafts 0001505]: Make doesn't seem to specify unset macro expansion behaviour
I'm somewhat nonplussed by this conversation. The goal of POSIX (as I understand it) is to standardize existing behaviors of tools and libraries so that USERS of these tools and libraries can feel confident that when they move to another conforming platform, everything will still work the same way. We have a POSIX tool, that's been in use for 45 years. All known implementations of that tool provide a specific behavior. No known implementations of that tool provide some different behavior. Users of the tool have written hundreds of thousands of files that expect and rely on the provided behavior and feel that it's valuable. This is the quintessentially perfect situation for standardizing the in-use behavior. There's no role for POSIX, IMO, to say that this behavior can no longer be relied on and some future implementation may instead substitute another behavior. Which by the way, no implementation will ever do because it would break the previously-mentioned hundreds of thousands of files. On Thu, 2021-12-16 at 10:46 +, Geoff Clare via austin-group-l at The Open Group wrote: > Quentin Rameau wrote, on 16 Dec 2021: > > That would indeed be the correct way to do it, you found it > > yourself so anybody else can do it to! And so? Does the fact that it CAN be done mean it's OK to FORCE everyone to do it? > > As a note, it's unneeded to use the ?= operator, classical = > > operator is enough, macros specified on the command line overtake > > the one defined in make files, and environment ones do that with -e > > option if that's what the user wants. > > Classical = operator is exactly what it needed to avoid relying on > any given macro expanding to an empty string. Classical = is not sufficient because it overrides environment variables. Build systems want to allow environment variable settings to impact their behaviors. Whether we think it's the right thing or a bad idea or whatever is irrelevant. They want to do it, implementations let them do it, and it's not the place of the standard to tell them they can't. Also, using -e is a complete non-starter. I personally think that if our actual goal is to improve the safety and reliability of make-based build systems the very first thing we should do is remove -e. Using "make -e" is virtually guaranteed to break your build almost before it starts.
Re: [Issue 8 drafts 0001505]: Make doesn't seem to specify unset macro expansion behaviour
On 12/16/21 5:27 AM, Geoff Clare via austin-group-l at The Open Group wrote: > Chet Ramey wrote, on 14 Dec 2021: >> >> On 12/14/21 5:15 AM, Geoff Clare via austin-group-l at The Open Group wrote: >>> Paul Smith wrote, on 13 Dec 2021: Why shouldn't we just state that make implementations must expand unset variables to the empty string, which is what all implementations (that I'm aware of) do anyway? >>> >>> The point is that any makefile that relies on an unset macro being >>> expanded to an empty string is not portable. The only reason it ever >>> works is purely by luck. >> >> These two paragraphs are clearly in conflict. They can't both be true. > > They are both true, but I could perhaps have phrased it differently > to make it clearer why the second one is true. > > When a makefile relies on an unset macro being expanded to an empty > string, the reason it is not portable has nothing to do with the way > current make implementations expand unset macros, it is because the > makefile is relying on the macro being unset. Then it doesn't fully address the point. Whether you write a makefile one way or another, when standardizing make behavior we should give more weight to the behavior of current make implementations. If all existing make implementations expand unset macros to the empty string -- and no one has identified an implementation that does not -- then that is the behavior that should be standardized. > The only way to be sure that a given macro will expand to an empty > string is to explicitly set it to an empty string. This is just saying that you can never be sure that a macro is unset. However, if it is unset, make implementations should behave consistently, and it appears they do. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: [Issue 8 drafts 0001505]: Make doesn't seem to specify unset macro expansion behaviour
Date:Thu, 16 Dec 2021 10:46:57 + From:"Geoff Clare via austin-group-l at The Open Group" Message-ID: <20211216104657.GB18517@localhost> | Using ?= would prevent make reporting an error, but it would | reintroduce the very problem that this change in the standard is | intended to address. Which by itself shows that the change does not, and cannot, solve the perceived problem. Consider make CC='rm -rf' which is exactly... | It is a more serious problem with | macros that are used in command lines and can thus cause commands | to fail or do unexpected things. but which is not solved by this change at all, as CC isn't unset, even if not set on the command line. The only (minor) benefit the change gives is to catch typos in macro names, resulting in them being unset when it was intended that would not happen. The cost is that all current makefile systems have to be modified to insert lots of MACRO?= lines to set all the vars that are not to be set to a value by default, rewriting all the code that actually tests if a macro is unset, as after the previous change, none will be, abd all for nithing, as make implementations are not going to start generating errors for this (or do you know of one which is planning to do that? If so, which?) kre
Re: [Issue 8 drafts 0001505]: Make doesn't seem to specify unset macro expansion behaviour
Date:Thu, 16 Dec 2021 10:27:49 + From:"Geoff Clare via austin-group-l at The Open Group" Message-ID: <20211216102749.GA18517@localhost> | The only way to be sure that a given macro will expand to an empty | string is to explicitly set it to an empty string. Geoff, no-one wants a macro that is certain to expand to nothing, much less expecting to achieve that by using an undefined macro. What is wanted is that *if* a macro is undefined, then it expands to nothing, if it has some value, then it should expand to that value. That is what happens now, everywhere, is it not? What is *not* wanted is make generating errors because a macro happens to be undefined. And yes, this can mean that if the macro us set inappropriately bad things can happen. Such is life. kre
Re: [Issue 8 drafts 0001505]: Make doesn't seem to specify unset macro expansion behaviour
Date:Thu, 16 Dec 2021 08:36:11 +0100 From:"Quentin Rameau via austin-group-l at The Open Group" Message-ID: <20211216083611.209af10a.quinq@fifth.space> Sorry, this combination of statements: | It's not safe to rely on a macro being unset, as a make implementation | is free to provide its own internal macro that might conflict with the | macro the user hope to be left unset. and | > Of course it would be possible to require all these makefiles to be | > rewritten as: | > | >VERBOSE ?= | >$(VERBOSE).SILENT: | That would indeed be the correct way to do it, you found it yourself so | anybody else can do it to! makes absolutely no sense. | Point is if one expects an empty variable by default, it should be set | explicitely as empty by default. What difference does that make to the supposed reason that it is unsafe to assume it will be unset by default. If "a make implementation is free to provide its own internal macro that might conflict with the macro the user hope to be left unset" (VERBOSE in the case of the example) then how would that method make any difference at all? VERBOSE would still be set, regardless of where that comes from, and the explicit setting to empty would not happen. The only way to avoid that would be to simply do VERBOSE= instead of VERBOSE?= | As a note, it's unneeded to use the ?= operator, classical = operator | is enough, macros specified on the command line overtake the one | defined in make files, and environment ones do that with -e option if | that's what the user wants. And then the user needs to know to use the -e option, and after all of this, how exactly does any of this justify make generating an error when expanding undefined variables. To me this smells far too much like like a bad experience someone had - a typo in a macro name in a makefile or something, thus generating nothing where some other value was expected, and an "I wish make had generated an error so I would have found my typo earlier" moment - that is now being foisted upon every makefile author on the planet (it won't affect make implementations, none of those will change to actually generate that error message, that would break backward compatibility far too badly to contemplate) but makefile authors will no longer be able to 100% safely rely upon what has always been true. | I hope those explanations helped understanding the decision proposal, | as I understood them. No. They don't at all. Furthermore, and once again, it is this groups responsibility to document the standard, not to invent a utopia (even if that were agreed) and attempt to mandate that.Doing more like this will just reinforce the "Isn't POSIX irrelevant these days?" question that I hear more and more often. kre
Re: [Issue 8 drafts 0001505]: Make doesn't seem to specify unset macro expansion behaviour
Quentin Rameau wrote, on 16 Dec 2021: > > > Let me reiterate my example (which is used in every single automake > > generated makefile, plus many many others) from the comment I added to > > the issue: > > > > Makefiles contain this: > > > > $(VERBOSE).SILENT: > > > > Now when you run "make", the .SILENT: feature is enabled. If you run > > "make VERBOSE=1" (or any other value) then the .SILENT: feature is > > disabled. Up until now this has been perfectly correct. > > > > Of course it would be possible to require all these makefiles to be > > rewritten as: > > > >VERBOSE ?= > >$(VERBOSE).SILENT: > > > > (leaving aside that ?= is not part of the currently published version > > of the standard) but I see no reason to make the original behavior non- > > conforming, which immediately changes all these otherwise-correct > > makefiles to be potentially wrong. > > That would indeed be the correct way to do it, you found it yourself so > anybody else can do it to! Using ?= would prevent make reporting an error, but it would reintroduce the very problem that this change in the standard is intended to address. If the VERBOSE macro is unexpectedly set to non-empty (either as a default in a different make implementation or from a user's environment where it was set for some completely unrelated reason), then ?= will not do anything. This particular example is quite benign, as all that happens if VERBOSE is unexpectedly non-empty is that users see output that would otherwise be suppressed. It is a more serious problem with macros that are used in command lines and can thus cause commands to fail or do unexpected things. > As a note, it's unneeded to use the ?= operator, classical = operator > is enough, macros specified on the command line overtake the one > defined in make files, and environment ones do that with -e option if > that's what the user wants. Classical = operator is exactly what it needed to avoid relying on any given macro expanding to an empty string. -- Geoff Clare The Open Group, Apex Plaza, Forbury Road, Reading, RG1 1AX, England
Re: [Issue 8 drafts 0001505]: Make doesn't seem to specify unset macro expansion behaviour
Chet Ramey wrote, on 14 Dec 2021: > > On 12/14/21 5:15 AM, Geoff Clare via austin-group-l at The Open Group wrote: > > Paul Smith wrote, on 13 Dec 2021: > >> Why shouldn't we just state that > >> make implementations must expand unset variables to the empty string, > >> which is what all implementations (that I'm aware of) do anyway? > > > > The point is that any makefile that relies on an unset macro being > > expanded to an empty string is not portable. The only reason it ever > > works is purely by luck. > > These two paragraphs are clearly in conflict. They can't both be true. They are both true, but I could perhaps have phrased it differently to make it clearer why the second one is true. When a makefile relies on an unset macro being expanded to an empty string, the reason it is not portable has nothing to do with the way current make implementations expand unset macros, it is because the makefile is relying on the macro being unset. The only way to be sure that a given macro will expand to an empty string is to explicitly set it to an empty string. -- Geoff Clare The Open Group, Apex Plaza, Forbury Road, Reading, RG1 1AX, England
Re: [Issue 8 drafts 0001505]: Make doesn't seem to specify unset macro expansion behaviour
On Tue, 2021-12-14 at 10:15 +, Geoff Clare via austin-group-l at The Open Group wrote: > Effectively, if a makefile expects an unset macro to expand to an > empty string, this is a bug waiting to be triggered. Can you clarify the reason for this position? I don't agree with it. Is there any implementation of make anywhere that would treat this as a bug? There are hundreds of thousands of makefiles out there that rely on this behavior and there has never been a single report (that I'm aware of) that any of them have ever failed due to this. Let me reiterate my example (which is used in every single automake generated makefile, plus many many others) from the comment I added to the issue: Makefiles contain this: $(VERBOSE).SILENT: Now when you run "make", the .SILENT: feature is enabled. If you run "make VERBOSE=1" (or any other value) then the .SILENT: feature is disabled. Up until now this has been perfectly correct. Of course it would be possible to require all these makefiles to be rewritten as: VERBOSE ?= $(VERBOSE).SILENT: (leaving aside that ?= is not part of the currently published version of the standard) but I see no reason to make the original behavior non- conforming, which immediately changes all these otherwise-correct makefiles to be potentially wrong.
Re: [Issue 8 drafts 0001505]: Make doesn't seem to specify unset macro expansion behaviour
On 12/14/21 5:15 AM, Geoff Clare via austin-group-l at The Open Group wrote: > Paul Smith wrote, on 13 Dec 2021: >> Why shouldn't we just state that >> make implementations must expand unset variables to the empty string, >> which is what all implementations (that I'm aware of) do anyway? > > The point is that any makefile that relies on an unset macro being > expanded to an empty string is not portable. The only reason it ever > works is purely by luck. These two paragraphs are clearly in conflict. They can't both be true. Can anyone point to a make implementation that throws an error when expanding an unset variable? -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: [Issue 8 drafts 0001505]: Make doesn't seem to specify unset macro expansion behaviour
Date:Tue, 14 Dec 2021 10:15:12 + From:"Geoff Clare via austin-group-l at The Open Group" Message-ID: <20211214101512.GA21879@localhost> | The point is that any makefile that relies on an unset macro being | expanded to an empty string is not portable. The only reason it ever | works is purely by luck. That's nonsense, I agree with Paul, make expands unset macros to nothing. Always has. It is perfectly portable to rely upon that. | When moving to a different implementation | of make (or even a newer version of the same implementation) it might | no longer work because the new make might have a non-empty default | value for that macro That is part of the point. If the macro is set, we want to do something different. | Also, when the makefile is used by a different | user it might not work because that user happens to have set an | environment variable of the same name (for an unrelated purpose). Indeed it might not, as almost anything might not work if the user has something in their environment that causes an application (any application) to act differently than it otherwise would, and that was not intentional. No matter how much you try, this kind of issue is unavoidable as long as we permit applications to be tailored by users; and we do, all the time. | Effectively, if a makefile expects an unset macro to expand to an | empty string, this is a bug waiting to be triggered. Even if that were true, which I am not willing to waste the effort arguing, it is not the job of this group to "fix" that kind of bug. Let the users raise the issue with the maintainers of the implementations they use, and see what comes of that. | In such situations it is generally agreed that it is better for the | problem to be brought to the attention of the author during development, | rather than for it to become a ticking time bomb. In the situations Paul envisages, and similar ones I know of, the author is 100% aware of the issue, it is deliberately designed to work that way (it is, for example, how users can build debug versions of some applications, simply by setting the correct magic var (macro) when invoking make to build it). | This is, for example, why it is considered best practice for shell | script authors to put "set -u" at the top of scripts, I have no idea who considers that best practice, but it certainly is not me, and nor do I ever encounter much in the way of scripts that ever use set -u (not sure I remember a single one). This isn't helped by buggy implementations of that option in about half the available shells (including dash, yash, mksh, -- zsh too, though it seems to not care all that much about conforming, preferring to go its own way). 40 years now) - but leave the standard documenting what the implementations actually do. In this case, that is expanding unset macros to nothing - without error. kre
Re: [Issue 8 drafts 0001505]: Make doesn't seem to specify unset macro expansion behaviour
Paul Smith wrote, on 13 Dec 2021: > > Has anyone considered my comment on: > > https://austingroupbugs.net/view.php?id=1505 > > ? I'm unhappy with the resolution of this issue, to allow a conforming > make to throw an error when expanding an unset variable. No > implementation I'm aware of does that or ever did that, and by allowing > it we're suddenly changing tons of makefiles that would otherwise work > properly to be indeterminate since it's allowed for a conforming > implementation to throw such an error. > > What's the point of allowing this? Why shouldn't we just state that > make implementations must expand unset variables to the empty string, > which is what all implementations (that I'm aware of) do anyway? The point is that any makefile that relies on an unset macro being expanded to an empty string is not portable. The only reason it ever works is purely by luck. When moving to a different implementation of make (or even a newer version of the same implementation) it might no longer work because the new make might have a non-empty default value for that macro. Also, when the makefile is used by a different user it might not work because that user happens to have set an environment variable of the same name (for an unrelated purpose). Effectively, if a makefile expects an unset macro to expand to an empty string, this is a bug waiting to be triggered. In such situations it is generally agreed that it is better for the problem to be brought to the attention of the author during development, rather than for it to become a ticking time bomb. This is, for example, why it is considered best practice for shell script authors to put "set -u" at the top of scripts, or for perl script authors to put "use strict" there. To facilitate something similar in make, implementors may want to add an option to treat expansion of unset macros as an error (and encourage its use during makefile development), or, as with "use strict" in newer versions of perl, they may want to go a step further and have that be the default, adding an option to turn it off (for use with legacy makefiles that need that). The standard should allow both choices. -- Geoff Clare The Open Group, Apex Plaza, Forbury Road, Reading, RG1 1AX, England
Re: [Issue 8 drafts 0001505]: Make doesn't seem to specify unset macro expansion behaviour
Has anyone considered my comment on: https://austingroupbugs.net/view.php?id=1505 ? I'm unhappy with the resolution of this issue, to allow a conforming make to throw an error when expanding an unset variable. No implementation I'm aware of does that or ever did that, and by allowing it we're suddenly changing tons of makefiles that would otherwise work properly to be indeterminate since it's allowed for a conforming implementation to throw such an error. What's the point of allowing this? Why shouldn't we just state that make implementations must expand unset variables to the empty string, which is what all implementations (that I'm aware of) do anyway?
