bug#10828: [RFC] POSIX will say running "rm -f" with no argument is OK

[Karl Berry]

Hi Mike,

the issue isn't `rm -f` exactly, it's `rm -f` with a variable that
could be empty.

Sure.

i'm not sure we need to fully audit the tree 

Seems desirable to make a best effort to look at every viable case.

At any rate, it should be reproducible on OpenSolaris etc., as reported
by Bob F earlier in this report. So I hope Bob could try any proposed
patches and check for missing cases.

we could add a helper var to further collapse the boiler plate.
am__rm_f = rm -f $(am__rm_f_notfound)
then the rule is a bit more palatable:
$(am__rm_f) ...

Personally I find making "rm -f" into a variable less readable
than the original suggestion of:
rm -f ... $(am__rm_f_notfound)

The minor tweak that comes to my mind is that maybe the variable name
would be better as $(am__rm_f_notfound_file):
rm -f ... $(am__rm_f_notfound_file)

or if we want to claim namespace and make it something we want to export
for users of automake akin to AC_PROG_LN_S, we could use RM_F.

Our hope/belief is that the proposed change would silently fix the
problem.  In which case I don't see a win to doing something in user
namespace, having to document it, etc.

(Incidentally, I'd never want to define a variable as simple as "RM_F"
at this late date. Too much chance of collision ... AM_RM_F would be ok
if we wanted to go this route, but, as I say, I don't see a reason at
the moment.) --thanks, karl.

bug#10828: [RFC] POSIX will say running "rm -f" with no argument is OK

[Mike Frysinger]

On 15 Dec 2021 14:48, Karl Berry wrote:
> > rm -f NOTFOUND $(VAR)
> i think this is an interesting route.  
> 
> I agree.
> 
> we could do:
> rm -f $(am__rm_f_notfound) ...
> and am__rm_f_notfound could be set based on the configure test.
> 
> Sounds plausible to me. The "only" problem is that there are dozens or
> hundreds of places in the code that do rm -f. They would all have to be
> inspected and changed (or not) ... -k

the issue isn't `rm -f` exactly, it's `rm -f` with a variable that could be
empty.  so that narrows it down at least by half.

if we focus on ones using the pattern `test -z ... || rm -f ...`, there's
barely a dozen of those.  i'm not sure we need to fully audit the tree and
review every `rm`, just do a best effort and wait for feedback (i.e. bugs).

we could add a helper var to further collapse the boiler plate.
am__rm_f = rm -f $(am__rm_f_notfound)
then the rule is a bit more palatable:
$(am__rm_f) ...

or if we want to claim namespace and make it something we want to export
for users of automake akin to AC_PROG_LN_S, we could use RM_F.
-mike


signature.asc
Description: PGP signature

bug#10828: [RFC] POSIX will say running "rm -f" with no argument is OK

[Karl Berry]

> rm -f NOTFOUND $(VAR)
i think this is an interesting route.  

I agree.

we could do:
rm -f $(am__rm_f_notfound) ...
and am__rm_f_notfound could be set based on the configure test.

Sounds plausible to me. The "only" problem is that there are dozens or
hundreds of places in the code that do rm -f. They would all have to be
inspected and changed (or not) ... -k

bug#10828: [RFC] POSIX will say running "rm -f" with no argument is OK

[Mike Frysinger]

On 11 Dec 2021 23:19, Moritz Klammler wrote:
> An alternative trick which I have used in my code and found much less 
> disturbing is to prepend an arbitrary, hopefully non-existent, file name 
> so the list of arguments will never be empty even if the variable is.
> 
> rm -f NOTFOUND $(VAR)

i think this is an interesting route.  we could do:
rm -f $(am__rm_f_notfound) ...
and am__rm_f_notfound could be set based on the configure test.
am__rm_f_notfound = WORKAROUND_RM_F_BUG
that way most people won't see any noise.
-mike


signature.asc
Description: PGP signature

bug#10828: [RFC] POSIX will say running "rm -f" with no argument is OK

[Moritz Klammler]

Hi folks,

hope you don't mind an outsider (occasional Autotools user) joining the 
discussion with some trivia.


On 12/11/21 22:52, Karl Berry wrote:

Hi Mike, Peter,

  > or do we document that we expect `rm -f` to work,

I don't see a reason to break working code wrt rm -f merely for cosmetic
purposes.
   test -z "$(VAR)" || rm -f $(VAR)
does not actually seem "awful" to me. And I fail to see any significant
gain by simplifying it.


I do have to say that I found it visually disturbing and the noise makes 
it harder for me to understand what an Automake recipe does. Especially 
when not looking at the code but the expended commands as printed during 
execution.



Whatever POSIX says, we can be sure that reality differs.
Although systems that fail on "rm -f" are few and far between by now,
they are not nonexistent. So why break them?

My idea is to let the existing code stand (forever).

In the alternative, I agree with Peter that at least providing an
overridable variable would be desirable. --thanks, karl.



An alternative trick which I have used in my code and found much less 
disturbing is to prepend an arbitrary, hopefully non-existent, file name 
so the list of arguments will never be empty even if the variable is.


rm -f NOTFOUND $(VAR)

Note that the suggested

RM = rm -f
$(RM) $(VAR)

makes it very easy to inject this behavior; just set RM="rm -f NOTFOUND" 
and you're good to go. You don't have to be able to expand the argument 
list twice, which would be less trivial to inject.


I'm not aware of any situation where this trick doesn't work.


OpenPGP_signature
Description: OpenPGP digital signature

bug#10828: [RFC] POSIX will say running "rm -f" with no argument is OK

[Karl Berry]

it probably comes down to being a bit clearer on what autotools'
goals are and what it considers reasonable prerequisites.

I'd find it hard to define precisely. As a general rule, I wouldn't want
to lose support for any system that is working now, until we can be
really sure that it is no longer in use anywhere.

For me, compatibility is the dominant criterion for any change.

I made a foolish mistake recently wrt Python variables (a nice new
feature, but I failed to ensure that existing behavior did not change),
but hopefully it is back to normal in 1.16.5.

about whether it can use POSIX shell features that have been
standardized for over 30 years.  it has decided that it cannot for
some.

And thank goodness that misguided $(...) change finally got reverted.

clearly it's not "anything in the latest POSIX standard is fair
game, and anything not supporting the latest POSIX standard is SOL".

Clearly, and thank goodness for that too :).

Thanks,
Karl

bug#10828: [RFC] POSIX will say running "rm -f" with no argument is OK

[Karl Berry]

Hi Mike, Peter,

 > or do we document that we expect `rm -f` to work, 

I don't see a reason to break working code wrt rm -f merely for cosmetic
purposes.
  test -z "$(VAR)" || rm -f $(VAR)
does not actually seem "awful" to me. And I fail to see any significant
gain by simplifying it.

Whatever POSIX says, we can be sure that reality differs.
Although systems that fail on "rm -f" are few and far between by now,
they are not nonexistent. So why break them?

My idea is to let the existing code stand (forever).

In the alternative, I agree with Peter that at least providing an
overridable variable would be desirable. --thanks, karl.

bug#10828: [RFC] POSIX will say running "rm -f" with no argument is OK

[Peter Johansson]

Hi Mike,

On 11/12/21 15:14, Mike Frysinger wrote:

On 11 Dec 2021 09:33, Peter Johansson wrote:

On 10/12/21 15:47, Mike Frysinger wrote:

if it's dropped, i'm not sure how users are supposed to fix things.
the error message says to install GNU coreutils, but if GNU coreutils
uses automake and presents the same error ...

FWIW, automake is not needed to build and install GNU coreutils from a
released tarball.

the issue isn't about running `automake`, it's about the generated Makefile
using functionality that requires GNU coreutils.


Correct. I was sloppy in my analysis. Apologies for the noise.



  if an empty `rm -f` was
invoked as part of the coreutils build process, then the user would be a
bit stuck.

admittedly it's a bit unlikely to show up outside of `make clean`, but
coreutils is known to do some "extra" things (optional tools, generation
of man pages, etc...), and i haven't fully looked at every place where
automake will generate a `rm -f` pattern.  do we have to self-regulate
every time we try to use that in case it might break the bootstrap flow
of coreutils ?

i'm not saying we have to support this scenario.  i'm just pointing out
that it's not as cut & dry as we might hope it would be.

it probably comes down to being a bit clearer on what autotools' goals are
and what it considers reasonable prerequisites.  the GNU config project has
undergone a bit of lively debate in the last year about whether it can use
POSIX shell features that have been standardized for over 30 years.  it has
decided that it cannot for some.

automake clearly doesn't require GNU make.  the manual says:

https://www.gnu.org/software/automake/manual/automake.html#Why-Autotools
* As we have seen, the GNU Build System has a lot of features ...
* Implementing these features portably is difficult and exhausting. Think of
writing portable shell scripts, and portable Makefiles, for systems you may
not have handy. See Portable Shell Programming in The Autoconf Manual, to
convince yourself.
* The GNU Autotools take all this burden off your back and provide:
* Tools to create a portable, complete, and self-contained GNU Build System,
from simple instructions. Self-contained meaning the resulting build system
does not require the GNU Autotools.

so how is "portable" being defined ?  clearly it's not "anything in the latest
POSIX standard is fair game, and anything not supporting the latest POSIX
standard is SOL".  i think users of automake would be surprised by that when
automake has a history of producing Makefile's packages that can be reliably
built on older systems.

if POSIX were a hard requirement, automake wouldn't carry support logic for
Microsoft's lib archiver via `ar-lib`.

so circling back, can we see a way of walking the `rm -f` line without the
maintenance burden being too high ?  e.g. a configure test that checks if
`rm -f` exits non-zero, and some Makefile logic that is able to switch in
the test in/out as needed ?  we could set am__rm_f_suppress to - and just
ignore all failures.  it would allow packages to continue to build (if at
the cost of being extra chatty) for such deficient setups.
am__rm_f_suppress = @am__rm_f_suppress@
am__rm_f = $(am__rm_f_suppress)rm -f

or do we document that we expect `rm -f` to work, and if it doesn't on
your system, then hopefully you won't happen to run into the limitation
when building the project (even if `make clean` might break).  i.e. we
turn the configure-time error into a warning, and then just hope for the
best.  nowhere do we use the `test ... || rm -f ...` pattern.


That seems reasonable. I'd say the minimal change would be to at least 
not call 'rm -f' directly but define a variable


RM = rm -f

so a user has some way around potential problems.

Cheers,

Peter

bug#10828: [RFC] POSIX will say running "rm -f" with no argument is OK

[Mike Frysinger]

On 11 Dec 2021 09:33, Peter Johansson wrote:
> On 10/12/21 15:47, Mike Frysinger wrote:
> > if it's dropped, i'm not sure how users are supposed to fix things.
> > the error message says to install GNU coreutils, but if GNU coreutils
> > uses automake and presents the same error ...
> 
> FWIW, automake is not needed to build and install GNU coreutils from a 
> released tarball.

the issue isn't about running `automake`, it's about the generated Makefile
using functionality that requires GNU coreutils.  if an empty `rm -f` was
invoked as part of the coreutils build process, then the user would be a
bit stuck.

admittedly it's a bit unlikely to show up outside of `make clean`, but
coreutils is known to do some "extra" things (optional tools, generation
of man pages, etc...), and i haven't fully looked at every place where
automake will generate a `rm -f` pattern.  do we have to self-regulate
every time we try to use that in case it might break the bootstrap flow
of coreutils ?

i'm not saying we have to support this scenario.  i'm just pointing out
that it's not as cut & dry as we might hope it would be.

it probably comes down to being a bit clearer on what autotools' goals are
and what it considers reasonable prerequisites.  the GNU config project has
undergone a bit of lively debate in the last year about whether it can use
POSIX shell features that have been standardized for over 30 years.  it has
decided that it cannot for some.

automake clearly doesn't require GNU make.  the manual says:
> https://www.gnu.org/software/automake/manual/automake.html#Why-Autotools
> * As we have seen, the GNU Build System has a lot of features ...
> * Implementing these features portably is difficult and exhausting. Think of
> writing portable shell scripts, and portable Makefiles, for systems you may
> not have handy. See Portable Shell Programming in The Autoconf Manual, to
> convince yourself.
> * The GNU Autotools take all this burden off your back and provide:
> * Tools to create a portable, complete, and self-contained GNU Build System,
> from simple instructions. Self-contained meaning the resulting build system
> does not require the GNU Autotools.

so how is "portable" being defined ?  clearly it's not "anything in the latest
POSIX standard is fair game, and anything not supporting the latest POSIX
standard is SOL".  i think users of automake would be surprised by that when
automake has a history of producing Makefile's packages that can be reliably
built on older systems.

if POSIX were a hard requirement, automake wouldn't carry support logic for
Microsoft's lib archiver via `ar-lib`.

so circling back, can we see a way of walking the `rm -f` line without the
maintenance burden being too high ?  e.g. a configure test that checks if
`rm -f` exits non-zero, and some Makefile logic that is able to switch in
the test in/out as needed ?  we could set am__rm_f_suppress to - and just
ignore all failures.  it would allow packages to continue to build (if at
the cost of being extra chatty) for such deficient setups.
am__rm_f_suppress = @am__rm_f_suppress@
am__rm_f = $(am__rm_f_suppress)rm -f

or do we document that we expect `rm -f` to work, and if it doesn't on
your system, then hopefully you won't happen to run into the limitation
when building the project (even if `make clean` might break).  i.e. we
turn the configure-time error into a warning, and then just hope for the
best.  nowhere do we use the `test ... || rm -f ...` pattern.

based on the user reports thus far, i'm OK with the latter.
-mike


signature.asc
Description: PGP signature

bug#10828: [RFC] POSIX will say running "rm -f" with no argument is OK

[Peter Johansson]

Hi Mike,

On 10/12/21 15:47, Mike Frysinger wrote:

if it's dropped, i'm not sure how users are supposed to fix things.
the error message says to install GNU coreutils, but if GNU coreutils
uses automake and presents the same error ...


FWIW, automake is not needed to build and install GNU coreutils from a 
released tarball.


Cheers,

Peter

bug#10828: [RFC] POSIX will say running "rm -f" with no argument is OK

[Mike Frysinger]

On 03 Jan 2013 20:14, Stefano Lattarini wrote:
> Reference:
> 
> 
> [This is posted also to the automake list to ensure a wider audience.
>  Discussion should continue exclusively on the bug-automake list]
> 
> OK, time to resurrect this thread.
> 
> To summarize, POSIX will say in a future version that running "rm -f"
> with no argument is OK; and that usage seems today portable to all
> systems that are well on their way to obsolescence.
> 
> So, in Automake 1.14, we might start to simplify several automake
> generated "cleaning" rules accordingly, to get rid of the awful
> idiom:
> 
>   test -z "$(VAR)" || rm -f $(VAR)
> 
> In your opinion, can this be done without further fuss, or should we
> add a temporary "probe check" in AM_INIT_AUTOMAKE that verifies that
> the no-args "rm -f" usage is supported on the system configure is
> being run on?  (As for a testsuite probe, that is already present;
> see tests 't/spy-rm.tap', already present in Automake 1.13).
> 
> Comments welcome (and if your answer is "we should add a probe
> check", patches are even more welcome ;-)

what is the game plan here ?  are we intending to keep the backwards
compatible behavior ?  or drop it entirely ?  it seems like if we're
going to go through the effort of adding a configure test to see if
the `rm -f` behavior predates POSIX-2013, then doing an AC_SUBST to
get the backwards compat behavior isn't that much overhead ?

if it's dropped, i'm not sure how users are supposed to fix things.
the error message says to install GNU coreutils, but if GNU coreutils
uses automake and presents the same error ...

in the years since this merged, we've gotten three reports:
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=19692
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=22074
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=23563
-mike


signature.asc
Description: PGP signature