Re: [PATCH] up_to_date_p: treat equal mtime as outdated.

[Karl Berry]

So fine to remove from automake after all. Good!

So removed. Thanks to all. -k

Re: [PATCH] up_to_date_p: treat equal mtime as outdated.

[Jim Meyering]

On Sat, May 9, 2020 at 8:05 PM Harald van Dijk  wrote:
> On 10/05/2020 02:10, Karl Berry wrote:
> > +Although we would like to remove this function from Automake, since it's
> > +not used, that would break older versions of Autoconf, which seems
> > +gratuitious. So we leave it, unchanged.
>
> Autoconf has its own copy of this function, so why would it be affected
> by the removal of the function from Automake?

Good point. You're right:

> ac$ git grep -n sub.up_to_date_p
> lib/Autom4te/FileUtils.pm:190:sub up_to_date_p ($@)

in that removing it from automake should not impact any existing
installation of autoconf. However, autoconf's lib/Autom4te hierarchy
is sync'd from automake, so we would still need to make an adjustmake
to autoconf before its next sync.

So fine to remove from automake after all. Good!

Re: [PATCH] up_to_date_p: treat equal mtime as outdated.

[Harald van Dijk]

On 10/05/2020 02:10, Karl Berry wrote:

+Although we would like to remove this function from Automake, since it's
+not used, that would break older versions of Autoconf, which seems
+gratuitious. So we leave it, unchanged.


Autoconf has its own copy of this function, so why would it be affected 
by the removal of the function from Automake?


Cheers,
Harald van Dijk

Re: [PATCH] up_to_date_p: treat equal mtime as outdated.

[Jim Meyering]

On Sat, May 9, 2020 at 6:10 PM Karl Berry  wrote:
>
> Probably best to leave it, as is, and mark it as known-to-be-unused at
> least via comment.
>
> How does the text below look for an explanation?

Very good! Thanks for dealing with this.
Two suggestions below.

> (By the way, I noticed that FileUtils.pm, unlike the other *.pm in
> lib/Automake, doesn't have an =over 4 ... =back around all the other
> items, causing pod2text to complain. I'll fix that too unless there is
> some magical reason for it.)

Sounds fine to fix that. Thanks!

> --- a/lib/Automake/FileUtils.pm
> +++ b/lib/Automake/FileUtils.pm
> @@ -181,7 +181,20 @@ sub update_file ($$;$)
>
>  =item C
>
> -Is C<$file> more recent than C<@dep>?
> +Is C<$file> strictly more recent than C<@dep>?
> +If mtimes are equal, returns true.
> +
> +This function is not used anywhere in Automake.  Where it is used is in

Perhaps s/Where it is used is in/However, it @emph{is} used in/

also, s/subsecond/sub-second/ to pacify spell-checkers :-)

> +C, which is part of Autoconf.  And its use there poses a
> +problem with respect to subsecond timestamps, as discussed at
> +L
> +(thread continues into May).  The best approach seems to be to leave this
> +function alone, and have C use a different test, one which is
> +not part of Automake.
> +
> +Although we would like to remove this function from Automake, since it's
> +not used, that would break older versions of Autoconf, which seems
> +gratuitious. So we leave it, unchanged.

Re: [PATCH] up_to_date_p: treat equal mtime as outdated.

[Karl Berry]

Probably best to leave it, as is, and mark it as known-to-be-unused at
least via comment.

How does the text below look for an explanation?

(By the way, I noticed that FileUtils.pm, unlike the other *.pm in
lib/Automake, doesn't have an =over 4 ... =back around all the other
items, causing pod2text to complain. I'll fix that too unless there is
some magical reason for it.)

--- a/lib/Automake/FileUtils.pm
+++ b/lib/Automake/FileUtils.pm
@@ -181,7 +181,20 @@ sub update_file ($$;$)
 
 =item C
 
-Is C<$file> more recent than C<@dep>?
+Is C<$file> strictly more recent than C<@dep>?
+If mtimes are equal, returns true.
+
+This function is not used anywhere in Automake.  Where it is used is in
+C, which is part of Autoconf.  And its use there poses a
+problem with respect to subsecond timestamps, as discussed at
+L
+(thread continues into May).  The best approach seems to be to leave this
+function alone, and have C use a different test, one which is
+not part of Automake.
+
+Although we would like to remove this function from Automake, since it's
+not used, that would break older versions of Autoconf, which seems
+gratuitious. So we leave it, unchanged.
 
 =cut
 

compile finished at Sat May  9 18:09:15 2020

Re: [PATCH] up_to_date_p: treat equal mtime as outdated.

[Jim Meyering]

On Mon, Apr 20, 2020 at 6:13 PM Karl Berry  wrote:
> the command that updates autom4te.cache/traces.0 does not modify
> configure.ac at the same time.
>
> No argument for that specific case. But looking at the change in
> isolation:
>
> -  if ($mtime < mtime ($dep))
> +  if ($mtime <= mtime ($dep))
> {
>   verb "up_to_date ($file): outdated: $dep";
>
> it says that a dependency with the same mtime as the target will now be
> "outdated", instead of being up to date. This just seems contrary to the
> standard behavior in make (and wherever else), and what would generally
> be expected.
>
>   998 Autom4te::C4che->load ($icache_file)
>   999   if -f $icache && mtime ($icache) > mtime ($0);
>
> I agree it's inconsistent with the current up_to_date_p.
>
> I understand some change needs to be made for autom4te to work
> consistently with subsecond timestamps. I have no problem with that, but
> autom4te is part of autoconf, not automake, as you know.
>
> So ... the change that comes to my mind is to avoid using up_to_date_p
> in autom4te. There could be a new function, or since it is apparently
> only used at one place (line 923), just write out the mtime tests as
> needed, as you cited above. Seems like the mtime test could be part of
> the   while ($_ = $deps->getline)  loop just above the present call to
> up_to_date_p.

I agree that we should break this dependency: either way would work
for me: new function or inline.

> As for the up_to_date_p function, it is not used in automake at all. As
> far as I can see, it is only used in autom4te.  But perhaps it is used
> entirely outside automake/autoconf by other random (user?) programs.
> I doubt it, but can any of us be sure? And if it does have outside users,
> that would be more reason not to change it. Alternatively, if we're sure
> there are no outside users, my proposal would be to delete it, as unused.

I confess that I like the idea of removing it from automake, since no
search I ran found any other use of it, but that would render the next
automake unusable with any older autoconf that still requires it. We
could of course mandate a newer version of autoconf that doesn't need
it, but even that could probably go awry...

Probably best to leave it, as is, and mark it as known-to-be-unused at
least via comment.
Thanks for working on this.

Re: [PATCH] up_to_date_p: treat equal mtime as outdated.

[Karl Berry]

the command that updates autom4te.cache/traces.0 does not modify
configure.ac at the same time.

No argument for that specific case. But looking at the change in
isolation:

-  if ($mtime < mtime ($dep))
+  if ($mtime <= mtime ($dep))
{
  verb "up_to_date ($file): outdated: $dep";

it says that a dependency with the same mtime as the target will now be
"outdated", instead of being up to date. This just seems contrary to the
standard behavior in make (and wherever else), and what would generally
be expected.

  998 Autom4te::C4che->load ($icache_file)
  999   if -f $icache && mtime ($icache) > mtime ($0);

I agree it's inconsistent with the current up_to_date_p.

I understand some change needs to be made for autom4te to work
consistently with subsecond timestamps. I have no problem with that, but
autom4te is part of autoconf, not automake, as you know.

So ... the change that comes to my mind is to avoid using up_to_date_p
in autom4te. There could be a new function, or since it is apparently
only used at one place (line 923), just write out the mtime tests as
needed, as you cited above. Seems like the mtime test could be part of
the   while ($_ = $deps->getline)  loop just above the present call to
up_to_date_p.

As for the up_to_date_p function, it is not used in automake at all. As
far as I can see, it is only used in autom4te.  But perhaps it is used
entirely outside automake/autoconf by other random (user?) programs.
I doubt it, but can any of us be sure? And if it does have outside users,
that would be more reason not to change it. Alternatively, if we're sure
there are no outside users, my proposal would be to delete it, as unused.

Best,
Karl

Re: [PATCH] up_to_date_p: treat equal mtime as outdated.

[Nick Bowler]

On 2020-04-13, Paul Eggert  wrote:
> I just checked, and GNU Make uses high-resolution file timestamps when
> available, and considers a file to be up-to-date if it has exactly the same
> timestamp as its dependency. I suspect that this is because Makefile rules
> like
> this:
>
> a: b
>   cp -p b a
>
> would otherwise cause needless work if one ran 'make; make'.

If the Autoconf manual's portability notes are to be believed, then
such a rule is really not a good idea because on filesystems suporting
subsecond timestamp precision some "cp -p" implementations can create
the destination file with a strictly older timestamp than the source.

It's worth pointing out that even on filesystems supporting high-
precision timestamps, actual mtime updates usually have significantly
lower resolution (typically on the order of a millisecond or so).
Sufficiently fast make rules can frequently create many files with
exactly the same mtime.  Therefore considering equal timestamps to be
"up-to-date" is really the only reasonable option.

In cases where it is required to create two files with distinct mtimes,
the only real portable option I've found is to touch one of the files
in a loop and compare timestamps until they are different.

Cheers,
  Nick

Re: [PATCH] up_to_date_p: treat equal mtime as outdated.

[Dima Pasechnik]

On Tue, Apr 14, 2020 at 08:06:14AM +0100, Harald van Dijk via Bug reports for 
autoconf wrote:
> On 14/04/2020 04:34, Paul Eggert wrote:
> > On 4/13/20 4:21 PM, Harald van Dijk wrote:
> > > For better or worse, FAT is the most universally accepted file
> > > system, and for that reason it is widely used. It does not even
> > > support second precision timestamps.
> > 
> > Let's not worry much about that. In practice, little development of
> > Automake-using software occurs on FAT file systems, and even if it did
> > those file systems are low priority for GNU development.

Let me point out that, regarding timestumps, a lot of automake-using
software gets built on clusters, with not really well-maintained
toolchains, and on networked filesystems, which are usually not sufficiently
fast for correct clock synching. If AM_MAINTAINER_MODE is used with
default enable-maintainer-mode option, you usually get an obscure build
error from missing script. And if AM_MAINTAINER_MODE is not used,
following a rather unfortunate advise that it leads to (basically,
nowadays, absent) eventual problems, it's even worse, as you basically need to
patch the configure.ac on a system with working autotools toolchain,
ideally convince upstream that AM_MAINTAINER_MODE is not as bad so that
it gets properly fixed, etc etc.

(Quote: "Still many people continue to use AM_MAINTAINER_MODE, because it helps
them working on projects where all files are kept under version control,
and because missing isn’t enough if you have the wrong version of the
tools." - Do we need to discuss how much past century this is?)

While this is tangential to this thread, an update in the direction
rectifying this isssue would be very welcome.

Best,
Dmitrii
> 
> It's by no means the only file system without sub-second timestamps
> though.  I went with FAT because it is so widely used, but if you want
> a more UNIXy example, there's ext2/ext3, isn't there? Sub-second
> precision is one of the new features of ext4.
> 
> > I just checked, and GNU Make uses high-resolution file timestamps
> > when available, and considers a file to be up-to-date if it has
> > exactly the same timestamp as its dependency. I suspect that this is
> > because Makefile rules like this:
> > 
> > a: b cp -p b a
> > 
> > would otherwise cause needless work if one ran 'make; make'. > > If
> > Automake followed the same rule as GNU Make, we'd at least have the
> > benefit of consistency
> 
> I suspect it's not (or not just) because of that, but because of rules
> that depend on generated files making it extremely likely that in
> practice you will see rules that are executed within a second of deps
> being modified.  This is a tradeoff between correctness and avoiding
> unnecessary updates where autoconf/automake are not in the same
> situation as make. There are three things to consider:
> 
> 1. Given a dep and target with equal mtime, how likely is it that the
> target is up to date?
> 
> 2. Given a dep and target with equal mtime, if the target is up to
> date, what are the consequences of treating it as out of date?
> 
> 3. Given a dep and target with equal mtime, if the target is out of
> date, what are the consequences of treating it as up to date?
> 
> My assumptions would be that 1 is more likely for make than it is for
> autoconf/automake (given that autoconf/automake are generally not
> using generated files as deps), that 2 is more harmful for make than
> it is for autoconf/automake (given command sequences such as make
> VAR=value && make install, where a spurious recompile during make
> install is not harmless but causes the effect of VAR=value to be
> lost), and that 3 is equally bad for make as it is for
> autoconf/automake.
> 
> Cheers, Harald van Dijk
> 

Re: [PATCH] up_to_date_p: treat equal mtime as outdated.

[Harald van Dijk via Patches for Automake]

On 14/04/2020 04:34, Paul Eggert wrote:

On 4/13/20 4:21 PM, Harald van Dijk wrote:
For better or worse, FAT is the most universally accepted file system, 
and for that reason it is widely used. It does not even support second 
precision timestamps.


Let's not worry much about that. In practice, little development of 
Automake-using software occurs on FAT file systems, and even if it did 
those file systems are low priority for GNU development.


It's by no means the only file system without sub-second timestamps 
though. I went with FAT because it is so widely used, but if you want a 
more UNIXy example, there's ext2/ext3, isn't there? Sub-second precision 
is one of the new features of ext4.


I just checked, and GNU Make uses high-resolution file timestamps when 
available, and considers a file to be up-to-date if it has exactly the 
same timestamp as its dependency. I suspect that this is because 
Makefile rules like this:


a: b
 cp -p b a

would otherwise cause needless work if one ran 'make; make'. > > If Automake 
followed the same rule as GNU Make, we'd at least have the
benefit of consistency


I suspect it's not (or not just) because of that, but because of rules 
that depend on generated files making it extremely likely that in 
practice you will see rules that are executed within a second of deps 
being modified. This is a tradeoff between correctness and avoiding 
unnecessary updates where autoconf/automake are not in the same 
situation as make. There are three things to consider:


1. Given a dep and target with equal mtime, how likely is it that the 
target is up to date?


2. Given a dep and target with equal mtime, if the target is up to date, 
what are the consequences of treating it as out of date?


3. Given a dep and target with equal mtime, if the target is out of 
date, what are the consequences of treating it as up to date?


My assumptions would be that 1 is more likely for make than it is for 
autoconf/automake (given that autoconf/automake are generally not using 
generated files as deps), that 2 is more harmful for make than it is for 
autoconf/automake (given command sequences such as make VAR=value && 
make install, where a spurious recompile during make install is not 
harmless but causes the effect of VAR=value to be lost), and that 3 is 
equally bad for make as it is for autoconf/automake.


Cheers,
Harald van Dijk

Re: [PATCH] up_to_date_p: treat equal mtime as outdated.

[Paul Eggert]

On 4/13/20 4:21 PM, Harald van Dijk wrote:
For better or worse, FAT is the most universally accepted file system, and for 
that reason it is widely used. It does not even support second precision 
timestamps.


Let's not worry much about that. In practice, little development of 
Automake-using software occurs on FAT file systems, and even if it did those 
file systems are low priority for GNU development.


I just checked, and GNU Make uses high-resolution file timestamps when 
available, and considers a file to be up-to-date if it has exactly the same 
timestamp as its dependency. I suspect that this is because Makefile rules like 
this:


a: b
cp -p b a

would otherwise cause needless work if one ran 'make; make'.

If Automake followed the same rule as GNU Make, we'd at least have the benefit 
of consistency

Re: [PATCH] up_to_date_p: treat equal mtime as outdated.

[Harald van Dijk]

On 13/04/2020 23:58, Karl Berry wrote:

Hi Harald and all - well, Paul or Jim or any other automake worker will
know better than me, but my reactions, FWIW:

 This caused intermittent failures in automake's t/subobj.sh test, which

This problem seems like the tip of the iceberg to me. With this change,
any dependency which gets updated in the same second as the target will
now have to get remade. Possibly looping forever, it seems to me?


Hi Karl,

A dependency is a file that needs to already exist beforehand. It is not 
going to get updated by the command that creates the target. For 
instance, for the dep configure.ac, one of the output files is 
autom4te.cache/traces.0. If configure.ac is modified, 
autom4te.cache/traces.0 needs to be updated, but the command that 
updates autom4te.cache/traces.0 does not modify configure.ac at the same 
time. My patch may cause autom4te.cache/traces.0 to be updated multiple 
times in the worst case, but that's harmless and only lasts a second at 
most.



I have not seen a deluge of (or any) reports about timestamp precision
being a problem before this.


It's probably pretty rare. I only saw it when running the automake 
testsuite on an otherwise idle system, and even then not reliably. I 
cannot speak for others, but that's not how I usually run the testsuite: 
normally the system is busy with other stuff as well, slowing it down 
enough for the automake test to pass.



 runs aclocal+automake, modifies configure.ac, and immediately re-runs
 aclocal+automake. On sufficiently fast computers, this would not update
 the cache files.

Maybe aclocal and/or automake and/or other autoprograms should just
sleep 1 before exiting (or at whatever crucial junctures) if this
problem must be avoided at all costs.

But better -- as Paul, I guess, said, how about if up_to_date_p uses
subsecond timestamps if available, so the < test remains unchanged?
After all, if subsecond timestamps are not available, it's highly
probable that it's an old machine and it's all a non-issue. --best, karl.


Leaving the check as < is wrong, but it is also inconsistent with what 
autom4te does outside of this module. From bin/autom4te.in:


 995 # Read the cache index if available and older than autom4te itself.
 996 # If autom4te is younger, then some structures such as C4che might
 997 # have changed, which would corrupt its processing.
 998 Autom4te::C4che->load ($icache_file)
 999   if -f $icache && mtime ($icache) > mtime ($0);

Note how this only reads the cache file if it's newer, not if it has the 
same mtime.


And I would not be too confident that nobody is still using file systems 
without sub-second precision on modern machines. For better or worse, 
FAT is the most universally accepted file system, and for that reason it 
is widely used. It does not even support second precision timestamps.


Cheers,
Harald van Dijk

Re: [PATCH] up_to_date_p: treat equal mtime as outdated.

[Karl Berry]

Hi Harald and all - well, Paul or Jim or any other automake worker will
know better than me, but my reactions, FWIW:

This caused intermittent failures in automake's t/subobj.sh test, which

This problem seems like the tip of the iceberg to me. With this change,
any dependency which gets updated in the same second as the target will
now have to get remade. Possibly looping forever, it seems to me?

I have not seen a deluge of (or any) reports about timestamp precision
being a problem before this.

runs aclocal+automake, modifies configure.ac, and immediately re-runs
aclocal+automake. On sufficiently fast computers, this would not update
the cache files.

Maybe aclocal and/or automake and/or other autoprograms should just
sleep 1 before exiting (or at whatever crucial junctures) if this
problem must be avoided at all costs.

But better -- as Paul, I guess, said, how about if up_to_date_p uses
subsecond timestamps if available, so the < test remains unchanged?
After all, if subsecond timestamps are not available, it's highly
probable that it's an old machine and it's all a non-issue. --best, karl.

Re: [PATCH] up_to_date_p: treat equal mtime as outdated.

[Harald van Dijk]

On 11/04/2020 21:07, Paul Eggert wrote:

On 4/11/20 12:28 PM, Harald van Dijk wrote:

The up_to_date_p function uses mtime, which returns timestamps with
second precision. Even if it were capable of returning higher precision
timestamps, the file system might not support it.


These days file systems typically do support it, so can't we use 
higher-resolution file timestamps when they're available?


Yes, that is possible. It doesn't avoid the need for this patch and 
makes it harder to test this patch though, so I would suggest doing that 
separately after this one if it's worthwhile. It can be done by making 
mtime return fractional results and making sure callers handle that, 
which is mainly just a matter of adding use Time::HiRes qw(stat); to 
avoid using perl's built-in stat function which does not support 
sub-second precision, and use Time::HiRes's version instead which does.


I notice now though in the header of this file:


###
# The main copy of this file is in Automake's git repository. #
# Updates should be sent to automake-patches@gnu.org. #
###


Sorry for missing this the first time. Adjusted to modify automake's 
copy, and that address added to the CC.


---

The up_to_date_p function uses mtime, which returns timestamps with
second precision. Even if it were capable of returning higher precision
timestamps, the file system might not support it.

With second precision timestamps, when two timestamps are equal, we do
not know which is newer and need to account for the worst case scenario.

This caused intermittent failures in automake's t/subobj.sh test, which
runs aclocal+automake, modifies configure.ac, and immediately re-runs
aclocal+automake. On sufficiently fast computers, this would not update
the cache files.
---
 lib/Automake/FileUtils.pm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/Automake/FileUtils.pm b/lib/Automake/FileUtils.pm
index 7ebf547e0..2c8cfc52f 100644
--- a/lib/Automake/FileUtils.pm
+++ b/lib/Automake/FileUtils.pm
@@ -195,7 +195,7 @@ sub up_to_date_p ($@)

   foreach my $dep (@dep)
 {
-  if ($mtime < mtime ($dep))
+  if ($mtime <= mtime ($dep))
{
  verb "up_to_date ($file): outdated: $dep";
  return 0;
--
2.25.2