Re: POSIX ruling on up-to-date vs. identical timestamps
On Sat, 2014-08-23 at 18:33 -0700, Paul Eggert wrote: Paul Smith wrote: It needs to be considered carefully. How about having GNU 'make' do what GNU 'cp -u' does? The idea is to infer filesystem timestamp resolution by looking at every file timestamp that crosses your desk. When you see a file timestamp whose tv_nsec is nonzero modulo 100, for example, you know that its filesystem's resolution is finer-grained than 1 millisecond. When computation starts, you are conservative and assume that filesystems have 1-second resolution, but as computation goes on you gain more information about each filesystem and can become less and less conservative. It concerns me that this could cause the build to be non-deterministic in certain pathological situations, but it could be managed I'm sure. The immediate issue is that today, make doesn't track filesystems. It has no idea which files live on which filesystems, so we can't really keep track of the resolution on a per-filesystem basis. Of course the ability to track filesystems could be added without too much effort. It's trivial to determine the filesystem in POSIX via the device ID available from stat(), of course, but I'm not sure what facilities are available on the other ports. Although, I don't think either Windows or VMS support high-resolution timestamps right now (I don't know why Windows doesn't, because NTFS does support millisecond resolution timestamps I believe) so perhaps this is irrelevant. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: POSIX ruling on up-to-date vs. identical timestamps
From: Paul Smith psm...@gnu.org Date: Tue, 26 Aug 2014 10:52:32 -0400 Cc: Autoconf autoc...@gnu.org, Eric Blake ebl...@redhat.com, bug-make bug-make@gnu.org Of course the ability to track filesystems could be added without too much effort. It's trivial to determine the filesystem in POSIX via the device ID available from stat(), of course, but I'm not sure what facilities are available on the other ports. It's trivial on Windows as well. Although, I don't think either Windows or VMS support high-resolution timestamps right now Correct. (I don't know why Windows doesn't, because NTFS does support millisecond resolution timestamps I believe) Because no one wrote the code, of course. The main problem is that this requires to write a replacement 'stat' (not rocket science). ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: POSIX ruling on up-to-date vs. identical timestamps
On Tue, 2014-08-26 at 18:04 +0300, Eli Zaretskii wrote: (I don't know why Windows doesn't, because NTFS does support millisecond resolution timestamps I believe) Because no one wrote the code, of course. Ah, the oldest reason in free software :-). The main problem is that this requires to write a replacement 'stat' (not rocket science). Can't we just #define stat(_p,_b) _stat(_p,_b)? Not sure if that's sufficient: I'm not overly familiar with the limitations on the POSIX emulation functions in Windows. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: POSIX ruling on up-to-date vs. identical timestamps
Paul Smith wrote: It's trivial to determine the filesystem in POSIX via the device ID available from stat() Yes, that's what the Gnulib utimecmp module does: the idea is that after every stat-like operation you look at the file's time stamps to infer more information about the containing file system's resolution. As far as Windows goes, NTFS file systems have 100 ns resolution, and FAT file systems are the joker as they have a 2-second resolution for last-modified time. While we're on the topic, perhaps I should warn you that in old buggy Linux kernels file time stamps could spontaneously be truncated to file system resolution. According to https://gcc.gnu.org/ml/gcc/2004-04/msg00152.html this bug was introduced into the Linux kernel in 2002; I don't recall when it was fixed in Linux, but a few years later POSIX prohibited the buggy behavior. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: POSIX ruling on up-to-date vs. identical timestamps
From: Paul Smith psm...@gnu.org Cc: egg...@cs.ucla.edu, autoc...@gnu.org, ebl...@redhat.com, bug-make@gnu.org Date: Tue, 26 Aug 2014 11:18:35 -0400 The main problem is that this requires to write a replacement 'stat' (not rocket science). Can't we just #define stat(_p,_b) _stat(_p,_b)? Not sure if that's sufficient: I'm not overly familiar with the limitations on the POSIX emulation functions in Windows. _stat *is* the problem: the time-related members of the structure it returns report time only at 1-sec resolution. We need a replacement that supports sub-second resolution in additional members, like on some Posix platforms. Not a big deal, but Someone(TM) needs to sit down and write the code. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: POSIX ruling on up-to-date vs. identical timestamps
Date: Tue, 26 Aug 2014 08:25:38 -0700 From: Paul Eggert egg...@cs.ucla.edu Cc: Autoconf autoc...@gnu.org, Eric Blake ebl...@redhat.com, bug-make bug-make@gnu.org As far as Windows goes, NTFS file systems have 100 ns resolution, and FAT file systems are the joker as they have a 2-second resolution for last-modified time. That's true, but FAT filesystems are hardly important these days. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: POSIX ruling on up-to-date vs. identical timestamps
Date: Tue, 26 Aug 2014 18:30:12 +0100 From: Keith Marshall keithmarsh...@users.sourceforge.net CC: autoc...@gnu.org, ebl...@redhat.com, bug-make@gnu.org FAT filesystems are hardly important these days. Except insofar as they tend to be prevalent on removable media devices, such as USB flash drives; woe betide anyone who happens to store their working files on one of those. Indeed, let alone builds software on such volumes. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: POSIX ruling on up-to-date vs. identical timestamps
On 26/08/14 16:18, Paul Smith wrote: Can't we just #define stat(_p,_b) _stat(_p,_b)? Not sure if that's sufficient: I'm not overly familiar with the limitations on the POSIX emulation functions in Windows. That's effectively what MinGW does anyway, (although it does it through an import library mapping from the undecorated POSIX name, to the decorated name exported by MSVCRT.DLL). The upshot is that, when you call stat() in a Windows program compiled by MinGW's GCC, you actually call the function provided by Microsoft themselves, with whatever limitations that imposes. -- Regards, Keith. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: POSIX ruling on up-to-date vs. identical timestamps
On 26/08/14 18:22, Eli Zaretskii wrote: Date: Tue, 26 Aug 2014 08:25:38 -0700 From: Paul Eggert egg...@cs.ucla.edu Cc: Autoconf autoc...@gnu.org, Eric Blake ebl...@redhat.com, bug-make bug-make@gnu.org As far as Windows goes, NTFS file systems have 100 ns resolution, and FAT file systems are the joker as they have a 2-second resolution for last-modified time. That's true, but FAT filesystems are hardly important these days. Except insofar as they tend to be prevalent on removable media devices, such as USB flash drives; woe betide anyone who happens to store their working files on one of those. -- Regards, Keith. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: POSIX ruling on up-to-date vs. identical timestamps
On Thu, 2014-08-21 at 13:57 -0700, Paul Eggert wrote: David Boyce wrote: The obvious compromise would be to change the behavior only in the presence of the .POSIX: special target. We should limit .POSIX to what POSIX requires. Even if the ruling stands POSIX won't require the HP-UX behavior, so .POSIX should be independent of this issue. I pretty much agree with everything Paul says in this thread. First, I can't remember getting bug reports on GNU make that the current way it handles identical timestamps is a problem. I'm not saying that such a thing has never happened (my memory is not what it was for one thing) but certainly there have not been enough complaints to make this a known issue. And since this is just the kind of seemingly small change that will end up annoying and frustrating many people, I'm not excited about it. Similarly, unless POSIX mandates this change in behavior I'm not that excited about having the .POSIX target imply this change either: that seems to be mixing up too many obscure differences in a single flag. It'd be OK to introduce a new special target to enable the HP-UX behavior. .EQUAL_TIMES_ARE_OUT-OF-DATE, say. We could document the new target next to .LOW_RESOLUTION_TIME, since they're related issues. The new target could act like .LOW_RESOLUTION_TIME, except that it imposes HP-UX rather than low-res behavior. I think something like this may be the way to go, but it might need to be more comprehensive than this even. Consider the Savannah bug https://savannah.gnu.org/bugs/index.php?40056 which complains about builds where some of the files live on filesystems that do support hires timestamps and some files do not. Despite the interesting review of the 10th grade concept of significant digits (*rolleyes*) I don't particularly care for the solution provided there: assuming that a subsecond value of 0 always means there is no subsecond resolution seems to me to be problematic. However, it needs to be considered carefully. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: POSIX ruling on up-to-date vs. identical timestamps
Paul Smith wrote: It needs to be considered carefully. How about having GNU 'make' do what GNU 'cp -u' does? The idea is to infer filesystem timestamp resolution by looking at every file timestamp that crosses your desk. When you see a file timestamp whose tv_nsec is nonzero modulo 100, for example, you know that its filesystem's resolution is finer-grained than 1 millisecond. When computation starts, you are conservative and assume that filesystems have 1-second resolution, but as computation goes on you gain more information about each filesystem and can become less and less conservative. It's a hack, but the heuristic doesn't require any system calls (and for what it's worth we don't get bug reports about it...). This idea is implemented by Gnulib's 'utimecmp' module, which GNU Make is of course welcome to steal. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: POSIX ruling on up-to-date vs. identical timestamps
On Thu, Aug 21, 2014 at 4:32 PM, Eric Blake ebl...@redhat.com wrote: Make folks: You may want to check out http://austingroupbugs.net/view.php?id=857 and add comments and/or change GNU make behavior accordingly. There, the argument is made that HP-UX make behavior is nicer than GNU's current behavior when two files have identical timestamps: HP-UX considers the file as out-of-date, while GNU make considers it up-to-date. A strict reading of POSIX can argue that GNU's behavior was required, but this reading has been called into question. GNU's behavior is an optimization that avoids needless churn on file systems with course timestamps (well, FAT still exists, but these days, MOST file systems have sub-second resolution, so it is harder to get files with identical timestamps without actually touch'ing them that way) - but it risks leaving a tree in an incomplete state. HP-UX behavior guarantees the rules are run, even if they were not strictly necessary, but has the nice property that the tree is never left in an incomplete state due to unfortunate timing on a file system with course timestamps. The POSIX recommendation was therefore that GNU should change its behavior to act like HP-UX, and consider identical timestamps as out-of-date, because the standard will be fixed to allow HP-UX behavior. Autoconf folks: The section of the autoconf manual that discusses this should probably be modernized, particularly if changes to POSIX and/or GNU make result from this discussion. https://www.gnu.org/software/autoconf/manual/autoconf.html#Timestamps-and-Make Some projects that want to have deterministic builds by using the gitian builder [1] use libfaketime to set the timestamps to a known constant, and gnu make trips up on this, occasionally causing broken builds and othertimes autoconf declares suck a build system insane. There was a bug in libfaketime so that the nanosecond field wasn't cleared and this allowed you to avoid both these bugs if the planets were aligned correctly (which has since been fixed). I agree that these issues should be fixed in make and autoconf. [1] https://gitian.org/ -- Eric Blake eblake redhat com+1-919-301-3266 Libvirt virtualization library http://libvirt.org ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: POSIX ruling on up-to-date vs. identical timestamps
On Thu, Aug 21, 2014 at 4:57 PM, Ray Donnelly mingw.andr...@gmail.com wrote: On Thu, Aug 21, 2014 at 4:32 PM, Eric Blake ebl...@redhat.com wrote: Make folks: You may want to check out http://austingroupbugs.net/view.php?id=857 and add comments and/or change GNU make behavior accordingly. There, the argument is made that HP-UX make behavior is nicer than GNU's current behavior when two files have identical timestamps: HP-UX considers the file as out-of-date, while GNU make considers it up-to-date. A strict reading of POSIX can argue that GNU's behavior was required, but this reading has been called into question. GNU's behavior is an optimization that avoids needless churn on file systems with course timestamps (well, FAT still exists, but these days, MOST file systems have sub-second resolution, so it is harder to get files with identical timestamps without actually touch'ing them that way) - but it risks leaving a tree in an incomplete state. HP-UX behavior guarantees the rules are run, even if they were not strictly necessary, but has the nice property that the tree is never left in an incomplete state due to unfortunate timing on a file system with course timestamps. The POSIX recommendation was therefore that GNU should change its behavior to act like HP-UX, and consider identical timestamps as out-of-date, because the standard will be fixed to allow HP-UX behavior. Autoconf folks: The section of the autoconf manual that discusses this should probably be modernized, particularly if changes to POSIX and/or GNU make result from this discussion. https://www.gnu.org/software/autoconf/manual/autoconf.html#Timestamps-and-Make Some projects that want to have deterministic builds by using the gitian builder [1] use libfaketime to set the timestamps to a known constant, and gnu make trips up on this, occasionally causing broken builds and othertimes autoconf declares suck a build system insane. Freudian slip? I meant such, not suck. There was a bug in libfaketime so that the nanosecond field wasn't cleared and this allowed you to avoid both these bugs if the planets were aligned correctly (which has since been fixed). I agree that these issues should be fixed in make and autoconf. [1] https://gitian.org/ -- Eric Blake eblake redhat com+1-919-301-3266 Libvirt virtualization library http://libvirt.org ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: POSIX ruling on up-to-date vs. identical timestamps
Eric Blake wrote: You may want to check out http://austingroupbugs.net/view.php?id=857 and add comments and/or change GNU make behavior accordingly. Let's leave GNU 'make' alone. Its behavior is better for rules like this: copy: original cp -p original copy I've added a comment to the Austin Group page. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: POSIX ruling on up-to-date vs. identical timestamps
On Thu, 21 Aug 2014, Eric Blake wrote: The POSIX recommendation was therefore that GNU should change its behavior to act like HP-UX, and consider identical timestamps as out-of-date, because the standard will be fixed to allow HP-UX behavior. A change like this may result in some builds which never complete (or take much longer) because the timestamp rules require that something be rebuilt (tail chasing). This is particularly a problem for recursive builds which may enter the same directories multiple times. Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/ ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: POSIX ruling on up-to-date vs. identical timestamps
Ray Donnelly wrote: There was a bug in libfaketime so that the nanosecond field wasn't cleared That sounds like it's a different issue. If a program botches the nanosecond component of timestamps, it shouldn't matter whether 'make' uses the traditional/GNU or the HP-UX approach; either way, you can get into trouble. If I'm misunderstanding the libfaketime issue, could you please explain it with an example? ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: POSIX ruling on up-to-date vs. identical timestamps
The obvious compromise would be to change the behavior only in the presence of the .POSIX: special target. On Thu, Aug 21, 2014 at 9:13 AM, Paul Eggert egg...@cs.ucla.edu wrote: Eric Blake wrote: You may want to check out http://austingroupbugs.net/view.php?id=857 and add comments and/or change GNU make behavior accordingly. Let's leave GNU 'make' alone. Its behavior is better for rules like this: copy: original cp -p original copy I've added a comment to the Austin Group page. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: POSIX ruling on up-to-date vs. identical timestamps
On Thu, Aug 21, 2014 at 12:27 PM, Ray Donnelly mingw.andr...@gmail.com wrote: On Thu, Aug 21, 2014 at 8:03 PM, David Boyce david.s.bo...@gmail.com wrote: The obvious compromise would be to change the behavior only in the presence of the .POSIX: special target. Sounds pragmatic; the repeatable builds people would probably like a solution that doesn't require changing Makefiles though, either an env. var or a command line option? The command line is always possible for a special target given Paul's clever --eval option: make --eval .POSIX: ... And for an EV approach it should be possible to add that to an exported MAKEFLAGS. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: POSIX ruling on up-to-date vs. identical timestamps
David Boyce wrote: The obvious compromise would be to change the behavior only in the presence of the .POSIX: special target. We should limit .POSIX to what POSIX requires. Even if the ruling stands POSIX won't require the HP-UX behavior, so .POSIX should be independent of this issue. It'd be OK to introduce a new special target to enable the HP-UX behavior. .EQUAL_TIMES_ARE_OUT-OF-DATE, say. We could document the new target next to .LOW_RESOLUTION_TIME, since they're related issues. The new target could act like .LOW_RESOLUTION_TIME, except that it imposes HP-UX rather than low-res behavior. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
