Re: [dev] requirements of build systems
On Tue, Feb 01, 2022 at 09:29:55PM +0900, Pontus Stenetorp wrote: > Speaking for myself, I certainly have experienced issues with > inaccurate timestamps on NFS for compute clusters where its use is > very common. Not saying this as a supporter of NFS and the likes, just > as evidence that it does occur in practice. That's good to know, I wasn't thinking about NFS at all. Also just to be clear, my question wasn't rhetorical. Since I've never experienced issue with make, I was genuinely curious if this issue occurs in practice or not. - NRK
Re: [dev] requirements of build systems
*** Pedro Lucas Porcellis [2022-02-01 11:00]: >I think for most people out there, relying on mtime is just perfectly fine. No. mtime depends on time and filesystem implementation specifics. There are many systems where sysadmins like to do cron-ed ntpdate, which leads to jumping clocks. There could be filesystem/kernel issues like low mtime value granularity: https://stackoverflow.com/questions/14392975/timestamp-accuracy-on-ext4-sub-millsecond There are many other possible unexpected problems: https://apenwarr.ca/log/20181113 >I believe that generally, this kind of "futorology development thinking" >can lead to developing very messy and complex solutions, because "we >might hit that edgecase in the future that only 0.5% of users have >then". People like me want reliability, predictability and some expected behaviour from the software, to trust and rely on it. mtime usage safety depends on quantity of pitfalls, implementation specifics and OS configuration options. We can "fix" those problems to make usable systems friendly with software relying solely on mtime. Or we can just can make less expectation of underlying FS/OS and write software (redo in current case) that does not depend on all of that fragile metainformation. Instead of depending on some OS specific mtime/ctime/whatever inode field, that *should* be updated if file's content is changed (but we already know that is not expected from mmap-ed changes and FUSE-based filesystems), we can depend literally on content itself (on its cryptographic hash not to waste space). Reliable, predictable, OS/FS-independent solution. Personally I am completely ok to have slightly added delay because of data transfer and hash computation, because if we talk about practical development tasks, there there are no huge amounts of data involved. Possibly slightly worse performance, but perfect reliability and predictability I can rely on. -- Sergey Matveev (http://www.stargrave.org/) OpenPGP: CF60 E89A 5923 1E76 E263 6422 AE1A 8109 E498 57EF
Re: [dev] requirements of build systems
On Tue, Feb 01, 2022 at 09:29:55PM +0900, Pontus Stenetorp wrote: > On Tue 01 Feb 2022, NRK wrote: > > On Mon, Jan 31, 2022 at 12:10:27AM +0200, Petros Pateros wrote: > > > However, it was pointed out to me that relying on mtime can give > > > wrong results, for example: (a) if the clock is set backwards or > > > in case of insufficient granularity in mtime then a file that gets > > > modified might have the same mtime (b) an mmap(2)-ed file can get > > > modified but its mtime might not get updated soon enough > > > > How likely is it for these situations to occur in practice? If these > > are practical problems, then it makes sense to solve them. Otherwise > > I think it's best not to waste resource solving theoretical > > problems. > > Speaking for myself, I certainly have experienced issues with > inaccurate timestamps on NFS for compute clusters where its use is > very common. Not saying this as a supporter of NFS and the likes, just > as evidence that it does occur in practice. Yeah, but again, how common is that scenario on a regular, day-to-day development? I think for most people out there, relying on mtime is just perfectly fine. I believe that generally, this kind of "futorology development thinking" can lead to developing very messy and complex solutions, because "we might hit that edgecase in the future that only 0.5% of users have then". In the end it's like NRK said, in the end it's just a waste of resources. With best regards, Pedro Lucas Porcellis
Re: [dev] requirements of build systems
On Tue 01 Feb 2022, NRK wrote: > On Mon, Jan 31, 2022 at 12:10:27AM +0200, Petros Pateros wrote: > > However, it was pointed out to me that relying on mtime can give wrong > > results, > > for example: > > (a) if the clock is set backwards or in case of insufficient granularity in > > mtime > > then a file that gets modified might have the same mtime > > (b) an mmap(2)-ed file can get modified but its mtime might not get updated > > soon enough > > How likely is it for these situations to occur in practice? If these are > practical problems, then it makes sense to solve them. Otherwise I think > it's best not to waste resource solving theoretical problems. Speaking for myself, I certainly have experienced issues with inaccurate timestamps on NFS for compute clusters where its use is very common. Not saying this as a supporter of NFS and the likes, just as evidence that it does occur in practice.
Re: [dev] requirements of build systems
On Mon, Jan 31, 2022 at 12:10:27AM +0200, Petros Pateros wrote: > However, it was pointed out to me that relying on mtime can give wrong > results, > for example: > (a) if the clock is set backwards or in case of insufficient granularity in > mtime > then a file that gets modified might have the same mtime > (b) an mmap(2)-ed file can get modified but its mtime might not get updated > soon enough How likely is it for these situations to occur in practice? If these are practical problems, then it makes sense to solve them. Otherwise I think it's best not to waste resource solving theoretical problems. - NRK
Re: [dev] requirements of build systems
On Mon, Jan 31, 2022 at 09:34:52AM -0500, LM wrote:
Dear Laura,
> This looks like a very interesting project. I've been looking for some
> alternatives to GNU make.
I hope you find something to fit your needs.
> I do use a few GNU make features that I'm not sure about porting to other
> alternatives. I use wildcards a lot for specifying files. I use the
> substitute features. I also use static pattern rules a lot:
> https://www.gnu.org/software/make/manual/make.html#Static-Pattern Was
> wondering if redo has functionality to replace those features. How hard
> would it be to target .po files and generate .mo files with the proper
> naming conventions using redo? So, en.po generates
> en/LC_MESSAGES/programname.mo. I currently use GNU make and wildcards to
> figure out what .po files there are and generate the .mo files placed in
> applicably named directories.
By default, /bin/sh is called to interpret the .do files, like
/bin/sh -e /path/to/script.do args.., unless they are executables, in which
case they are just executed. So expanding a wildcard is trivial.
all.do: (contents of all.do)
redo listofmo # always (re)generate it
redo-ifchange $(cat listofmo) # IFS is relevant here
listofmo.do:
# create/update the list of .mo's
# stdout is captured and will atomically (re)place listofmo,
# unless an error occurs
ls *.po |
while read -r po
do
printf '%s\n' "${po%.po}/LC_MESSAGES/programname.mo"
done
default.mo.do:
# $1: the name of the file this .do file is called to produce,
as a path relative to $0 ($0 is /path/to/default.mo.do)
# $2: $1 but without the .mo suffix
# $3: see below
po=${1%/*/*}.po
redo-ifchange "$po" # declare that $1 depends on $po
# now generate .mo from .po, either by creating $3 or by writing
# to stdout. Doing both is or affecting $1 directly is an error.
#po2mo "$po" > "$3"
#po2mo "$po" -o "$3"
#po2mo "$po" # if po2mo writes to stdout
With the above, running
$ redo all
will generate listofmo and the */LC_MESSAGES/programname.mo files
from the *.po files.
With this approach, however, listofmo is generated every time so that they are
up to date (in case of adding/removing a .po file).
Some redo implementations, like goredo [0], also store a hash of the created
file so nothing that depends on them gets rebuild when its content hasn't
changed.
> Any pointers to documentation on how to use
> redo in a project in place of GNU make?
You can find a lot in:
https://redo.readthedocs.io/en/latest/
http://www.goredo.cypherpunks.ru/Usage-rules.html
I also plan to write manual pages for baredo.
Cheers!
[0] http://www.goredo.cypherpunks.ru/index.html
