Re: [dev] requirements of build systems
On Tue, Feb 01, 2022 at 08:09:12AM -0500, LM wrote: Dear Laura, > Very nice and definitely better than just using an ordinary shell script. Calling other programs to determine and report a target's dependencies, is convenient, efficient and not limited by any pattern description language. This is the reason why I prefer using redo over make: it assumes nothing about the target-dependencies relation but makes specifying it actually easier, even on "complicated" cases, by letting you use tools/(combination of tools) that are more appropriate. > Thanks for the links. If you write a manual, hope you'll let us know. > I'd be very interested in reading it. I've spent some time preparing the manual and it seems complete now. Have fun. Cheers!
Re: [dev] requirements of build systems
On 22/02/05 03:23, Petros Pateros wrote: > On Mon, Jan 31, 2022 at 06:38:01PM +0100, Страхиња Радић wrote: > > apenwarr/redo is the implementation of djb redo I settled on for my > > programs. It > > has the most features among the current implementations. > > I'm glad to hear that. But it doesn't implement the less-features feature > (that comes > along with the only-the-features-necessary-for-the-do-one-job-well-plan) that > I > was looking for ;-). I'm not talking about the unnecessary cruft here, just the most features within the djb's redo specification. signature.asc Description: PGP signature
Re: [dev] requirements of build systems
Thank you all for your answers. On Mon, Jan 31, 2022 at 06:38:01PM +0100, Страхиња Радић wrote: > apenwarr/redo is the implementation of djb redo I settled on for my programs. > It > has the most features among the current implementations. I'm glad to hear that. But it doesn't implement the less-features feature (that comes along with the only-the-features-necessary-for-the-do-one-job-well-plan) that I was looking for ;-). > For redistribution, it > has a shell script named "do", so even if users don't have full apenwarr/redo > installed on their system, they can execute > > $ ./do -c > > to build my programs, or > > # ./do -c install > > to install them. I have actually overseen this, thank you for mentioning. It seems that we need to solve the problem of characterising a dependency as out-of-date - when mtime is reliable - when mtime is not. It also seems to me that mtime is unreliable when a component of the system "breaks" mtime (NFS, etc.). So, since the degree of mtime reliability is known beforehand, I'd prefer having two separate build-system implementations, one that always relies on mtime and one that checks the file's contents. One would use the first for the majority of practical cases, and the second for the rest cases. This would solve the simple ones simply (with simplicity) and efficiently and the rest with the necessary complexity. Particularly with redo, one is able use both with the same interface (without any need for adapting the build scripts). If one deems mtime always unreliable, then they can only use the second. Personally, I think the first case has me covered. For those interested, baredo only hanldes the simple case, but can be trivially extented to handle the other cases as well (trivially when one has the code for the hashing or whatever). Feel free to contact me if I you need help in regards to this. Cheers!
Re: [dev] requirements of build systems
On Tue, Feb 01, 2022 at 09:29:55PM +0900, Pontus Stenetorp wrote: > Speaking for myself, I certainly have experienced issues with > inaccurate timestamps on NFS for compute clusters where its use is > very common. Not saying this as a supporter of NFS and the likes, just > as evidence that it does occur in practice. That's good to know, I wasn't thinking about NFS at all. Also just to be clear, my question wasn't rhetorical. Since I've never experienced issue with make, I was genuinely curious if this issue occurs in practice or not. - NRK
Re: [dev] requirements of build systems
*** Pedro Lucas Porcellis [2022-02-01 11:00]: >I think for most people out there, relying on mtime is just perfectly fine. No. mtime depends on time and filesystem implementation specifics. There are many systems where sysadmins like to do cron-ed ntpdate, which leads to jumping clocks. There could be filesystem/kernel issues like low mtime value granularity: https://stackoverflow.com/questions/14392975/timestamp-accuracy-on-ext4-sub-millsecond There are many other possible unexpected problems: https://apenwarr.ca/log/20181113 >I believe that generally, this kind of "futorology development thinking" >can lead to developing very messy and complex solutions, because "we >might hit that edgecase in the future that only 0.5% of users have >then". People like me want reliability, predictability and some expected behaviour from the software, to trust and rely on it. mtime usage safety depends on quantity of pitfalls, implementation specifics and OS configuration options. We can "fix" those problems to make usable systems friendly with software relying solely on mtime. Or we can just can make less expectation of underlying FS/OS and write software (redo in current case) that does not depend on all of that fragile metainformation. Instead of depending on some OS specific mtime/ctime/whatever inode field, that *should* be updated if file's content is changed (but we already know that is not expected from mmap-ed changes and FUSE-based filesystems), we can depend literally on content itself (on its cryptographic hash not to waste space). Reliable, predictable, OS/FS-independent solution. Personally I am completely ok to have slightly added delay because of data transfer and hash computation, because if we talk about practical development tasks, there there are no huge amounts of data involved. Possibly slightly worse performance, but perfect reliability and predictability I can rely on. -- Sergey Matveev (http://www.stargrave.org/) OpenPGP: CF60 E89A 5923 1E76 E263 6422 AE1A 8109 E498 57EF
Re: [dev] requirements of build systems
On Tue, Feb 01, 2022 at 09:29:55PM +0900, Pontus Stenetorp wrote: > On Tue 01 Feb 2022, NRK wrote: > > On Mon, Jan 31, 2022 at 12:10:27AM +0200, Petros Pateros wrote: > > > However, it was pointed out to me that relying on mtime can give > > > wrong results, for example: (a) if the clock is set backwards or > > > in case of insufficient granularity in mtime then a file that gets > > > modified might have the same mtime (b) an mmap(2)-ed file can get > > > modified but its mtime might not get updated soon enough > > > > How likely is it for these situations to occur in practice? If these > > are practical problems, then it makes sense to solve them. Otherwise > > I think it's best not to waste resource solving theoretical > > problems. > > Speaking for myself, I certainly have experienced issues with > inaccurate timestamps on NFS for compute clusters where its use is > very common. Not saying this as a supporter of NFS and the likes, just > as evidence that it does occur in practice. Yeah, but again, how common is that scenario on a regular, day-to-day development? I think for most people out there, relying on mtime is just perfectly fine. I believe that generally, this kind of "futorology development thinking" can lead to developing very messy and complex solutions, because "we might hit that edgecase in the future that only 0.5% of users have then". In the end it's like NRK said, in the end it's just a waste of resources. With best regards, Pedro Lucas Porcellis
Re: [dev] requirements of build systems
On Tue 01 Feb 2022, NRK wrote: > On Mon, Jan 31, 2022 at 12:10:27AM +0200, Petros Pateros wrote: > > However, it was pointed out to me that relying on mtime can give wrong > > results, > > for example: > > (a) if the clock is set backwards or in case of insufficient granularity in > > mtime > > then a file that gets modified might have the same mtime > > (b) an mmap(2)-ed file can get modified but its mtime might not get updated > > soon enough > > How likely is it for these situations to occur in practice? If these are > practical problems, then it makes sense to solve them. Otherwise I think > it's best not to waste resource solving theoretical problems. Speaking for myself, I certainly have experienced issues with inaccurate timestamps on NFS for compute clusters where its use is very common. Not saying this as a supporter of NFS and the likes, just as evidence that it does occur in practice.
Re: [dev] requirements of build systems
On Mon, Jan 31, 2022 at 12:10:27AM +0200, Petros Pateros wrote: > However, it was pointed out to me that relying on mtime can give wrong > results, > for example: > (a) if the clock is set backwards or in case of insufficient granularity in > mtime > then a file that gets modified might have the same mtime > (b) an mmap(2)-ed file can get modified but its mtime might not get updated > soon enough How likely is it for these situations to occur in practice? If these are practical problems, then it makes sense to solve them. Otherwise I think it's best not to waste resource solving theoretical problems. - NRK
Re: [dev] requirements of build systems
On Mon, Jan 31, 2022 at 09:34:52AM -0500, LM wrote:
Dear Laura,
> This looks like a very interesting project. I've been looking for some
> alternatives to GNU make.
I hope you find something to fit your needs.
> I do use a few GNU make features that I'm not sure about porting to other
> alternatives. I use wildcards a lot for specifying files. I use the
> substitute features. I also use static pattern rules a lot:
> https://www.gnu.org/software/make/manual/make.html#Static-Pattern Was
> wondering if redo has functionality to replace those features. How hard
> would it be to target .po files and generate .mo files with the proper
> naming conventions using redo? So, en.po generates
> en/LC_MESSAGES/programname.mo. I currently use GNU make and wildcards to
> figure out what .po files there are and generate the .mo files placed in
> applicably named directories.
By default, /bin/sh is called to interpret the .do files, like
/bin/sh -e /path/to/script.do args.., unless they are executables, in which
case they are just executed. So expanding a wildcard is trivial.
all.do: (contents of all.do)
redo listofmo # always (re)generate it
redo-ifchange $(cat listofmo) # IFS is relevant here
listofmo.do:
# create/update the list of .mo's
# stdout is captured and will atomically (re)place listofmo,
# unless an error occurs
ls *.po |
while read -r po
do
printf '%s\n' "${po%.po}/LC_MESSAGES/programname.mo"
done
default.mo.do:
# $1: the name of the file this .do file is called to produce,
as a path relative to $0 ($0 is /path/to/default.mo.do)
# $2: $1 but without the .mo suffix
# $3: see below
po=${1%/*/*}.po
redo-ifchange "$po" # declare that $1 depends on $po
# now generate .mo from .po, either by creating $3 or by writing
# to stdout. Doing both is or affecting $1 directly is an error.
#po2mo "$po" > "$3"
#po2mo "$po" -o "$3"
#po2mo "$po" # if po2mo writes to stdout
With the above, running
$ redo all
will generate listofmo and the */LC_MESSAGES/programname.mo files
from the *.po files.
With this approach, however, listofmo is generated every time so that they are
up to date (in case of adding/removing a .po file).
Some redo implementations, like goredo [0], also store a hash of the created
file so nothing that depends on them gets rebuild when its content hasn't
changed.
> Any pointers to documentation on how to use
> redo in a project in place of GNU make?
You can find a lot in:
https://redo.readthedocs.io/en/latest/
http://www.goredo.cypherpunks.ru/Usage-rules.html
I also plan to write manual pages for baredo.
Cheers!
[0] http://www.goredo.cypherpunks.ru/index.html
Re: [dev] requirements of build systems
On 22/01/31 12:10, Petros Pateros wrote: > What would you expect from a build system? Should it trust mtime? > Is it responsible for verifing the file's contents for actual changes? > More generally, what are the key features that make a build system useful? apenwarr/redo is the implementation of djb redo I settled on for my programs. It has the most features among the current implementations. For redistribution, it has a shell script named "do", so even if users don't have full apenwarr/redo installed on their system, they can execute $ ./do -c to build my programs, or # ./do -c install to install them. signature.asc Description: PGP signature
Re: [dev] requirements of build systems
>From my experience with using pure-POSIX make for many years now, I have
almost no troubles. The only recurring trouble I have is with complex
dependency chains (which I aim to keep to a minimum anyway for
simplicity) and mtime over sshfs connections. Other than that, make has
never failed me.
My best advice for make-based build systems is to try and keep things
simple. Unless you have a *really* complicated build process (like a
kernel image or something), don't be too fancy. Just use a single,
central Makefile and put your sources in one place. If you find yourself
having to use GNU extensions, that's often a sign of your process being
too complicated.
For me, make would be more useful if it could:
1) Auto-detect sources based on regular expression (build src/*.{ch})
2) Have a clean way of managing dependencies between files
But I am generally happy with what I have.
Ethan
[dev] requirements of build systems
Hello, I hope you all are doing well. Some time ago, I was using redo for building a C project, but redo, unlike make is non-standard, so one needed a redo implementation to build the project. At the time, redo implementations where either written in Python,Go or had annoying bugs like creating a target even when it was phony. Also, it seemed to me that no redo really reflected how powerful redo's simplicity is, so I tried implementing it myself, resulting in baredo [0]. It should be stable enough to be used in C projects without much effort, since a C compiler is already required. However, it was pointed out to me that relying on mtime can give wrong results, for example: (a) if the clock is set backwards or in case of insufficient granularity in mtime then a file that gets modified might have the same mtime (b) an mmap(2)-ed file can get modified but its mtime might not get updated soon enough Various redo implementations store a hash of the file's contents so they can safely decide if its contents have changed or not. Personally, I find that mtime is reliable enough, (redo especially compares for equality). What would you expect from a build system? Should it trust mtime? Is it responsible for verifing the file's contents for actual changes? More generally, what are the key features that make a build system useful? For example, redo: - handles filenames correctly (including those with spaces and newlines). This depends on the scripts respecting that - can use any executable as building "script" - does no macro processing etc. and instead leaves that to the .do file - flexible/easy dependencies declartions, redo doesn't need to know the whole dependency tree in advance, e.g. redo-ifchange hdrs $(cat hdrs) - can handle dependencies for inexistent files - each target is treated the same independently of who requires it - creates its targets atomically - it basically does only dependecy checking and executing the .do files so it is needs few code - easilly supports building targets in parallel without issues make: - has standardized behavior - depends on error-prone mtime comparisons - doesn't support filenames with spaces (feel free to correct me) I'm interested in hearing your opinions/experiences about building systems in general. Cheers! [0] https://github.com/gotroyb127/baredo
