Hello Nataneal, >> It works as follows (somewhat simplified): >> -At startup (pre-init) the sha1-sums of files listed in a >> <package>.local >> file are calculated and saved in a <package>.sha1 file. The files listed >> in <package>.local are user changable files like config files and f.e. >> ssh-keys -At backup time the contents of the <package>.sha1 is compared >> with the actual files on the system and only the changed files are >> backuped. This is done for all packages on the system and only one >> "configdb" is >> created with changes from all packages. It's also dynamic so files no >> longer available on the system (the package is removed) will be removed >> from it. > > This is really interesting. I have been working on something very > similar on the alpine linux project (http://alpinelinux.org). I think the > concept is basicly the same so I'd like to share how we were thinking. > > First, splitting configs from binaries are essensial for upgrading. > Second, you are normally not interested in configs per package but more > configs per host, which leads to 1 config backup per host. It Looks like > thats what you found out too. > Correct, although you and Nathan pointed us in that direction. The packages still contain a "sane" base config, just as it is now. So when no "configdb" is available the system still boots. The "configdb" only contains the user changes.
> Second, we found out that all that actually got saved on the floppy, was > all found in /etc direcotory. I think there was one pacakge (ppp) that > needed to backup something in from /var but it was really config files > that belongs in /etc. So the conlusion was, you only need to backup things > in /etc. If you need to save things in /var, then you actually need a > harddrive and are using the wrong tool for th task. > > This leads us to: create checksums (or save "status") of everything > in /etc during package installation (during boot in other words). This is > the "default", or "pristine" database. Do your changes in /etc. Then > compare current /etc with the recorded status. Now you know what you need > to backup (and thus don't really need the .local files) > Correct, although we took a slightly different direction. The reason we use .local files is that when you boot the /etc directory grows with every installed package (files are added), so the sha1sum is calculated again and again for the same files which cost time. Ofcourse it's possible to calculate the sha1sums at the end (when all packages are installed) but that would remove the link between packages and sha1sums and give some problems when adding a new package at runtime. Currently we create a <package>.sha1 file for every package instead of one big sha1 file. I played with it a few days and this implementation seems to work most flexible, although I can't rememember all the reasons exactly anymore ;) An other reason is that using .local keeps the possibility to add some files normally not in /etc, like shorewall macros in /usr/share/shorewall, files in /root or ssh-keys, etc. > Nathan wrote a simple file integration checker > (http://sfic.sourceforge.net) that could recursively prase a directory, > and then saves the filesizes, create/mod times, permissions, checksums etc > in a tdb database (http://sourceforge.net/projects/tdb) I wrote a wrapper > script, a local backup manager that do the check and creates a tar archive > on floppy. The local backup utility (lbu) is inspired by svn and works > pretty much similar. > > lbu status - will list all changes shince last commit to floppy. "lbu > status -a" will show all the changes. > > lbu commit floppy - will save all the modified files to a tar archine on > floppy(you can also do a "lbu commit usb" to save to usb stick instead of > floppy) > > lbu add - will add a file manually to the backup list (if you need to > backup something thats outside /etc) > > Funny think is I got it working at the end of last week actually. > That's a great tool! I did look at sfic and I really like the idea of a real database for storing the changes, but we still take size into account and the current "script" implementation doesn't add any size and has more or less the same functionality (not as advanced ofcourse :). But there is ofcourse nothing against using a tool like this when there is enough space, afterall when backups are done outside the package every tool can be used to store the config. The only thing is, how are the changes saved in the database used at boottime? The "configdb" package in our implementation is always loaded last in pre-init, but when using a database this seems more difficult to me (it has to be loaded before init). >> >> Some of the advantages: >> -Upgrading should be easier in most cases (config is preserved) >> > > I'm (currently) using apk-tools for the alpine project. It supports > package upgrades. apk_add -u <package>. It will look for config files, > backup, and then install the new config file as .apk-new and preserve the > original one. It also updates the default/pristine database so next time > you do a "lbu commit" it won't backup files that are identical to the new, > updated config files. > > Then it should be possible to parse the /etc dir and look for .apk-new > files, run a diff and merge in changes. (like gentoo's etc-update or > freebsd's mergemaster) > Yes, that's something I'm also thinking of as a next step. I see the proposed config/backup system as a first step, this kind of improvements can be added later without changing the base system. The reason we take the lrp package format as starting point is the enormous amount of work it would take to change all of our buildtools, documentation, etc, etc to change the extension. But at least using the same kind of backup direction brings us closer together to a point that we can even use eachothers ideas and tools ;) >> -It is possible to create specific "configd" and "moddb" packages for >> specific hardware or functionality without creating special packages >> (WRAP, ...) >> -The packages are "static", so no more corruption or "growing" root.lrp >> -<package>.list and <package>.exclude.list can be removed >> > > We even removed <package>.local, since every config files should live > in /etc. > Agree, but see my comments above. Ofcourse having a .local file in the package doesn't "hurt". It's a question of the configtools to use it or not. >> Because this system creates some package incompatibility (no more .list >> files) we plan to do this with an uClibc upgrade (which isn't >> backwards compatible also). The changes are not very drastic and >> relative easy implemented in buildtool, so we could make the change >> without a lot of effort. >> >> Comments? >> > > If you are interested in testing the very same concept, but without > any .local files and using Nathan's sfic utility, you can download the > latest beta of alpine. > > http://tanael.org/alpine/v1.1/iso/alpine-1.1.3_beta3-060610-i386.iso > http://tanael.org/alpine/v1.1/iso/alpine-1.1.3_beta3-060610-i386.iso.sha1 > > > (warning! it's 137 MB on a 2Mbit shdsl line) > > > You can also try run it from USB: > > > http://tanael.org/alpine/v1.1/usbdrive/ > > > Since the documetation is not so good, here are the basics to get > started: > > > To list available packages: > apk_fetch -lv > > To install a package: > apk_add <package> > > To configure the basics (incl networking): > setup-alpine > > Try add/delete/modify anything in /etc, then run: > lbu -h lbu status lbu commit -h lbu commit -n floppy > Thanks for your comments and pointers! Looking forward for more cooperation. > > -- > Natanael Copa > Eric > > _______________________________________________ leaf-devel mailing list leaf-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/leaf-devel
