On Mon, 2006-06-12 at 11:39 +0200, Eric Spakman wrote: > Hello Nataneal,
[ cutted a few lines ] > > This leads us to: create checksums (or save "status") of everything > > in /etc during package installation (during boot in other words). This is > > the "default", or "pristine" database. Do your changes in /etc. Then > > compare current /etc with the recorded status. Now you know what you need > > to backup (and thus don't really need the .local files) > > > Correct, although we took a slightly different direction. The reason we > use .local files is that when you boot the /etc directory grows with every > installed package (files are added), so the sha1sum is calculated again > and again for the same files which cost time. You cannot do that. install package A. modify config for package A. install package B. Now, you cannot recaluclate the checksums for all packages, since some are already changed. So you can only create checksums for the new files in /etc. I grep the file list (generated by tar -ztf ...) for ^etc/ and calculate the checksums for those files only. Actually, sfic does it and appends it to the global database. > Ofcourse it's possible to > calculate the sha1sums at the end (when all packages are installed) but > that would remove the link between packages and sha1sums and give some > problems when adding a new package at runtime. Yes. You have to calculate the checksums at package installation for every package. > Currently we create a <package>.sha1 file for every package instead of one > big sha1 file. I played with it a few days and this implementation seems > to work most flexible, although I can't rememember all the reasons exactly > anymore ;) Probably because its difficult to remove pacakges otherwise. Thats probably also why Nathan wrote the sfic tool at all. yafic cannot add/remove files without rebuilding the entire database. > An other reason is that using .local keeps the possibility to add some > files normally not in /etc, like shorewall macros in /usr/share/shorewall, > files in /root or ssh-keys, etc. Generally, if you need to store something outside /etc you need a disk. But, sure, there are situations where you need to add things thats not in /etc. We worked around that by having a backuplist of all files to be backed up. so I can add files that is not handeled by sfic at all: lbu add /root/.ssh/* /usr/share/shorewall Those files would not be included in the tests if anything have changed. But they would be included in local backup. (maybe I need to do some work here...?) However, I can specify what directories should be handeled by the lbu, so if I want those dirs to be controled by lbu/sfic I just define the dirs. export APK_LBUDIRS="etc/:usr/share/shorewall/:root/.ssh/" Now would "lbu status" detect any changes in those dirs too. > > Nathan wrote a simple file integration checker > > (http://sfic.sourceforge.net) that could recursively prase a directory, > > and then saves the filesizes, create/mod times, permissions, checksums etc > > in a tdb database (http://sourceforge.net/projects/tdb) I wrote a wrapper > > script, a local backup manager that do the check and creates a tar archive > > on floppy. The local backup utility (lbu) is inspired by svn and works > > pretty much similar. > > > > lbu status - will list all changes shince last commit to floppy. "lbu > > status -a" will show all the changes. > > > > lbu commit floppy - will save all the modified files to a tar archine on > > floppy(you can also do a "lbu commit usb" to save to usb stick instead of > > floppy) > > > > lbu add - will add a file manually to the backup list (if you need to > > backup something thats outside /etc) > > > > Funny think is I got it working at the end of last week actually. > > > That's a great tool! I did look at sfic and I really like the idea of a > real database for storing the changes, but we still take size into account > and the current "script" implementation doesn't add any size and has more > or less the same functionality (not as advanced ofcourse :). Dize does matter for us also, but probably not as mucha s for you (we have SSP compiled packages so we have sacrificed some size/speed for security) sfic uses tdb and not gdbm because of size. I just checed, the sfic utility is approx 30kb (with SSP) and the libtdb.so is approx 20kb. Thats 50kb. Nathan... thats much. the entire apk-tools suit is approx 46kb. > But there is > ofcourse nothing against using a tool like this when there is enough > space, afterall when backups are done outside the package every tool can > be used to store the config. Yes. We were talking about separating local backup handling from package manager. (we have been talking of switch to ipkg). However, you do need to create > The only thing is, how are the changes saved in the database used at > boottime? The "configdb" package in our implementation is always loaded > last in pre-init, but when using a database this seems more difficult to > me (it has to be loaded before init). I have runtimes init.d script that loads the runtimes and local package overlays (local configs in other words). The overlay itself is just a tar archive that is extracted to $ROOT. Right after the overlay is extracted I run "lbu update": Here is the code: cmd_update() { [ $# -lt 1 ] && usage_update cd "$ROOT" list=$( for i in "$@" ; do echo $i | sed 's:^/::' done | lbu_filter ) $SFIC --delete "$COMMITED_TDB" $list $SFIC --add "$COMMITED_TDB" $list } lbu_filter will filter out everything except whats in APK_LBUDIRS. So after the local config is loaded, I replace the entries in the commited.tdb (recreate the checksums for those files) Nathan: maybe we should have a "sfic --update" or "sfic -u --add" so we dont need to delete the files? Just a thought before I forget. > >> > >> Some of the advantages: > >> -Upgrading should be easier in most cases (config is preserved) > >> > > > > I'm (currently) using apk-tools for the alpine project. It supports > > package upgrades. apk_add -u <package>. It will look for config files, > > backup, and then install the new config file as .apk-new and preserve the > > original one. It also updates the default/pristine database so next time > > you do a "lbu commit" it won't backup files that are identical to the new, > > updated config files. > > > > Then it should be possible to parse the /etc dir and look for .apk-new > > files, run a diff and merge in changes. (like gentoo's etc-update or > > freebsd's mergemaster) > > > Yes, that's something I'm also thinking of as a next step. I see the > proposed config/backup system as a first step, this kind of improvements > can be added later without changing the base system. Yes. > The reason we take the lrp package format as starting point is the > enormous amount of work it would take to change all of our buildtools, > documentation, etc, etc to change the extension. find . -exec sed -i 's/\.lrp/\.apk/g' {} \; Job done! ;-) just kidding... I know its lot of work. However, you should know that apk-tools can install an existing lrp package if you only rename it. mv foo.lrp foo-1.0.apk apk_add foo-1.0.apk If you dont have sfic installed on your system, it will work anyway, you just cannot use "lbu". For ourselves, we found out that it indeed is to much work, so we just started from "scratch". We needed 2.6 kernel anyway. Thats why alpine exists at all. > But at least using the > same kind of backup direction brings us closer together to a point that we > can even use eachothers ideas and tools ;) Yes! The basic concepts are the same. Boot kernel, load runtimes into ram, load configs from floppy/usbstick. > >> -It is possible to create specific "configd" and "moddb" packages for > >> specific hardware or functionality without creating special packages > >> (WRAP, ...) > >> -The packages are "static", so no more corruption or "growing" root.lrp > >> -<package>.list and <package>.exclude.list can be removed > >> > > > > We even removed <package>.local, since every config files should live > > in /etc. > > > Agree, but see my comments above. Ofcourse having a .local file in the > package doesn't "hurt". It's a question of the configtools to use it or > not. I actually think I have it myself. I think apk_add will look if there is a CONFFILES listing, which is exactly the same thing as .local in lrp. If its there, use it, if its not look up APK_LBUDIRS. So you can override it. After a second thought, its probably not working. when comparing against "current", the files outside etc/ will show up as deleted. I can remove that code. Need to think mor on that. [ a few lines cutted ] > Thanks for your comments and pointers! Looking forward for more cooperation. Thanks to you too! ;-) -- Natanael Copa _______________________________________________ leaf-devel mailing list leaf-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/leaf-devel
