Hello Nataneal,
>>
>
> [ cutted a few lines ]
>
>> Correct, although we took a slightly different direction. The reason we
>> use .local files is that when you boot the /etc directory grows with
>> every installed package (files are added), so the sha1sum is calculated
>> again and again for the same files which cost time.
>
> You cannot do that.
>
> install package A. modify config for package A. install package B.
>
> Now, you cannot recaluclate the checksums for all packages, since some
> are already changed. So you can only create checksums for the new files in
> /etc. I grep the file list (generated by tar -ztf ...) for ^etc/ and
> calculate the checksums for those files only. Actually, sfic does it and
> appends it to the global database.
>
That's not what I meant ;)
I meant if you boot the system and everytime a package is installed (still
pre-init) and you would calculate the sha1 sum, the files in /etc from
packages already installed are calculated again. But I see you are using
the filelist for that which solves the duplication.
>> Ofcourse it's possible to
>> calculate the sha1sums at the end (when all packages are installed) but
>> that would remove the link between packages and sha1sums and give some
>> problems when adding a new package at runtime.
>
> Yes. You have to calculate the checksums at package installation for
> every package.
>
>> Currently we create a <package>.sha1 file for every package instead of
>> one big sha1 file. I played with it a few days and this implementation
>> seems to work most flexible, although I can't rememember all the reasons
>> exactly anymore ;)
>
> Probably because its difficult to remove pacakges otherwise. Thats
> probably also why Nathan wrote the sfic tool at all. yafic cannot
> add/remove files without rebuilding the entire database.
>
Correct, although it's somewhat difficult to remove a package without a
.list file. Currently package removing is not implemented, although an
implementation where a package list is created dynamicaly (with tar -ztf
at boottime) should be possible. I'm not sure if package removing is worth
the hassle, it has a lot of corner cases so the easiest way to remove a
package without problems is to remove it from the load list and reboot :)
That's one of advantages of a system in RAM.
>> An other reason is that using .local keeps the possibility to add some
>> files normally not in /etc, like shorewall macros in
>> /usr/share/shorewall,
>> files in /root or ssh-keys, etc.
>
> Generally, if you need to store something outside /etc you need a disk.
> But, sure, there are situations where you need to add things thats not
> in /etc.
>
> We worked around that by having a backuplist of all files to be backed
> up. so I can add files that is not handeled by sfic at all:
>
> lbu add /root/.ssh/* /usr/share/shorewall
>
> Those files would not be included in the tests if anything have changed.
> But they would be included in local backup. (maybe I need to do some
> work here...?)
>
> However, I can specify what directories should be handeled by the lbu,
> so if I want those dirs to be controled by lbu/sfic I just define the dirs.
>
>
> export APK_LBUDIRS="etc/:usr/share/shorewall/:root/.ssh/"
>
> Now would "lbu status" detect any changes in those dirs too.
>
That's also an option, but I personal like this sort of functionality in
the "base" without working around. But anyway it's only a choice, and like
I said having .local files in the package don't hurt, there is always a
choice to use them or not.
<snip>
>> That's a great tool! I did look at sfic and I really like the idea of a
>> real database for storing the changes, but we still take size into
>> account and the current "script" implementation doesn't add any size and
>> has more or less the same functionality (not as advanced ofcourse :).
>
> Dize does matter for us also, but probably not as mucha s for you (we
> have SSP compiled packages so we have sacrificed some size/speed for
> security)
>
> sfic uses tdb and not gdbm because of size.
>
> I just checed, the sfic utility is approx 30kb (with SSP) and the
> libtdb.so is approx 20kb. Thats 50kb.
>
> Nathan... thats much. the entire apk-tools suit is approx 46kb.
>
That's indeed too much for us (in a floppy setup), the code we are using
is only ~20 lines of shell scripting.
>
>> But there is
>> ofcourse nothing against using a tool like this when there is enough
>> space, afterall when backups are done outside the package every tool
>> can be used to store the config.
>
> Yes. We were talking about separating local backup handling from package
> manager. (we have been talking of switch to ipkg). However, you do need to
> create
>
Some text is missing?
>> The only thing is, how are the changes saved in the database used at
>> boottime? The "configdb" package in our implementation is always loaded
>> last in pre-init, but when using a database this seems more difficult
>> to me (it has to be loaded before init).
>
> I have runtimes init.d script that loads the runtimes and local package
> overlays (local configs in other words). The overlay itself is just a tar
> archive that is extracted to $ROOT. Right after the overlay is extracted I
> run "lbu update":
>
> Here is the code:
>
>
> cmd_update() { [ $# -lt 1 ] && usage_update
> cd "$ROOT" list=$( for i in "$@" ; do echo $i | sed 's:^/::' done |
> lbu_filter ) $SFIC --delete "$COMMITED_TDB" $list
> $SFIC --add "$COMMITED_TDB" $list
> }
>
>
> lbu_filter will filter out everything except whats in APK_LBUDIRS.
>
> So after the local config is loaded, I replace the entries in the
> commited.tdb (recreate the checksums for those files)
>
Our approach is to load the "configdb" package as last package in linuxrc,
so before init is started. The files in the configdb will overwrite the
existing config files pre-init so those are used instead of the ones in
the packages themselve. The sha1sums of the original files in the packages
are calculated before that.
Ofcourse this means that the backup system can't see if a change is saved
before, it always notes a difference (the calculated sha1 sum is from the
original package). This doesn't matter that much because the "configdb"
package is small, so saving is fast and it also makes sure "old" files are
removed from the configdb package.
<snip>
>
>> The reason we take the lrp package format as starting point is the
>> enormous amount of work it would take to change all of our buildtools,
>> documentation, etc, etc to change the extension.
>
> find . -exec sed -i 's/\.lrp/\.apk/g' {} \;
>
> Job done! ;-)
>
LOL
It's not only the extension ofcourse, also mime types, package creation
and package format changes and other thing I forget ;)
>
> just kidding... I know its lot of work. However, you should know that
> apk-tools can install an existing lrp package if you only rename it.
>
> mv foo.lrp foo-1.0.apk apk_add foo-1.0.apk
>
> If you dont have sfic installed on your system, it will work anyway, you
> just cannot use "lbu".
>
> For ourselves, we found out that it indeed is to much work, so we just
> started from "scratch". We needed 2.6 kernel anyway. Thats why alpine
> exists at all.
>
>> But at least using the
>> same kind of backup direction brings us closer together to a point that
>> we can even use eachothers ideas and tools ;)
>
> Yes! The basic concepts are the same. Boot kernel, load runtimes into
> ram, load configs from floppy/usbstick.
>
Indeed!
For the list:
Currently the first device in PKGPATH is the "backup" device where configs
and modules live (and ofcourse packages if PKGPATH has only one entry,
just as it is now).
Some notes about modules: the moddb package is just a backup of the
/lib/modules/ directory.
>>>> -It is possible to create specific "configd" and "moddb" packages
>>>> for specific hardware or functionality without creating special
>>>> packages (WRAP, ...)
>>>> -The packages are "static", so no more corruption or "growing"
>>>> root.lrp -<package>.list and <package>.exclude.list can be removed
>>>>
>>>>
>>>
>>> We even removed <package>.local, since every config files should live
>>> in /etc.
>>>
>> Agree, but see my comments above. Ofcourse having a .local file in the
>> package doesn't "hurt". It's a question of the configtools to use it or
>> not.
>
> I actually think I have it myself. I think apk_add will look if there is
> a CONFFILES listing, which is exactly the same thing as .local in lrp. If
> its there, use it, if its not look up APK_LBUDIRS. So you can override it.
>
It's not exactly the same, but close. Some files like ssh-keys, password
and group shadow files and so on are not config files and not listed in a
conf entry but should be backupped.
>
> After a second thought, its probably not working. when comparing against
> "current", the files outside etc/ will show up as deleted. I can remove
> that code. Need to think mor on that.
>
> [ a few lines cutted ]
>
>
>> Thanks for your comments and pointers! Looking forward for more
>> cooperation.
>
> Thanks to you too! ;-)
>
>
> --
> Natanael Copa
>
>
Eric
>
_______________________________________________
leaf-devel mailing list
leaf-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/leaf-devel
