On 11/24/2017 08:11 AM, Luiz Angelo Daros de Luca wrote: >>> After applied, the list of files in backup in a fresh installation change >>> from: >>> to: >>> >>> root at LEDE:~# sysupgrade -l >>> /etc/config/dhcp >>> /etc/config/firewall >>> /etc/config/network >>> /etc/config/system >>> /etc/dropbear/dropbear_rsa_host_key >>> >> >> Hi, >> >> when I set up new devices I usually do an export of the initial >> configuration to track >> the changes I made externally in a git repo. >> With this patch I would just get the five files mentioned above, thus an >> incomplete config and would not >> be able to track my config changes. > > Hi, > > Yes, a diff between two backups will not show actually what has changed as > modified files will appear as new. > > However, maybe a configuration backup is not the best way to keep track of > changes on configuration files. > The list of "configuration file" can change as long as their configuration > are not listed in /lib/upgrade/keep.d. > Backup consists of files listed in /etc/sysupgrade.conf, /lib/upgrade/keep.d > and changed config files (tracked by opkg). > For example, /etc/firewall.user will not appear in backup if not modified (or > opkg is missing) because it is not mentioned in > /lib/upgrade/keep.d. There are many more examples. Try yourself: > > # find $(cat /lib/upgrade/keep.d/*) -type f -o type l 2>/dev/null | sort -u > >/tmp/a.txt > # cat /usr/lib/opkg/info/*.conffiles | sort -u>/tmp/b.txt > # diff -u /tmp/a.txt /tmp/b.txt | grep + > > So, even today, it is already expected that a backup after fresh installation > to have less files than a backup > after some packages where installed. I just made the difference bigger. > > BTW, the backup is broken whenever opkg is missing, as those files will never > get into backup. I don't know if it's worth it to try > to fix backup without opkg.
I'll eventually get to it, as the LEDE-based Gluon mesh framework generally ships without opkg, and without most of /usr/lib/opkg, on small (4MB) devices. I plan to ship a minimal /usr/lib/opkg that just contains the config hashes, and include a small replacement for `opkg list-changed-conffiles` in sysupgrade). > > I could add a new option for sysupgrade in order to keep all files listed, > even those untouched, in order to get the old behaviour. > However, in order to use it, you'll need to call it using a terminal. In this > case, you could also retrieve those files using ssh+tar, or rsync if avaiable. > Anyway, this option would only make sense if I grab also all config files > mentioned in any installed package, even those untouched. > > I'm trying to make OpenWrt/LEDE upgrade process closer to what we get in a > normal Linux distribution. The inclusion of only > modified files is a trick to reproduce the same logic rpm/deb uses for > configuration updates (replace what wasn't touched, > keep what was modified). The current behaviour is already breaking > installations by, at least, propagating outdated /etc/profile. > I'm trying to fix backup/upgrade process from now on, even if it could > "break" the use of backup for things not related to restore/upgrade. > IMHO, it pays off. As mentioned before, I don't like the idea of checking /rom to find out what has changed. It will make sysupgrade behave differently on devices that don't have an overlay and thus no /rom. A better fix (that would also be closer to what "normal Linux distributions" do) would be to reduce the use of /lib/upgrade/keep.d, and instead rely on opkg's config hash tracking for a lot more files (we should make this work without opkg itself first though). This would also work with Moritz's usecase of initial config files supplied in the image build via /files, as these files would not match the hashes known to opkg. /lib/upgrade/keep.d should still be supported IMO, it is useful for adding directories and files to the backups that generated dynamically (Gluon has a few usecases for this, e.g. packages that clean up behind themselves by adding cleanup scripts that remove old configuration on the first boot with a new image that doesn't have the package using the configuration anymore). As /lib/upgrade/keep.d would mostly be used for files that don't exist in /rom anyways, we can keep the behaviour of backupping these files unconditionally, and don't need to introduce (more) overlay-specific hacks. > > I already have a patch v2 ready > https://github.com/luizluca/source/tree/better-backups_skip-unchanged. > I'm just waiting because it is now based on the submitted "[PATCH] > base-files: fix sysupgrade -b/-l when -c is used". > Or maybe I should just send already v2 including both. > > Luiz Thanks for bringing this up. I generally agree that our backup strategy could use some mending, but I'd prefer to keep /rom hacks out of our scripts as much as possible. Matthias
signature.asc
Description: OpenPGP digital signature
_______________________________________________ Lede-dev mailing list Lede-dev@lists.infradead.org http://lists.infradead.org/mailman/listinfo/lede-dev
