Hi list This long email is just a report on my recent stretch to buster upgrade experience. I had a bit of an adventure, didn't handle some steps well, and thought the experience would be useful to put out there for others to learn from / avoid some mistakes I made.
THERE IS NO QUESTION / PROBLEM TO SOLVE AT THE END OF THIS -- it's just here for info and in the hope it will help someone. If you don't like long emails, please know that I couldn't care less. I was upgrading a Stretch desktop system to Buster. I've moved most of my most critical activities into the cloud and so my machine isn't nearly as mission-critical as it once was, but outages would still be annoying enough that I would want ideally to avoid significant downtime. I have other users on this machine, who occasionally access my machine remotely via a VPN, but they are very much minority users and I am the most common user on my machine by a long way. I perform 5-days-a-week backups using Amanda. The machine in question here happens to be the Amanda server as well as a client. The amanda virtual tapes and holding disk are in a USB3 disk cage. The cage also contains a RAID1 disk pair which contains data of various types. Internally my machine has 2 1TB SSDs, one of which cotains most of my filesystem and the other is mounted at /opt. /opt contains a fairly extensive mariadb server, a SVN repository, and hard disk space for 2 Windows VMs. The upgrade was expected to have a large effect on the disk containing / and minimal effect on the disk containing /opt. I verified before starting that my most recent backups had been successful. I disabled the systemd timer job I'd created to run the backups each night so that if I did have a messed-up system half-upgraded during the process, it didn't corrupt my backups. However I wasn't super confident that I could restore from my backups willy-nilly as I had had only limited occasion to do so before (turns out I didn't need to worry about this, I ended up restoring liberally from my backups and it went fine). Before starting I additionally rsync'd my entire home directory and select parts of /etc (mysql / mariadb config, hard-won exim4 config, openvpn config etc) to spare disk capacity in the drive cage ie off-machine, just in case. I MISSED my SVN config in this backup, incidentally.I had an automatic nightly backup of it but I did not hand-crank an additional backup of it before starting, because I forgot about it. I carefully read the upgrade instructions on the Debian website. I noticed that I needed to migrate to ostensibly-predictable network interface names before doing the upgrade, so I duly executed the steps prescribed for that. That worked fine and internet access was none the worse for it afterward, before upgrading. I noticed and didn't completely understand some warnings about ssh and, I think, old types of keys. I didn't do anything about that because I didn't really understand what that was saying. Sometimes I think some of these notes could do with a "What This Means In Plain English" section... Anyway that didn't seem to have any problems in the end. I went through my sources.list.d directory and peeled out a couple of things I didn't expect to do in the upgrade eg Virtualbox's repository, the Google Talk plugin (gawd alone knows where that came from), and Christian Marillat's deb-multimedia repository. I thought I should do the upgrade without them and hoped to wean myself off deb-multimedia while I was at it. This, as it turned out, was a huge mistake. After all this I modified my sources.list according to the upgrade instructions and apt update'd. Next, I did an apt upgrade to upgrade some stuff before the main apt dist-upgrade, again following the upgrade instructions. This was the last command my system executed as a healthy system... Obviously there was a lot to download and so on, so I let it get on with it. Eventually, after a lot of downloading and quite a lot of package upgrading, the process failed with an error. The issue was gstreamer and its library, which had hit a problem trying to upgrade because they were trying to overwrite a file also contained in the gstreamer-plugins-bad package. This turned out to be a deb-multimedia problem. I had hoped that upgrading to Buster versions of packages would supercede the deb-multimedia repository package versions, but they didn't. Now, in this situation, it didn't seem to be possible to use apt to go forward or back. If I tried to repeat the upgrade, install anything new, remove anything, or anything else, it would immediately stop saying there are broken packages, fix them with apt --fix-broken install. And if I did apt --fix-broken install, it would complain about the file write conflict I highlighted above. At this point I tried to use the instructions in the upgrade instructions for handling this error message, and I don't think I did it right because the situation just steadily got worse and worse. Force removing packages using dpkg --force-depends -r <package> just moved the problem from one place to another. I ended up chasing one tree of dependencies all the way down to a conflict involving libstdc++6. By then I was so trigger-happy I force-removed that, and that was the end of using apt to do much of anything... I was still able to use wget to re-download the libstdc++6 package for stretch, and tried to re-install it using dpkg (which doesn't need libstdc++6). When that actually failed to install, despite the fact it had just been installed, I concluded that my package management was so corrupted on this machine that there was nothing for it but a re-install. Since I had all my backups, I downloaded the netinst iso on another computer and burned it to a USB stick. I also put my /etc/fstab on that stick as it would help me re-mount /opt and my drive cage when I was done with the re-install. I also made sure I had my amanda config separately backed up as I wasn't confident (again, oh me of little faith as it turned out) that I had my config set up appropriately to be able to be restored into a fresh machine. The re-install went smoothly and, using my retained copy of my old /etc/fstab, I was able to bring my second SSD back in as /opt and get the disks in my drive cage back, including my amanda backup disks. I was then able to restore my home directory, and key config such as exim4, SVN, mariadb etc etc from my amanda backups, so one positive thing to come out of this is I'm now much more confident my backups are actually worthwhile as they've now been successfully battle-tested. The only thing I haven't yet gotten around to since the re-install is to re-install VirtualBox from the VB repositories. I have the virtual disks etc for the VMs, but I haven't yet restored the VM configs from my backups (although there's now nothing stopping me from doing that, I just haven't gotten around to it yet). In the end I think my problems were caused by previously having the deb-multimedia repositories in use and trying to move away from them at the same time as upgrading. I don't know if there's a way to get away from deb-multimedia once items have been installed from it, but now I know that way is NOT to try to upgrade to the next Debian release, not including deb-multimedia... Although I do think some of the multimedia-facing packages in Debian are (or used to be) configured in a less than optimal way (ffmpeg, I'm looking at you) I now think rebuilding the relevant packages from source might be a better approach than using deb-multimedia. I'm not sure what the nature of the fight between Marillat and the rest of the Debian community is (and I'm _very_ sure I don't want to wade into the middle of it), but I think it's a pity he wasn't somehow able to be kept in the fold because he does seem to know what he's doing when it comes to multimedia tools. He was also very helpful some years back when I was trying to debug a problem I was having with one of the tools (that's long pre-stretch). But if there's no non-destructive way out of a deb-multimedia-ised computer, it's probably best not to venture there... Mark
