Hi All,
I know this might have been asked before but I'm curious what you all are doing
for backups. I mostly want to hear from people who have a method in place
they're using right now, rather than tools you've "heard of" but don't have
firsthand experience with. We all know that backups are important but the
"how" and "why" is something I don't see talked about very much (but that might
just be a "me" thing).
Please share the following info. I think it would be very informative of how
we're backing up Linux in real world scenarios!
1. What tools are you using for OS backups and what's your process?
2. How many systems do you backup and how often?
3. What does your backup process look like, both in terms of downtime and
recovery?
4. How large are your backups and where do you keep them? How long do you store
backups for?
5. What ways do you test your backups and have you ever had to restore from a
backup?
6. What risk factors do you consider when making backups which influence your
backup strategy? For example, if you don't follow the 3-2-1 principle (3 copies
on at least 2 types of media and 1 offsite), explain why?
I'll go first! I'll preface by saying this is not the most practical or ideal
and I'd love to find a cleaner approach.
1. I take tarballs of my rootfs (excluding home and some distro specific
non-essential files) using a bash script, compress them with gzip, and encrypt
with gnupg. I store the sha256sum hashes of the tar, tgz, and final tgz.gpg
file in a clear-signed text file and upload both to my NAS with syncs with
rsync.net.
2. I backup 2 systems; my laptop and PC whenever I decide to do a full system
update.
3. I reboot and pick the recovery option in Grub that drops me into an
initramfs shell for maintenance (which is the easiest method because I have
secureboot enabled). I mount the filesystem and run my script from the root
directory to create a tar first since it's faster. This takes about 5-10min,
then I continue booting and gzip -k9 while I use the system as normal which can
take about 20min to compress. Then I encrypt and upload the signed hashes to my
NAS. Total "felt" downtime is about 5-10 min but the whole process is more like
30min but I'm flexible since these are personal devices. I use a helper script
that walks me through building partitions, LUKS disk encryption, and unpacking
the tarball which can be done in about 15min. I would then need to checkout my
"dotfiles" from github for my home directory and mount any personal files I
need from my NAS (I try not to store anything important locally to avoid
syncing files between the 2 devices I use).
4. My OS backups for my laptop are 8-9gb as are the NAS archives of my
important files. I store backups for a couple months at a time before manually
removing whenever I feel like it (I should really update my cron script for
this!).
5. I don't really test my backups if I'm being honest which bit me in the butt
when I had to restore once and realize all my backups were missing a vital
system directory I removed to save space. I ended up reinstalling the OS and
unpacking configs and files from the archive manually (which wasn't bad since I
did a better install job this time). My backup script reuses UUIDs from
/etc/fstab when building the partitions again so testing is sometimes a pain
because I need to remove the current hardrive when I want to test the recovery
using a spare drive (this is a weakness of my process).
6. My risk factors are me doing dumb things and breaking the OS and drive
failure. I prioritize convenience, storage space, and security which is why I
like using encrypted rootfs ("stage 4") tarballs. I like having a discrete
snapshot of files rather than storing only the deltas of data so I can recover
individual files and directories if necessary. I don't perfectly follow 3-2-1
but I keep the latest tarball on my laptop (ssd) and my NAS uses HDDs in raid10
for availability (side note: RAID is great for laziness since all you need to
do is hotswap the dead drives plus it lowers risk enough to source used SAS
drives from ebay for cheap!). My offsite is rysnc.net because it cheap and easy
to automate. I also have a tiny vhdx Debian image (with disk encrypted of
course) on a Ventoy multiboot USB with various live OS images. I keep it on my
keychain and it has all my private keys and SSH configured to access my rsync
backups as a last resort for disaster cases (housefire).
I'd love to hear what you all do because I feel like my process is a bit crude
and I'm open to tips. Plus I'm sure there's a lot of different methods so it'll
be fun learning about each one (especially "why"). Thanks in advance for your
responses!
Health and Peace,
Elaina Jones
