On Wed, 22 Nov 2023 05:25:22 -0300 Crystal Kolipe <kolip...@exoticsilicon.com> wrote: > We have been doing "something similar", in fact much simpler, on > OpenBSD and other unix-like systems for > 25 years. > > It's trivially simple to protect your data, and you along with > 99.999% of other people seem to be over thinking it. > > 1. Once data is no longer "work in progress", archive it to write-only > media and take it out of the regular backup loop. In most cases > this drastically reduces the volume of data you need to manage. Feel > free to keep a local on-line copy on a regular disk too for faster > access. > > 2. Write scripts to copy data that matters elsewhere automatically. > This can be another drive in the local machine, or even another > partition on the same disk. This takes the place of your "RAID-1 or > RAID-5", and is much, much less error-prone because it's just copying > files around. > > 3. Write a script to verify the copy with the original version and > highlight changes. (Ours is 18 lines of shell script.) > > 4. Write a script to create and verify a file of checksums in the > current directory. (Also not complicated - ours is 15 lines of shell > script.) > > We have kept many Tb of data intact and free of bitrot for decades > using simple methods like this. No need for fancy filesystems or > command line parity tools, just use tar, sha256 and crucially a > little bit of intelligence, and the problem is solved. > > And yes, we have certainly seen bitflips when reading from disk, > reading from SSDs, (which overall seem _worse_ for random unreported > bit flipping), and also bad system RAM which causes data in the > buffer cache to be corrupted. All of these threats are easily > mitigated with tar and sha256, and the aforementioned application of > some intelligence to the problem. > > The only problem is that it doesn't have a flashy name like "ZFS".
Thank you for this super interesting answer! I am very much for functional simplicity over complexity one does not understand. Presently, I use a script, utilising rsync, for fast backups and sync diffs, but I'd like a more long-term reliable and checksummed solution. Would you mind sharing the scripts you mentioned for us newbies? Some additional portability rant: Does anyone recommend FAT32-formatted 1 TB external HDDs for OS-portable backups (using archive splitting to bypass the 4 GB limit)? I've heard FAT32 is very inefficient with big partitions. I currently have a mess of ext4 for Linux, ZFS (yes ...) for FreeBSD, and nothing yet for OpenBSD (sadly, my favourite OS does not support redshift on my Nvidia card, and that is a requirement for my eyes).
