On 1/17/24 17:23, Default User wrote:
On Wed, 2024-01-17 at 09:19 -0800, David Christensen wrote:
On 1/17/24 08:19, Default User wrote:
Opinions, please.
...
Hi guys, thanks for the replies.
YW. :-)
BTW, the two backup drives are external 4 Gb USB HDDs. The secondary
backup drive is always kept away from the computer, in a locked steel
box, except when it is attached to the computer to have the primary
backup drive copied to it.
That means the live data and all the backups are exposed to the same
computer if and when it is compromised. It would be safer to do the
copy on another computer. If you only have one computer, then boot live
media to do the copy.
The primary backup drive is almost always attached to the computer, so
that I can access files archived there, that are not on the computer.
Accidental file modification and deletion are probably the most common
failure modes.
One of the features of ZFS is snapshots. Snapshots can be accessed via
the hidden ".zfs/snapshot/" directory in the root of each file system.
Users can browse the snapshots and copy out whatever files and
directories they need, subject to the file system permissions at the
time the snapshot was taken. So, recovery from the above failure is
"self serve" -- no sysadmin required, no backup/ restore software required.
Probably not good practice, but that's why I have the secondary backup
drive.
I guess in the back of my mind I was thinking of a scenario where a
file on the primary backup drive might be corrupted or deleted before
being copied to the secondary backup drive. Then if it is not present
on the primary backup drive, rsync dutifully deletes it from the
secondary backup drive. If the file is no longer on the computer's
internal SSD, I am then SOL.
ZFS snapshots are read-only. Individual files and directories within a
snapshot cannot be created, updated, or deleted.
Another feature of ZFS is that both data and metadata are checksummed.
So, if you can read a file, the metadata and data are good. "bitrot" is
extremely unlikely.
I have also thought of trying to use partclone to copy the data from
the primary backup drive to the secondary backup drive. Why not try,
Transferring 4 TB at 150 MB/s would require:
4 TB / 150 MB/s = 26,667 s
Or, about 7.4 hours.
since rsync takes an hour and a half, every day!
How much data is rsync(1) transferring?
Another feature of ZFS snapshots is that they can be replicated from one
pool to another. The first replication is full. Subsequent
replications are incremental. Here are some recent transfers from my
backup server to a removable HDD:
frequency bytes real time bandwidth
========= ============== =========== =========
weekly 11,125,755,324 25m08.848s 7.37 MB/s
monthly 75,218,102,216 140m53.710s 8.90 MB/s
As for ZFS . . . I wish! But I think the resource requirements would
be too high for my setup, so it probably would be impractical - or
impossible.
ZFS will use as much or as little resources as you provide.
(ZFS is known to consume most of available memory OOTB; this can be
tuned. The simple answer is to put it on a dedicated file server/ NAS.)
Please tell us about your setup -- hardware, stored data, and I/O workload.
And then there's the complexity.
I find ZFS has more conceptual integrity than cobbling together
fdisk(8), mdadm(8), cryptsetup(8), lvm(8), mkfs(8), etc..
And the learning curve.
The Lucas books got me up the learning curve:
https://mwl.io/nonfiction/os#fmzfs
https://mwl.io/nonfiction/os#fmaz
While both have "FreeBSD" in the title, most of the content applies
equally to OpenZFS on Debian.
Finally I really should have a third backup drive in the mix. Yes, I
am familiar with the 1-2-3 backup theory. But the third backup drive
could not be off-site, for various reasons. And I do have other things
to do rather than spend all day, every day, managing backups.
Disaster preparedness is an open-ended problem. Only you can decide how
much solution to give it.
David
