On Sun, 2 Aug 2026, elainatj--- via PLUG wrote:
1. What tools are you using for OS backups and what's your process?
We use Bacula to back up both Linux and Windows systems, but -- and
this is an important point -- we backup system data only; our backups
don't provide full DR recovery. A full system failure would involve
installing the OS from scratch, running Puppet (if it's a Linux
system) or whatever sysconfig tool the Windows folks are using these
days, and then restore data from the Bacula archive.
2. How many systems do you backup and how often?
We backup about 100 systems: full backups one weekend a month,
differentials the other weekends, and incrementals nightly.
3. What does your backup process look like, both in terms of
downtime and recovery?
I assume you mean "recovery process." There is no downtime for
backups. See above for recovery.
4. How large are your backups and where do you keep them? How long
do you store backups for?
Our standard system backups have two different storage targets: full
and differential jobs go to an out-of-region Ceph/S3 cluster, nightly
incrementals go to local disk on the backup server.
Full backups are kept for one year; differentials and incrementals for
35 days. Weekend Full+Differential jobs run ca. 16 TB weekly. We only
started using Ceph/S3 at the very end of 2025, so our aggregate data
store will continue to grow for a while, but right now Ceph holds
about 370 TB.
We also have a couple customers with non-standard configurations;
backups for them have run 100 to 200 TB. But these configurations
relied on our tape library, which is being retired. My suspicion is
that we'll move similar jobs in the future to AWS Glacier Deep
Archive, but that's still being ironed out.
5. What ways do you test your backups and have you ever had to
restore from a backup?
I do periodic testing of random files, but we typically receive three
to fix requests per year to restore files, a process that is much
simpler using Ceph/S3 than it was using tape.
6. What risk factors do you consider when making backups which
influence your backup strategy? For example, if you don't follow
the 3-2-1 principle (3 copies on at least 2 types of media and 1
offsite), explain why?
We are in the middle of retiring our long-suffering LTO6/7 tape
system, so we essentially only backup to disk. I don't know if that
will remain our long-term strategy.
We tell our customers that backups server two purposes, which are not
mutually exclusive. They can protect against
* human failure: faulty edits, mistaken deletions, etc.
* hardware failure: system death, etc
Protection against human failure is typically handled by snapshots on
our network-shared file systems, which is where we advise customers to
store working files. This is a self-help system, which is nice.
Files not stored on systems with snapshots, or files living on
hardware that fails, are recovered from Bacula.
--
Paul Heinlein
[email protected]
45°22'48" N, 122°35'36" W
