FYI:
i. Three different backup sets, each around 100TB and growing.
ii. Fri evening - Mon morning weekend backups, rotated quarterly/every 12 weeks.
Filesystem snapshots done hourly, rotated each day, and daily snapshots,
rotated
each week, in place of daily incremental backups.
iii. The two biggest challenges are the explosion of the total volume of data
and the increase in IOPS limited small file I/O, both mostly due to deep
learning.
We are currently using NAS servers with ZFS special devices to store
small files
on NVMe SSD drives, but are planning to migrate/consolidate to a Panasas
system.
Also Sprach David Simpson:
Hi all,
Am interested to hear about your Amanda setup. Particularly if you are dealing
with an HPC home file system/server, where both size and churn can be issues.
And/or scalable storage.
i) How big is your regular backup?
ii) How frequent?
iii) What challenges do you have/face?
I'm currently in the process of thinking about how a new backup regime will
look, armed with a new 40 slot tape library and some limited disk storage too.
thanks
David
--
C. Chan <c-chan at uchicago.edu>
GPG Public Key registered at pgp.mit.edu
