Also Sprach David Simpson:
Hi,
That's interesting.
What is the motivation/main benefits of the Panasas system? (what else did you
consider)
PanFS has multi-tier storage with metadata going to NVMe, IOPS-bound small
files to SATA
SSD, and regular files to hard disk. Also seems a bit easier to manage and
less pricey
compared to GPFS/Spectrum Scale and WekaIO.
Yes inode trawling becoming a big issue.
----
Am wondering if it might be possible to [sensibly] do full (e.g. monthly) to
tape and incremental to disk (e.g. weekly).
David
You don't really need Amanda for that, since its major feature is the scheduler
which
mixes incrementals and full backups to maximize efficient tape usage.
________________________________
From: C. Chan <[email protected]>
Sent: 24 May 2021 17:06
To: David Simpson <[email protected]>; [email protected]
<[email protected]>
Subject: Re: scale of backup and frequency
External email to Cardiff University - Take care when replying/opening
attachments or links.
Nid ebost mewnol o Brifysgol Caerdydd yw hwn - Cymerwch ofal wrth ateb/agor
atodiadau neu ddolenni.
FYI:
i. Three different backup sets, each around 100TB and growing.
ii. Fri evening - Mon morning weekend backups, rotated quarterly/every 12 weeks.
Filesystem snapshots done hourly, rotated each day, and daily snapshots,
rotated
each week, in place of daily incremental backups.
iii. The two biggest challenges are the explosion of the total volume of data
and the increase in IOPS limited small file I/O, both mostly due to deep
learning.
We are currently using NAS servers with ZFS special devices to store
small files
on NVMe SSD drives, but are planning to migrate/consolidate to a Panasas
system.
Also Sprach David Simpson:
Hi all,
Am interested to hear about your Amanda setup. Particularly if you are dealing
with an HPC home file system/server, where both size and churn can be issues.
And/or scalable storage.
i) How big is your regular backup?
ii) How frequent?
iii) What challenges do you have/face?
I'm currently in the process of thinking about how a new backup regime will
look, armed with a new 40 slot tape library and some limited disk storage too.
thanks
David
--
C. Chan <c-chan at uchicago.edu>
GPG Public Key registered at pgp.mit.edu
--
C. Chan <c-chan at uchicago.edu>
GPG Public Key registered at pgp.mit.edu
