On 04/01/2021 12:21, Simon Thompson wrote:
CAUTION: This email originated outside the University. Check before
clicking links or attachments.
Hi All,
We use Spectrum Protect (TSM) to backup our Scale filesystems. We have
the backup setup to use multiple nodes with the PROXY node function
turned on (and to some extent also use multiple target servers).
This all feels like it is nice and parallel, on the TSM servers, we have
disk pools for any “small” files to drop into (I think we set anything
smaller than 20GB) to prevent lots of small files stalling tape drive
writes.
Whilst digging into why we have slow backups at times, we found that the
disk pool empties with a single thread (one drive). And looking at the docs:
https://www.ibm.com/support/pages/concurrent-migration-processes-and-constraints
<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ibm.com%2Fsupport%2Fpages%2Fconcurrent-migration-processes-and-constraints&data=04%7C01%7Cjonathan.buzzard%40strath.ac.uk%7C99158004dad04c79a58808d8b0ab39b8%7C631e0763153347eba5cd0457bee5944e%7C0%7C0%7C637453596745356438%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=ZPUkTB5Vy5S0%2BL67neMp4C1lxIuphMS5HuTkBYcmcMU%3D&reserved=0>
This implies that we are limited to the number of client nodes stored in
the pool. i.e. because we have one node and PROXY nodes, we are
essentially limited to a single thread streaming out of the disk pool
when full.
Have we understood this correctly as if so, this appears to make the
whole purpose of PROXY nodes sort of pointless if you have lots of small
files. Or is there some other setting we should be looking at to
increase the number of threads when the disk pool is emptying? (The disk
pool itself has Migration Processes: 6)
I have found in the past that the speed of the disk pool can make a
large difference. That is a RAID5/6 of 7200RPM drives was inadequate and
there was a significant boost in backup in moving to 15k RPM disks. Also
your DB really needs to be on SSD, again this affords a large boost in
backup speed.
The other rule of thumb I have always worked with is that the disk pool
should be sized for the daily churn. That is your backup should
disappear into the disk pool and then when the backup is finished you
can then spit the disk pool out to the primary and copy pools.
If you are needing to drain the disk pool mid backup your disk pool is
too small.
TL;DR your TSM disks (DB and disk pool) need to be some of the best
storage you have to maximize backup speed.
JAB.
--
Jonathan A. Buzzard Tel: +44141-5483420
HPC System Administrator, ARCHIE-WeSt.
University of Strathclyde, John Anderson Building, Glasgow. G4 0NG
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
