Re: [gpfsug-discuss] Spectrum Protect and disk pools

[Alec.Effrat]

I am not sure what platform you run on but for AIX with a fully virtualized 
LPAR we needed to enable "mtu_bypass" on the en device that was used for our 
backups.  Prior to this setting we could not exceed 250 MB/s on our 10G 
interface, after that we run at 1.6GB/s solid per 10G virtual adapter, fueled 
by Spectrum Scale and a different backup engine.   

We did lose a lot of sleep trying to figure this one out, but are very pleased 
with the end result.

Alec Effrat
SAS Lead, AVP
Business Intelligence Competency Center
SAS Administration
Cell 949-246-7713
alec.eff...@wellsfargo.com

-Original Message-
From: gpfsug-discuss-boun...@spectrumscale.org 
 On Behalf Of Jonathan Buzzard
Sent: Monday, January 4, 2021 8:24 AM
To: gpfsug-discuss@spectrumscale.org
Subject: Re: [gpfsug-discuss] Spectrum Protect and disk pools

On 04/01/2021 12:21, Simon Thompson wrote:
> CAUTION: This email originated outside the University. Check before 
> clicking links or attachments.
> 
> Hi All,
> 
> We use Spectrum Protect (TSM) to backup our Scale filesystems. We have 
> the backup setup to use multiple nodes with the PROXY node function 
> turned on (and to some extent also use multiple target servers).
> 
> This all feels like it is nice and parallel, on the TSM servers, we 
> have disk pools for any “small” files to drop into (I think we set 
> anything smaller than 20GB) to prevent lots of small files stalling 
> tape drive writes.
> 
> Whilst digging into why we have slow backups at times, we found that 
> the disk pool empties with a single thread (one drive). And looking at the 
> docs:
> 
> https://www.ibm.com/support/pages/concurrent-migration-processes-and-c
> onstraints 
> <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww
> .ibm.com%2Fsupport%2Fpages%2Fconcurrent-migration-processes-and-constr
> aints=04%7C01%7Cjonathan.buzzard%40strath.ac.uk%7C99158004dad04c7
> 9a58808d8b0ab39b8%7C631e0763153347eba5cd0457bee5944e%7C0%7C0%7C6374535
> 96745356438%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMz
> IiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=ZPUkTB5Vy5S0%2BL67neMp4C
> 1lxIuphMS5HuTkBYcmcMU%3D=0>
> 
> This implies that we are limited to the number of client nodes stored 
> in the pool. i.e. because we have one node and PROXY nodes, we are 
> essentially limited to a single thread streaming out of the disk pool 
> when full.
> 
> Have we understood this correctly as if so, this appears to make the 
> whole purpose of PROXY nodes sort of pointless if you have lots of 
> small files. Or is there some other setting we should be looking at to 
> increase the number of threads when the disk pool is emptying? (The 
> disk pool itself has Migration Processes: 6)
> 

I have found in the past that the speed of the disk pool can make a large 
difference. That is a RAID5/6 of 7200RPM drives was inadequate and there was a 
significant boost in backup in moving to 15k RPM disks. Also your DB really 
needs to be on SSD, again this affords a large boost in backup speed.

The other rule of thumb I have always worked with is that the disk pool should 
be sized for the daily churn. That is your backup should disappear into the 
disk pool and then when the backup is finished you can then spit the disk pool 
out to the primary and copy pools.

If you are needing to drain the disk pool mid backup your disk pool is too 
small.

TL;DR your TSM disks (DB and disk pool) need to be some of the best storage you 
have to maximize backup speed.

JAB.

-- 
Jonathan A. Buzzard Tel: +44141-5483420
HPC System Administrator, ARCHIE-WeSt.
University of Strathclyde, John Anderson Building, Glasgow. G4 0NG 
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] Spectrum Protect and disk pools

[Skylar Thompson]

I think the collocation settings of the target pool for the migration come
into play as well. If you have multiple filespaces associated with a node
and collocation is set to FILESPACE, then you should be able to get one
migration process per filespace rather than one per node/collocation group.

On Mon, Jan 04, 2021 at 12:21:05PM +, Simon Thompson wrote:
> Hi All,
> 
> We use Spectrum Protect (TSM) to backup our Scale filesystems. We have the 
> backup setup to use multiple nodes with the PROXY node function turned on 
> (and to some extent also use multiple target servers).
> 
> This all feels like it is nice and parallel, on the TSM servers, we have disk 
> pools for any ???small??? files to drop into (I think we set anything smaller 
> than 20GB) to prevent lots of small files stalling tape drive writes.
> 
> Whilst digging into why we have slow backups at times, we found that the disk 
> pool empties with a single thread (one drive). And looking at the docs:
> https://www.ibm.com/support/pages/concurrent-migration-processes-and-constraints
> 
> This implies that we are limited to the number of client nodes stored in the 
> pool. i.e. because we have one node and PROXY nodes, we are essentially 
> limited to a single thread streaming out of the disk pool when full.
> 
> Have we understood this correctly as if so, this appears to make the whole 
> purpose of PROXY nodes sort of pointless if you have lots of small files. Or 
> is there some other setting we should be looking at to increase the number of 
> threads when the disk pool is emptying? (The disk pool itself has Migration 
> Processes: 6)
> 
> Thanks
> 
> Simon

> ___
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss


-- 
-- Skylar Thompson (skyl...@u.washington.edu)
-- Genome Sciences Department (UW Medicine), System Administrator
-- Foege Building S046, (206)-685-7354
-- Pronouns: He/Him/His
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] Spectrum Protect and disk pools

[Simon Thompson]

Hi Jordi,

Thanks, yes it is a disk pool:

Protect: TSM01>q stg BACKUP_DISK f=d

 Storage Pool Name: BACKUP_DISK
 Storage Pool Type: Primary
 Device Class Name: DISK
 Storage Type: DEVCLASS
…
 Next Storage Pool: BACKUP_ONSTAPE

So it is a disk pool … though it is made up of multiple disk files …

/tsmdisk/stgpool/tsmins- BACKUP_DISK DISK 200.0 G   0.0 
On-Line
t3/bkup_diskvol01.dsm
/tsmdisk/stgpool/tsmins- BACKUP_DISK DISK 200.0 G   0.0 
On-Line
t3/bkup_diskvol02.dsm
/tsmdisk/stgpool/tsmins- BACKUP_DISK DISK 200.0 G   0.0 
On-Line
t3/bkup_diskvol03.dsm

Will look into the FILE pool as this sounds like it might be less single 
threaded than now 

Thanks

Simon

From:  on behalf of 
"jordi.cau...@es.ibm.com" 
Reply to: "gpfsug-discuss@spectrumscale.org" 
Date: Monday, 4 January 2021 at 13:36
To: "gpfsug-discuss@spectrumscale.org" 
Subject: Re: [gpfsug-discuss] Spectrum Protect and disk pools

Simon,

which kind of storage pool are you using, DISK or FILE ? I understand DISK pool 
from your mail. DISK pool does not behave the same as FILE pool.

DISK pool is limited by the number of nodes or MIGProcess setting (the minimum 
of both) as the document states. Using proxy helps you backup in parallel from 
multiple nodes to the stg pool but from Protect perspective it is a single 
node. Even multiple nodes are sending they run "asnodename" so single node from 
Protect perspective.

If using FILE pool, you can define the number of volumes within the FILE pool 
and when migrating to tape, it will migrate each volume in parallel with the 
limit of MIGProcess setting. So it would be the minimum of #volumes and 
MIGProcess value.

I know more deep technical skills in Protect are on this mailing list so feel 
free to add something or correct me.

Best Regards,
--
Jordi Caubet Serrabou
IBM Storage Client Technical Specialist (IBM Spain)
Ext. Phone: (+34) 679.79.17.84 (internal 55834)
E-mail: jordi.cau...@es.ibm.com<mailto:jordi.cau...@es.ibm.com>


-gpfsug-discuss-boun...@spectrumscale.org<mailto:-gpfsug-discuss-boun...@spectrumscale.org>
 wrote: -
To: "gpfsug-discuss@spectrumscale.org<mailto:gpfsug-discuss@spectrumscale.org>" 
mailto:gpfsug-discuss@spectrumscale.org>>
From: Simon Thompson
Sent by: 
gpfsug-discuss-boun...@spectrumscale.org<mailto:gpfsug-discuss-boun...@spectrumscale.org>
Date: 01/04/2021 01:21PM
Subject: [EXTERNAL] [gpfsug-discuss] Spectrum Protect and disk pools


Hi All,


We use Spectrum Protect (TSM) to backup our Scale filesystems. We have the 
backup setup to use multiple nodes with the PROXY node function turned on (and 
to some extent also use multiple target servers).


This all feels like it is nice and parallel, on the TSM servers, we have disk 
pools for any “small” files to drop into (I think we set anything smaller than 
20GB) to prevent lots of small files stalling tape drive writes.


Whilst digging into why we have slow backups at times, we found that the disk 
pool empties with a single thread (one drive). And looking at the docs:
https://www.ibm.com/support/pages/concurrent-migration-processes-and-constraints


This implies that we are limited to the number of client nodes stored in the 
pool. i.e. because we have one node and PROXY nodes, we are essentially limited 
to a single thread streaming out of the disk pool when full.


Have we understood this correctly as if so, this appears to make the whole 
purpose of PROXY nodes sort of pointless if you have lots of small files. Or is 
there some other setting we should be looking at to increase the number of 
threads when the disk pool is emptying? (The disk pool itself has Migration 
Processes: 6)


Thanks


Simon
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Salvo indicado de otro modo más arriba / Unless stated otherwise above:
International Business Machines, S.A.
Santa Hortensia, 26-28, 28002 Madrid
Registro Mercantil de Madrid; Folio 1; Tomo 1525; Hoja M-28146
CIF A28-010791
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] Spectrum Protect and disk pools

[Jordi Caubet Serrabou]

Simon,which kind of storage pool are you using, DISK or FILE ? I understand DISK pool from your mail. DISK pool does not behave the same as FILE pool.DISK pool is limited by the number of nodes or MIGProcess setting (the minimum of both) as the document states. Using proxy helps you backup in parallel from multiple nodes to the stg pool but from Protect perspective it is a single node. Even multiple nodes are sending they run "asnodename" so single node from Protect perspective.If using FILE pool, you can define the number of volumes within the FILE pool and when migrating to tape, it will migrate each volume in parallel with the limit of MIGProcess setting. So it would be the minimum of #volumes and MIGProcess value.I know more deep technical skills in Protect are on this mailing list so feel free to add something or correct me.Best Regards,--Jordi Caubet SerrabouIBM Storage Client Technical Specialist (IBM Spain)Ext. Phone: (+34) 679.79.17.84 (internal 55834)E-mail: jordi.cau...@es.ibm.com-gpfsug-discuss-boun...@spectrumscale.org wrote: -To: "gpfsug-discuss@spectrumscale.org" From: Simon Thompson Sent by: gpfsug-discuss-boun...@spectrumscale.orgDate: 01/04/2021 01:21PMSubject: [EXTERNAL] [gpfsug-discuss] Spectrum Protect and disk poolsHi All,
 
We use Spectrum Protect (TSM) to backup our Scale filesystems. We have the backup setup to use multiple nodes with the PROXY node function turned on (and to some extent also use multiple target servers).
 
This all feels like it is nice and parallel, on the TSM servers, we have disk pools for any “small” files to drop into (I think we set anything smaller than 20GB) to prevent lots of small files stalling tape
 drive writes.
 
Whilst digging into why we have slow backups at times, we found that the disk pool empties with a single thread (one drive). And looking at the docs:
https://www.ibm.com/support/pages/concurrent-migration-processes-and-constraints
 
This implies that we are limited to the number of client nodes stored in the pool. i.e. because we have one node and PROXY nodes, we are essentially limited to a single thread streaming out of the disk pool
 when full.
 
Have we understood this correctly as if so, this appears to make the whole purpose of PROXY nodes sort of pointless if you have lots of small files. Or is there some other setting we should be looking at to
 increase the number of threads when the disk pool is emptying? (The disk pool itself has Migration Processes: 6)
 
Thanks
 
Simon___gpfsug-discuss mailing listgpfsug-discuss at spectrumscale.orghttp://gpfsug.org/mailman/listinfo/gpfsug-discuss Salvo indicado de otro modo más arriba / Unless stated otherwise above:International Business Machines, S.A.Santa Hortensia, 26-28, 28002 MadridRegistro Mercantil de Madrid; Folio 1; Tomo 1525; Hoja M-28146CIF A28-010791

___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss