Re: [gpfsug-discuss] dependent versus independent filesets
as long as your currently do not need more than 1000 on a filesystem On Wed, 8 Jul 2020 at 04:20, Daniel Kidger wrote: > It is worth noting that Independent Filesets are a relatively recent > addition to Spectrum Scale, compared to Dependant Filesets. They havesolved > some of the limitations of the former. > > > My view would be to always use Independent FIlesets unless there is a > particular reason to use Dependant ones. > > Daniel > > _ > *Daniel Kidger Ph.D.* > IBM Technical Sales Specialist > Spectrum Scale, Spectrum Discover and IBM Cloud Object Store > > +44-(0)7818 522 266 > daniel.kid...@uk.ibm.com > > <https://www.youracclaim.com/badges/687cf790-fe65-4a92-b129-d23ae41862ac/public_url> > <https://www.youracclaim.com/badges/8153c6a7-3e02-40be-87ee-24e27ae9459c/public_url> > <https://www.youracclaim.com/badges/78197e2c-4277-4ec9-808b-ad6abe1e1b16/public_url> > > > > > - Original message - > From: "Frederick Stock" > Sent by: gpfsug-discuss-boun...@spectrumscale.org > To: gpfsug-discuss@spectrumscale.org > Cc: gpfsug-discuss@spectrumscale.org > Subject: [EXTERNAL] Re: [gpfsug-discuss] dependent versus independent > filesets > Date: Tue, Jul 7, 2020 17:25 > > One comment about inode preallocation. There was a time when inode > creation was performance challenged but in my opinion that is no longer the > case, unless you have need for file creates to complete at extreme speed. > In my experience it is the rare customer that requires extremely fast file > create times so pre-allocation is not truly necessary. As was noted once > an inode is allocated it cannot be deallocated. The more important item is > the maximum inodes defined for a fileset or file system. Yes, those do > need to be monitored so they can be increased if necessary to avoid out of > space errors. > > Fred > __ > Fred Stock | IBM Pittsburgh Lab | 720-430-8821 > sto...@us.ibm.com > > > > ----- Original message - > From: "Wahl, Edward" > Sent by: gpfsug-discuss-boun...@spectrumscale.org > To: gpfsug main discussion list > Cc: > Subject: [EXTERNAL] Re: [gpfsug-discuss] dependent versus independent > filesets > Date: Tue, Jul 7, 2020 11:59 AM > > We also went with independent filesets for both backup (and quota) reasons > for several years now, and have stuck with this across to 5.x. However we > still maintain a minor number of dependent filesets for administrative use. > Being able to mmbackup on many filesets at once can increase your > parallelization _quite_ nicely! We create and delete the individual snaps > before and after each backup, as you may expect. Just be aware that if you > do massive numbers of fast snapshot deletes and creates you WILL reach a > point where you will run into issues due to quiescing compute clients, and > that certain types of workloads have issues with snapshotting in general. > > You have to more closely watch what you pre-allocate, and what you have > left in the common metadata/inode pool. Once allocated, even if not being > used, you cannot reduce the inode allocation without removing the fileset > and re-creating. (say a fileset user had 5 million inodes and now only > needs 500,000) > > Growth can also be an issue if you do NOT fully pre-allocate each space. > This can be scary if you are not used to over-subscription in general. But > I imagine that most sites have some decent % of oversubscription if they > use filesets and quotas. > > Ed > OSC > > -Original Message- > From: gpfsug-discuss-boun...@spectrumscale.org < > gpfsug-discuss-boun...@spectrumscale.org> On Behalf Of Skylar Thompson > Sent: Tuesday, July 7, 2020 10:00 AM > To: gpfsug-discuss@spectrumscale.org > Subject: Re: [gpfsug-discuss] dependent versus independent filesets > > We wanted to be able to snapshot and backup filesets separately with > mmbackup, so went with independent filesets. > > On Tue, Jul 07, 2020 at 08:37:46AM -0500, Damir Krstic wrote: > > We are deploying our new ESS and are considering moving to independent > > filesets. The snapshot per fileset feature appeals to us. > > > > Has anyone considered independent vs. dependent filesets and what was > > your reasoning to go with one as opposed to the other? Or perhaps you > > opted to have both on your filesystem, and if, what was the reasoning > for it? > > > > Thank you. > > Damir > > > ___ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > https:/
Re: [gpfsug-discuss] dependent versus independent filesets
It is worth noting that Independent Filesets are a relatively recent addition to Spectrum Scale, compared to Dependant Filesets. They havesolved some of the limitations of the former. My view would be to always use Independent FIlesets unless there is a particular reason to use Dependant ones. Daniel _Daniel Kidger Ph.D. IBM Technical Sales Specialist Spectrum Scale, Spectrum Discover and IBM Cloud Object Store+44-(0)7818 522 266 daniel.kid...@uk.ibm.com - Original message -From: "Frederick Stock" Sent by: gpfsug-discuss-boun...@spectrumscale.orgTo: gpfsug-discuss@spectrumscale.orgCc: gpfsug-discuss@spectrumscale.orgSubject: [EXTERNAL] Re: [gpfsug-discuss] dependent versus independent filesetsDate: Tue, Jul 7, 2020 17:25 One comment about inode preallocation. There was a time when inode creation was performance challenged but in my opinion that is no longer the case, unless you have need for file creates to complete at extreme speed. In my experience it is the rare customer that requires extremely fast file create times so pre-allocation is not truly necessary. As was noted once an inode is allocated it cannot be deallocated. The more important item is the maximum inodes defined for a fileset or file system. Yes, those do need to be monitored so they can be increased if necessary to avoid out of space errors. Fred__Fred Stock | IBM Pittsburgh Lab | 720-430-8821sto...@us.ibm.com - Original message -From: "Wahl, Edward" Sent by: gpfsug-discuss-boun...@spectrumscale.orgTo: gpfsug main discussion list Cc:Subject: [EXTERNAL] Re: [gpfsug-discuss] dependent versus independent filesetsDate: Tue, Jul 7, 2020 11:59 AM We also went with independent filesets for both backup (and quota) reasons for several years now, and have stuck with this across to 5.x. However we still maintain a minor number of dependent filesets for administrative use. Being able to mmbackup on many filesets at once can increase your parallelization _quite_ nicely! We create and delete the individual snaps before and after each backup, as you may expect. Just be aware that if you do massive numbers of fast snapshot deletes and creates you WILL reach a point where you will run into issues due to quiescing compute clients, and that certain types of workloads have issues with snapshotting in general.You have to more closely watch what you pre-allocate, and what you have left in the common metadata/inode pool. Once allocated, even if not being used, you cannot reduce the inode allocation without removing the fileset and re-creating. (say a fileset user had 5 million inodes and now only needs 500,000) Growth can also be an issue if you do NOT fully pre-allocate each space. This can be scary if you are not used to over-subscription in general. But I imagine that most sites have some decent % of oversubscription if they use filesets and quotas.EdOSC-Original Message-From: gpfsug-discuss-boun...@spectrumscale.org On Behalf Of Skylar ThompsonSent: Tuesday, July 7, 2020 10:00 AMTo: gpfsug-discuss@spectrumscale.orgSubject: Re: [gpfsug-discuss] dependent versus independent filesetsWe wanted to be able to snapshot and backup filesets separately with mmbackup, so went with independent filesets.On Tue, Jul 07, 2020 at 08:37:46AM -0500, Damir Krstic wrote:> We are deploying our new ESS and are considering moving to independent> filesets. The snapshot per fileset feature appeals to us.>> Has anyone considered independent vs. dependent filesets and what was> your reasoning to go with one as opposed to the other? Or perhaps you> opted to have both on your filesystem, and if, what was the reasoning for it?>> Thank you.> Damir> ___> gpfsug-discuss mailing list> gpfsug-discuss at spectrumscale.org> https://urldefense.com/v3/__http://gpfsug.org/mailman/listinfo/gpfsug-> discuss__;!!KGKeukY!j-c9kslUrEaNslhTbLLfaY8TES7Xf4eUCxysOaXwroHhTMwiVY> vcGNh4M_no$ Skylar Thompson (skyl...@u.washington.edu)-- Genome Sciences Department (UW Medicine), System Administrator-- Foege Building S046, (206)-685-7354-- Pronouns: He/Him/His___gpfsug-discuss mailing listgpfsug-discuss at spectrumscale.orghttps://urldefense.com/v3/__http://gpfsug.org/mailman/listinfo/gpfsug-discuss__;!!KGKeukY!j-c9kslUrEaNslhTbLLfaY8TES7Xf4eUCxysOaXwroHhTMwiVYvcGNh4M_no$ ___gpfsug-discuss mailing listgpfsug-discuss at spectrumscale.orghttp://gpfsug.org/mailman/listinfo/gpfsug-discuss ___gpfsug-discuss mailing listgpfsug-discuss at spectrumscale.orghttp://gpfsug.org/mailman/listinfo/gpfsug-discuss Unless stated otherwise above: IBM United
Re: [gpfsug-discuss] dependent versus independent filesets
One comment about inode preallocation. There was a time when inode creation was performance challenged but in my opinion that is no longer the case, unless you have need for file creates to complete at extreme speed. In my experience it is the rare customer that requires extremely fast file create times so pre-allocation is not truly necessary. As was noted once an inode is allocated it cannot be deallocated. The more important item is the maximum inodes defined for a fileset or file system. Yes, those do need to be monitored so they can be increased if necessary to avoid out of space errors. Fred__Fred Stock | IBM Pittsburgh Lab | 720-430-8821sto...@us.ibm.com - Original message -From: "Wahl, Edward" Sent by: gpfsug-discuss-boun...@spectrumscale.orgTo: gpfsug main discussion list Cc:Subject: [EXTERNAL] Re: [gpfsug-discuss] dependent versus independent filesetsDate: Tue, Jul 7, 2020 11:59 AM We also went with independent filesets for both backup (and quota) reasons for several years now, and have stuck with this across to 5.x. However we still maintain a minor number of dependent filesets for administrative use. Being able to mmbackup on many filesets at once can increase your parallelization _quite_ nicely! We create and delete the individual snaps before and after each backup, as you may expect. Just be aware that if you do massive numbers of fast snapshot deletes and creates you WILL reach a point where you will run into issues due to quiescing compute clients, and that certain types of workloads have issues with snapshotting in general.You have to more closely watch what you pre-allocate, and what you have left in the common metadata/inode pool. Once allocated, even if not being used, you cannot reduce the inode allocation without removing the fileset and re-creating. (say a fileset user had 5 million inodes and now only needs 500,000) Growth can also be an issue if you do NOT fully pre-allocate each space. This can be scary if you are not used to over-subscription in general. But I imagine that most sites have some decent % of oversubscription if they use filesets and quotas.EdOSC-Original Message-From: gpfsug-discuss-boun...@spectrumscale.org On Behalf Of Skylar ThompsonSent: Tuesday, July 7, 2020 10:00 AMTo: gpfsug-discuss@spectrumscale.orgSubject: Re: [gpfsug-discuss] dependent versus independent filesetsWe wanted to be able to snapshot and backup filesets separately with mmbackup, so went with independent filesets.On Tue, Jul 07, 2020 at 08:37:46AM -0500, Damir Krstic wrote:> We are deploying our new ESS and are considering moving to independent> filesets. The snapshot per fileset feature appeals to us.>> Has anyone considered independent vs. dependent filesets and what was> your reasoning to go with one as opposed to the other? Or perhaps you> opted to have both on your filesystem, and if, what was the reasoning for it?>> Thank you.> Damir> ___> gpfsug-discuss mailing list> gpfsug-discuss at spectrumscale.org> https://urldefense.com/v3/__http://gpfsug.org/mailman/listinfo/gpfsug-> discuss__;!!KGKeukY!j-c9kslUrEaNslhTbLLfaY8TES7Xf4eUCxysOaXwroHhTMwiVY> vcGNh4M_no$ Skylar Thompson (skyl...@u.washington.edu)-- Genome Sciences Department (UW Medicine), System Administrator-- Foege Building S046, (206)-685-7354-- Pronouns: He/Him/His___gpfsug-discuss mailing listgpfsug-discuss at spectrumscale.orghttps://urldefense.com/v3/__http://gpfsug.org/mailman/listinfo/gpfsug-discuss__;!!KGKeukY!j-c9kslUrEaNslhTbLLfaY8TES7Xf4eUCxysOaXwroHhTMwiVYvcGNh4M_no$ ___gpfsug-discuss mailing listgpfsug-discuss at spectrumscale.orghttp://gpfsug.org/mailman/listinfo/gpfsug-discuss ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] dependent versus independent filesets
Ah, yes, I forgot about the quota rationale; we use independent filesets for that as well. We have run into confusion with inodes as one has to be careful to allocate inodes /and/ adjust a quota to expand a fileset. IIRC GPFS generates ENOSPC if it actually runs out of inodes, and EDQUOT if it hits a quota. We've also run into the quiescing issue but have been able to workaround it for now by increasing the splay between the different schedules. On Tue, Jul 07, 2020 at 02:44:16PM +, Wahl, Edward wrote: > We also went with independent filesets for both backup (and quota) reasons > for several years now, and have stuck with this across to 5.x. However we > still maintain a minor number of dependent filesets for administrative use. > Being able to mmbackup on many filesets at once can increase your > parallelization _quite_ nicely! We create and delete the individual snaps > before and after each backup, as you may expect. Just be aware that if you > do massive numbers of fast snapshot deletes and creates you WILL reach a > point where you will run into issues due to quiescing compute clients, and > that certain types of workloads have issues with snapshotting in general. > > You have to more closely watch what you pre-allocate, and what you have left > in the common metadata/inode pool. Once allocated, even if not being used, > you cannot reduce the inode allocation without removing the fileset and > re-creating. (say a fileset user had 5 million inodes and now only needs > 500,000) > > Growth can also be an issue if you do NOT fully pre-allocate each space. > This can be scary if you are not used to over-subscription in general. But I > imagine that most sites have some decent % of oversubscription if they use > filesets and quotas. > > Ed > OSC > > -Original Message- > From: gpfsug-discuss-boun...@spectrumscale.org > On Behalf Of Skylar Thompson > Sent: Tuesday, July 7, 2020 10:00 AM > To: gpfsug-discuss@spectrumscale.org > Subject: Re: [gpfsug-discuss] dependent versus independent filesets > > We wanted to be able to snapshot and backup filesets separately with > mmbackup, so went with independent filesets. > > On Tue, Jul 07, 2020 at 08:37:46AM -0500, Damir Krstic wrote: > > We are deploying our new ESS and are considering moving to independent > > filesets. The snapshot per fileset feature appeals to us. > > > > Has anyone considered independent vs. dependent filesets and what was > > your reasoning to go with one as opposed to the other? Or perhaps you > > opted to have both on your filesystem, and if, what was the reasoning for > > it? > > > > Thank you. > > Damir > > > ___ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > https://urldefense.com/v3/__http://gpfsug.org/mailman/listinfo/gpfsug- > > discuss__;!!KGKeukY!j-c9kslUrEaNslhTbLLfaY8TES7Xf4eUCxysOaXwroHhTMwiVY > > vcGNh4M_no$ > > > -- > -- Skylar Thompson (skyl...@u.washington.edu) > -- Genome Sciences Department (UW Medicine), System Administrator > -- Foege Building S046, (206)-685-7354 > -- Pronouns: He/Him/His > ___ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.com/v3/__http://gpfsug.org/mailman/listinfo/gpfsug-discuss__;!!KGKeukY!j-c9kslUrEaNslhTbLLfaY8TES7Xf4eUCxysOaXwroHhTMwiVYvcGNh4M_no$ > > ___ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- -- Skylar Thompson (skyl...@u.washington.edu) -- Genome Sciences Department (UW Medicine), System Administrator -- Foege Building S046, (206)-685-7354 -- Pronouns: He/Him/His ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] dependent versus independent filesets
We also went with independent filesets for both backup (and quota) reasons for several years now, and have stuck with this across to 5.x. However we still maintain a minor number of dependent filesets for administrative use. Being able to mmbackup on many filesets at once can increase your parallelization _quite_ nicely! We create and delete the individual snaps before and after each backup, as you may expect. Just be aware that if you do massive numbers of fast snapshot deletes and creates you WILL reach a point where you will run into issues due to quiescing compute clients, and that certain types of workloads have issues with snapshotting in general. You have to more closely watch what you pre-allocate, and what you have left in the common metadata/inode pool. Once allocated, even if not being used, you cannot reduce the inode allocation without removing the fileset and re-creating. (say a fileset user had 5 million inodes and now only needs 500,000) Growth can also be an issue if you do NOT fully pre-allocate each space. This can be scary if you are not used to over-subscription in general. But I imagine that most sites have some decent % of oversubscription if they use filesets and quotas. Ed OSC -Original Message- From: gpfsug-discuss-boun...@spectrumscale.org On Behalf Of Skylar Thompson Sent: Tuesday, July 7, 2020 10:00 AM To: gpfsug-discuss@spectrumscale.org Subject: Re: [gpfsug-discuss] dependent versus independent filesets We wanted to be able to snapshot and backup filesets separately with mmbackup, so went with independent filesets. On Tue, Jul 07, 2020 at 08:37:46AM -0500, Damir Krstic wrote: > We are deploying our new ESS and are considering moving to independent > filesets. The snapshot per fileset feature appeals to us. > > Has anyone considered independent vs. dependent filesets and what was > your reasoning to go with one as opposed to the other? Or perhaps you > opted to have both on your filesystem, and if, what was the reasoning for it? > > Thank you. > Damir > ___ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.com/v3/__http://gpfsug.org/mailman/listinfo/gpfsug- > discuss__;!!KGKeukY!j-c9kslUrEaNslhTbLLfaY8TES7Xf4eUCxysOaXwroHhTMwiVY > vcGNh4M_no$ -- -- Skylar Thompson (skyl...@u.washington.edu) -- Genome Sciences Department (UW Medicine), System Administrator -- Foege Building S046, (206)-685-7354 -- Pronouns: He/Him/His ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.com/v3/__http://gpfsug.org/mailman/listinfo/gpfsug-discuss__;!!KGKeukY!j-c9kslUrEaNslhTbLLfaY8TES7Xf4eUCxysOaXwroHhTMwiVYvcGNh4M_no$ ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] dependent versus independent filesets
We wanted to be able to snapshot and backup filesets separately with mmbackup, so went with independent filesets. On Tue, Jul 07, 2020 at 08:37:46AM -0500, Damir Krstic wrote: > We are deploying our new ESS and are considering moving to independent > filesets. The snapshot per fileset feature appeals to us. > > Has anyone considered independent vs. dependent filesets and what was your > reasoning to go with one as opposed to the other? Or perhaps you opted to > have both on your filesystem, and if, what was the reasoning for it? > > Thank you. > Damir > ___ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- -- Skylar Thompson (skyl...@u.washington.edu) -- Genome Sciences Department (UW Medicine), System Administrator -- Foege Building S046, (206)-685-7354 -- Pronouns: He/Him/His ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
