Re: [gpfsug-discuss] Rebalancing with mmrestripefs -P
I’m coming late to the party on this so forgive me, but I found that even using QoS I could not even snapshot my filesets in a timely fashion, so my rebalancing could only run at weekends with snapshotting disabled. Richard From: gpfsug-discuss-boun...@spectrumscale.org On Behalf Of David Johnson Sent: 20 August 2018 17:55 To: gpfsug main discussion list Subject: [gpfsug-discuss] Rebalancing with mmrestripefs -P I have one storage pool that was recently doubled, and another pool migrated there using mmapplypolicy. The new half is only 50% full, and the old half is 94% full. Disks in storage pool: cit_10tb (Maximum disk size allowed is 516 TB) d05_george_23 50.49T 23 No Yes 25.91T ( 51%) 18.93G ( 0%) d04_george_23 50.49T 23 No Yes 25.91T ( 51%) 18.9G ( 0%) d03_george_23 50.49T 23 No Yes 25.9T ( 51%) 19.12G ( 0%) d02_george_23 50.49T 23 No Yes 25.9T ( 51%) 19.03G ( 0%) d01_george_23 50.49T 23 No Yes 25.9T ( 51%) 18.92G ( 0%) d00_george_23 50.49T 23 No Yes 25.91T ( 51%) 19.05G ( 0%) d06_cit_33 50.49T 33 No Yes 3.084T ( 6%) 70.35G ( 0%) d07_cit_33 50.49T 33 No Yes 3.084T ( 6%) 70.2G ( 0%) d05_cit_33 50.49T 33 No Yes 3.084T ( 6%) 69.93G ( 0%) d04_cit_33 50.49T 33 No Yes 3.085T ( 6%) 70.11G ( 0%) d03_cit_33 50.49T 33 No Yes 3.084T ( 6%) 70.08G ( 0%) d02_cit_33 50.49T 33 No Yes 3.083T ( 6%) 70.3G ( 0%) d01_cit_33 50.49T 33 No Yes 3.085T ( 6%) 70.25G ( 0%) d00_cit_33 50.49T 33 No Yes 3.083T ( 6%) 70.28G ( 0%) - --- (pool total) 706.9T180.1T ( 25%) 675.5G ( 0%) Will the command "mmrestripfs /gpfs -b -P cit_10tb” move the data blocks from the _cit_ NSDs to the _george_ NSDs, so that they end up all around 75% full? Thanks, — ddj Dave Johnson Brown University CCV/CIS ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] Rebalancing with mmrestripefs -P
Yes the arrays are in different buildings. We want to spread the activity over more servers if possible but recognize the extra load that rebalancing would entail. The system is busy all the time. I have considered using QOS when we run policy migrations but haven’t yet because I don’t know what value to allow for throttling IOPS. We need to do weekly migrations off of 15k rpm pool onto 7.2k rpm pool, and previously I’ve just let it run at native speed. I’d like to know what other folks have used for QOS settings. I think we may leave things alone for now regarding the original question, rebalancing this pool. -- ddj Dave Johnson > On Aug 20, 2018, at 6:08 PM, valdis.kletni...@vt.edu wrote: > > On Mon, 20 Aug 2018 14:02:05 -0400, "Frederick Stock" said: > >> Note you have two additional NSDs in the 33 failure group than you do in >> the 23 failure group. You may want to change one of those NSDs in failure >> group 33 to be in failure group 23 so you have equal storage space in both >> failure groups. > > Keep in mind that the failure groups should be built up based on single > points of failure. > In other words, a failure group should consist of disks that will all stay up > or all go down on > the same failure (controller, network, whatever). > > Looking at the fact that you have 6 disks named 'dNN_george_33' and 8 named > 'dNN_cit_33', > it sounds very likely that they are in two different storage arrays, and you > should make your > failure groups so they don't span a storage array. In other words, taking a > 'cit' disk > and moving it into a 'george' failure group will Do The Wrong Thing, because > if you do > data replication, one copy can go onto a 'george' disk, and the other onto a > 'cit' disk > that's in the same array as the 'george' disk. If 'george' fails, you lose > access to both > replicas. > ___ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] Rebalancing with mmrestripefs -P
On Mon, 20 Aug 2018 14:02:05 -0400, "Frederick Stock" said: > Note you have two additional NSDs in the 33 failure group than you do in > the 23 failure group. You may want to change one of those NSDs in failure > group 33 to be in failure group 23 so you have equal storage space in both > failure groups. Keep in mind that the failure groups should be built up based on single points of failure. In other words, a failure group should consist of disks that will all stay up or all go down on the same failure (controller, network, whatever). Looking at the fact that you have 6 disks named 'dNN_george_33' and 8 named 'dNN_cit_33', it sounds very likely that they are in two different storage arrays, and you should make your failure groups so they don't span a storage array. In other words, taking a 'cit' disk and moving it into a 'george' failure group will Do The Wrong Thing, because if you do data replication, one copy can go onto a 'george' disk, and the other onto a 'cit' disk that's in the same array as the 'george' disk. If 'george' fails, you lose access to both replicas. pgpS9Xvy2S2JO.pgp Description: PGP signature ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] Rebalancing with mmrestripefs -P
Hey Dave, Can you say more about what you are trying to accomplish by doing the rebalance? IME, the performance hit from running the rebalance was higher than the performance hit from writes being directed to a subset of the disks. If you have any churn of the data, eventually they will rebalance anyway. Regards, Alex On Mon, Aug 20, 2018 at 11:06 AM wrote: > Does anyone have a good rule of thumb for iops to allow for background QOS > tasks? > > > > -- ddj > Dave Johnson > > On Aug 20, 2018, at 2:02 PM, Frederick Stock wrote: > > That should do what you want. Be aware that mmrestripefs generates > significant IO load so you should either use the QoS feature to mitigate > its impact or run the command when the system is not very busy. > > Note you have two additional NSDs in the 33 failure group than you do in > the 23 failure group. You may want to change one of those NSDs in failure > group 33 to be in failure group 23 so you have equal storage space in both > failure groups. > > Fred > __ > Fred Stock | IBM Pittsburgh Lab | 720-430-8821 > sto...@us.ibm.com > > > > From:David Johnson > To:gpfsug main discussion list > Date:08/20/2018 12:55 PM > Subject:[gpfsug-discuss] Rebalancing with mmrestripefs -P > Sent by:gpfsug-discuss-boun...@spectrumscale.org > -- > > > > I have one storage pool that was recently doubled, and another pool > migrated there using mmapplypolicy. > The new half is only 50% full, and the old half is 94% full. > > Disks in storage pool: cit_10tb (Maximum disk size allowed is 516 TB) > d05_george_23 50.49T 23 No Yes 25.91T ( 51%) >18.93G ( 0%) > d04_george_23 50.49T 23 No Yes 25.91T ( 51%) > 18.9G ( 0%) > d03_george_23 50.49T 23 No Yes 25.9T ( 51%) >19.12G ( 0%) > d02_george_23 50.49T 23 No Yes 25.9T ( 51%) >19.03G ( 0%) > d01_george_23 50.49T 23 No Yes 25.9T ( 51%) >18.92G ( 0%) > d00_george_23 50.49T 23 No Yes 25.91T ( 51%) >19.05G ( 0%) > d06_cit_33 50.49T 33 No Yes 3.084T ( 6%) >70.35G ( 0%) > d07_cit_33 50.49T 33 No Yes 3.084T ( 6%) > 70.2G ( 0%) > d05_cit_33 50.49T 33 No Yes 3.084T ( 6%) >69.93G ( 0%) > d04_cit_33 50.49T 33 No Yes 3.085T ( 6%) >70.11G ( 0%) > d03_cit_33 50.49T 33 No Yes 3.084T ( 6%) >70.08G ( 0%) > d02_cit_33 50.49T 33 No Yes 3.083T ( 6%) > 70.3G ( 0%) > d01_cit_33 50.49T 33 No Yes 3.085T ( 6%) >70.25G ( 0%) > d00_cit_33 50.49T 33 No Yes 3.083T ( 6%) >70.28G ( 0%) > - > --- > (pool total) 706.9T180.1T ( 25%) >675.5G ( 0%) > > Will the command "mmrestripfs /gpfs -b -P cit_10tb” move the data blocks > from the _cit_ NSDs to the _george_ NSDs, > so that they end up all around 75% full? > > Thanks, > — ddj > Dave Johnson > Brown University CCV/CIS___ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > ___ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > ___ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] Rebalancing with mmrestripefs -P
Does anyone have a good rule of thumb for iops to allow for background QOS tasks? -- ddj Dave Johnson > On Aug 20, 2018, at 2:02 PM, Frederick Stock wrote: > > That should do what you want. Be aware that mmrestripefs generates > significant IO load so you should either use the QoS feature to mitigate its > impact or run the command when the system is not very busy. > > Note you have two additional NSDs in the 33 failure group than you do in the > 23 failure group. You may want to change one of those NSDs in failure group > 33 to be in failure group 23 so you have equal storage space in both failure > groups. > > Fred > __ > Fred Stock | IBM Pittsburgh Lab | 720-430-8821 > sto...@us.ibm.com > > > > From:David Johnson > To:gpfsug main discussion list > Date:08/20/2018 12:55 PM > Subject:[gpfsug-discuss] Rebalancing with mmrestripefs -P > Sent by:gpfsug-discuss-boun...@spectrumscale.org > > > > I have one storage pool that was recently doubled, and another pool migrated > there using mmapplypolicy. > The new half is only 50% full, and the old half is 94% full. > > Disks in storage pool: cit_10tb (Maximum disk size allowed is 516 TB) > d05_george_23 50.49T 23 No Yes 25.91T ( 51%) > 18.93G ( 0%) > d04_george_23 50.49T 23 No Yes 25.91T ( 51%) > 18.9G ( 0%) > d03_george_23 50.49T 23 No Yes 25.9T ( 51%) > 19.12G ( 0%) > d02_george_23 50.49T 23 No Yes 25.9T ( 51%) > 19.03G ( 0%) > d01_george_23 50.49T 23 No Yes 25.9T ( 51%) > 18.92G ( 0%) > d00_george_23 50.49T 23 No Yes 25.91T ( 51%) > 19.05G ( 0%) > d06_cit_33 50.49T 33 No Yes 3.084T ( 6%) > 70.35G ( 0%) > d07_cit_33 50.49T 33 No Yes 3.084T ( 6%) > 70.2G ( 0%) > d05_cit_33 50.49T 33 No Yes 3.084T ( 6%) > 69.93G ( 0%) > d04_cit_33 50.49T 33 No Yes 3.085T ( 6%) > 70.11G ( 0%) > d03_cit_33 50.49T 33 No Yes 3.084T ( 6%) > 70.08G ( 0%) > d02_cit_33 50.49T 33 No Yes 3.083T ( 6%) > 70.3G ( 0%) > d01_cit_33 50.49T 33 No Yes 3.085T ( 6%) > 70.25G ( 0%) > d00_cit_33 50.49T 33 No Yes 3.083T ( 6%) > 70.28G ( 0%) > - > --- > (pool total) 706.9T180.1T ( 25%) > 675.5G ( 0%) > > Will the command "mmrestripfs /gpfs -b -P cit_10tb” move the data blocks > from the _cit_ NSDs to the _george_ NSDs, > so that they end up all around 75% full? > > Thanks, > — ddj > Dave Johnson > Brown University CCV/CIS___ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > ___ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] Rebalancing with mmrestripefs -P
That should do what you want. Be aware that mmrestripefs generates significant IO load so you should either use the QoS feature to mitigate its impact or run the command when the system is not very busy. Note you have two additional NSDs in the 33 failure group than you do in the 23 failure group. You may want to change one of those NSDs in failure group 33 to be in failure group 23 so you have equal storage space in both failure groups. Fred __ Fred Stock | IBM Pittsburgh Lab | 720-430-8821 sto...@us.ibm.com From: David Johnson To: gpfsug main discussion list Date: 08/20/2018 12:55 PM Subject:[gpfsug-discuss] Rebalancing with mmrestripefs -P Sent by:gpfsug-discuss-boun...@spectrumscale.org I have one storage pool that was recently doubled, and another pool migrated there using mmapplypolicy. The new half is only 50% full, and the old half is 94% full. Disks in storage pool: cit_10tb (Maximum disk size allowed is 516 TB) d05_george_23 50.49T 23 No Yes 25.91T ( 51%) 18.93G ( 0%) d04_george_23 50.49T 23 No Yes 25.91T ( 51%) 18.9G ( 0%) d03_george_23 50.49T 23 No Yes 25.9T ( 51%) 19.12G ( 0%) d02_george_23 50.49T 23 No Yes 25.9T ( 51%) 19.03G ( 0%) d01_george_23 50.49T 23 No Yes 25.9T ( 51%) 18.92G ( 0%) d00_george_23 50.49T 23 No Yes 25.91T ( 51%) 19.05G ( 0%) d06_cit_33 50.49T 33 No Yes 3.084T ( 6%) 70.35G ( 0%) d07_cit_33 50.49T 33 No Yes 3.084T ( 6%) 70.2G ( 0%) d05_cit_33 50.49T 33 No Yes 3.084T ( 6%) 69.93G ( 0%) d04_cit_33 50.49T 33 No Yes 3.085T ( 6%) 70.11G ( 0%) d03_cit_33 50.49T 33 No Yes 3.084T ( 6%) 70.08G ( 0%) d02_cit_33 50.49T 33 No Yes 3.083T ( 6%) 70.3G ( 0%) d01_cit_33 50.49T 33 No Yes 3.085T ( 6%) 70.25G ( 0%) d00_cit_33 50.49T 33 No Yes 3.083T ( 6%) 70.28G ( 0%) - --- (pool total) 706.9T180.1T ( 25%) 675.5G ( 0%) Will the command "mmrestripfs /gpfs -b -P cit_10tb” move the data blocks from the _cit_ NSDs to the _george_ NSDs, so that they end up all around 75% full? Thanks, — ddj Dave Johnson Brown University CCV/CIS___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
