Re: [gpfsug-discuss] Steps for gracefully handling bandwidth reduction during network maintenance
Please note that the maxmbps parameter of mmchconfig is not part of the QOS features of the mmchqos command. mmchqos can be used to precisely limit IOPs. You can even set different limits for NSD traffic originating at different nodes. However, use the "force" of QOS carefully! No doubt you can bring a system to a virtual standstill if you set the IOPS values incorrectly. ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] Steps for gracefully handling bandwidth reduction during network maintenance
Hi Writing from phone so excuse the typos. Assuming you have a system pool (metadata) and some other pool/s you can set limits on maintenance class that you done already and on other class that would affect all the other ops. You can add those per node or nodeclass that can be matched to what part/s of network you are working with. Changes are online and immediate. And you can measure normal load just by having QoS activated and looking into the values for few days. Hope makes some sense the above. -- Cheers > On 17 Jun 2019, at 19.48, Christopher Black wrote: > > The man page indicates that maxMBpS can be used to "artificially limit how much I/O one node can put on all of the disk servers", but it might not be the best choice. Man page also says maxmbps is in the class of mmchconfig changes take place immediately. > We've only ever used QoS for throttling maint operations (restripes, etc) and are unfamiliar with how to best use it to throttle client load. > > Best, > Chris > > On 6/17/19, 12:40 PM, "gpfsug-discuss-boun...@spectrumscale.org on behalf of Skylar Thompson" wrote: > >IIRC, maxMBpS isn't really a limit, but more of a hint for how GPFS should >use its in-memory buffers for read prefetches and dirty writes. > >>On Mon, Jun 17, 2019 at 09:31:38AM -0700, Alex Chekholko wrote: >> Hi Chris, >> >> I think the next thing to double-check is when the maxMBpS change takes >> effect. You may need to restart the nsds. Otherwise I think your plan is >> sound. >> >> Regards, >> Alex >> >> >> On Mon, Jun 17, 2019 at 9:24 AM Christopher Black >> wrote: >> >>> Our network team sometimes needs to take down sections of our network for >>> maintenance. Our systems have dual paths thru pairs of switches, but often >>> the maintenance will take down one of the two paths leaving all our nsd >>> servers with half bandwidth. >>> >>> Some of our systems are transmitting at a higher rate than can be handled >>> by half network (2x40Gb hosts with tx of 50Gb+). >>> >>> What can we do to gracefully handle network maintenance reducing bandwidth >>> in half? >>> >>> Should we set maxMBpS for affected nodes to a lower value? (default on our >>> ess appears to be maxMBpS = 3, would I reduce this to ~4000 for 32Gbps?) >>> >>> Any other ideas or comments? >>> >>> Our hope is that metadata operations are not affected much and users just >>> see jobs and processes read or write at a slower rate. >>> >>> >>> >>> Best, >>> >>> Chris >>> -- >>> This message is for the recipient???s use only, and may contain >>> confidential, privileged or protected information. Any unauthorized use or >>> dissemination of this communication is prohibited. If you received this >>> message in error, please immediately notify the sender and destroy all >>> copies of this message. The recipient should check this email and any >>> attachments for the presence of viruses, as we accept no liability for any >>> damage caused by any virus transmitted by this email. >>> ___ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=DopWM-bvfskhBn2zeglfyyw5U2pumni6m_QzQFYFepU&m=2ioq3oT4gzOlIvyQRqkdZF0GWKv1APEBmstC48AyVdo&s=fvxPTdT1cVT7av_-vR5-3wVgjIzEpUP8OY8vGx0i5kc&e= >>> > >> ___ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=DopWM-bvfskhBn2zeglfyyw5U2pumni6m_QzQFYFepU&m=2ioq3oT4gzOlIvyQRqkdZF0GWKv1APEBmstC48AyVdo&s=fvxPTdT1cVT7av_-vR5-3wVgjIzEpUP8OY8vGx0i5kc&e= > > >-- >-- Skylar Thompson (skyl...@u.washington.edu) >-- Genome Sciences Department, System Administrator >-- Foege Building S046, (206)-685-7354 >-- University of Washington School of Medicine >___ >gpfsug-discuss mailing list >gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=DopWM-bvfskhBn2zeglfyyw5U2pumni6m_QzQFYFepU&m=2ioq3oT4gzOlIvyQRqkdZF0GWKv1APEBmstC48AyVdo&s=fvxPTdT1cVT7av_-vR5-3wVgjIzEpUP8OY8vGx0i5kc&e= > > > > > This message is for the recipient’s use only, and may contain confidential, privileged or protected information. Any unauthorized use or dissemination of this communication is prohibited. If you received this message in error, please immediately notify the sender and destroy all copies of this message. The recipient should check this email and any attachments for the presence of viruses, as we accept no liability for any damage caused by any virus transmitted by this email. >
Re: [gpfsug-discuss] Steps for gracefully handling bandwidth reduction during network maintenance
Hi all, My experience with MaxMBpS was in the other direction but it did make a difference. We had lots of spare network bandwith (that is, the network was not the bottleneck) and in the course of various GPFS tuning it also looked like the disks were not too busy, and the NSDs were not too busy, so bumping up the MaxMBpS improved performance and allowed GPFS to do more. Of course, this was many years ago on different GPFS version and hardware, but I think it would work in the other direction. It should also be very safe to try. Regards, Alex On Mon, Jun 17, 2019 at 9:47 AM Christopher Black wrote: > The man page indicates that maxMBpS can be used to "artificially limit how > much I/O one node can put on all of the disk servers", but it might not be > the best choice. Man page also says maxmbps is in the class of mmchconfig > changes take place immediately. > We've only ever used QoS for throttling maint operations (restripes, etc) > and are unfamiliar with how to best use it to throttle client load. > > Best, > Chris > > On 6/17/19, 12:40 PM, "gpfsug-discuss-boun...@spectrumscale.org on > behalf of Skylar Thompson" behalf of skyl...@uw.edu> wrote: > > IIRC, maxMBpS isn't really a limit, but more of a hint for how GPFS > should > use its in-memory buffers for read prefetches and dirty writes. > > On Mon, Jun 17, 2019 at 09:31:38AM -0700, Alex Chekholko wrote: > > Hi Chris, > > > > I think the next thing to double-check is when the maxMBpS change > takes > > effect. You may need to restart the nsds. Otherwise I think your > plan is > > sound. > > > > Regards, > > Alex > > > > > > On Mon, Jun 17, 2019 at 9:24 AM Christopher Black < > cbl...@nygenome.org> > > wrote: > > > > > Our network team sometimes needs to take down sections of our > network for > > > maintenance. Our systems have dual paths thru pairs of switches, > but often > > > the maintenance will take down one of the two paths leaving all > our nsd > > > servers with half bandwidth. > > > > > > Some of our systems are transmitting at a higher rate than can be > handled > > > by half network (2x40Gb hosts with tx of 50Gb+). > > > > > > What can we do to gracefully handle network maintenance reducing > bandwidth > > > in half? > > > > > > Should we set maxMBpS for affected nodes to a lower value? > (default on our > > > ess appears to be maxMBpS = 3, would I reduce this to ~4000 > for 32Gbps?) > > > > > > Any other ideas or comments? > > > > > > Our hope is that metadata operations are not affected much and > users just > > > see jobs and processes read or write at a slower rate. > > > > > > > > > > > > Best, > > > > > > Chris > > > -- > > > This message is for the recipient???s use only, and may contain > > > confidential, privileged or protected information. Any > unauthorized use or > > > dissemination of this communication is prohibited. If you received > this > > > message in error, please immediately notify the sender and destroy > all > > > copies of this message. The recipient should check this email and > any > > > attachments for the presence of viruses, as we accept no liability > for any > > > damage caused by any virus transmitted by this email. > > > ___ > > > gpfsug-discuss mailing list > > > gpfsug-discuss at spectrumscale.org > > > > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=DopWM-bvfskhBn2zeglfyyw5U2pumni6m_QzQFYFepU&m=2ioq3oT4gzOlIvyQRqkdZF0GWKv1APEBmstC48AyVdo&s=fvxPTdT1cVT7av_-vR5-3wVgjIzEpUP8OY8vGx0i5kc&e= > > > > > > ___ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=DopWM-bvfskhBn2zeglfyyw5U2pumni6m_QzQFYFepU&m=2ioq3oT4gzOlIvyQRqkdZF0GWKv1APEBmstC48AyVdo&s=fvxPTdT1cVT7av_-vR5-3wVgjIzEpUP8OY8vGx0i5kc&e= > > > -- > -- Skylar Thompson (skyl...@u.washington.edu) > -- Genome Sciences Department, System Administrator > -- Foege Building S046, (206)-685-7354 > -- University of Washington School of Medicine > ___ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=DopWM-bvfskhBn2zeglfyyw5U2pumni6m_QzQFYFepU&m=2ioq3oT4gzOlIvyQRqkdZF0GWKv1APEBmstC48AyVdo&s=fvxPTdT1cVT7av_-vR5-3wVgjIzEpUP8OY8vGx0i5kc&e= > > > > > This message is for the recipient’s use only, and may contain >
Re: [gpfsug-discuss] Steps for gracefully handling bandwidth reduction during network maintenance
The man page indicates that maxMBpS can be used to "artificially limit how much I/O one node can put on all of the disk servers", but it might not be the best choice. Man page also says maxmbps is in the class of mmchconfig changes take place immediately. We've only ever used QoS for throttling maint operations (restripes, etc) and are unfamiliar with how to best use it to throttle client load. Best, Chris On 6/17/19, 12:40 PM, "gpfsug-discuss-boun...@spectrumscale.org on behalf of Skylar Thompson" wrote: IIRC, maxMBpS isn't really a limit, but more of a hint for how GPFS should use its in-memory buffers for read prefetches and dirty writes. On Mon, Jun 17, 2019 at 09:31:38AM -0700, Alex Chekholko wrote: > Hi Chris, > > I think the next thing to double-check is when the maxMBpS change takes > effect. You may need to restart the nsds. Otherwise I think your plan is > sound. > > Regards, > Alex > > > On Mon, Jun 17, 2019 at 9:24 AM Christopher Black > wrote: > > > Our network team sometimes needs to take down sections of our network for > > maintenance. Our systems have dual paths thru pairs of switches, but often > > the maintenance will take down one of the two paths leaving all our nsd > > servers with half bandwidth. > > > > Some of our systems are transmitting at a higher rate than can be handled > > by half network (2x40Gb hosts with tx of 50Gb+). > > > > What can we do to gracefully handle network maintenance reducing bandwidth > > in half? > > > > Should we set maxMBpS for affected nodes to a lower value? (default on our > > ess appears to be maxMBpS = 3, would I reduce this to ~4000 for 32Gbps?) > > > > Any other ideas or comments? > > > > Our hope is that metadata operations are not affected much and users just > > see jobs and processes read or write at a slower rate. > > > > > > > > Best, > > > > Chris > > -- > > This message is for the recipient???s use only, and may contain > > confidential, privileged or protected information. Any unauthorized use or > > dissemination of this communication is prohibited. If you received this > > message in error, please immediately notify the sender and destroy all > > copies of this message. The recipient should check this email and any > > attachments for the presence of viruses, as we accept no liability for any > > damage caused by any virus transmitted by this email. > > ___ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=DopWM-bvfskhBn2zeglfyyw5U2pumni6m_QzQFYFepU&m=2ioq3oT4gzOlIvyQRqkdZF0GWKv1APEBmstC48AyVdo&s=fvxPTdT1cVT7av_-vR5-3wVgjIzEpUP8OY8vGx0i5kc&e= > > > ___ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=DopWM-bvfskhBn2zeglfyyw5U2pumni6m_QzQFYFepU&m=2ioq3oT4gzOlIvyQRqkdZF0GWKv1APEBmstC48AyVdo&s=fvxPTdT1cVT7av_-vR5-3wVgjIzEpUP8OY8vGx0i5kc&e= -- -- Skylar Thompson (skyl...@u.washington.edu) -- Genome Sciences Department, System Administrator -- Foege Building S046, (206)-685-7354 -- University of Washington School of Medicine ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=DopWM-bvfskhBn2zeglfyyw5U2pumni6m_QzQFYFepU&m=2ioq3oT4gzOlIvyQRqkdZF0GWKv1APEBmstC48AyVdo&s=fvxPTdT1cVT7av_-vR5-3wVgjIzEpUP8OY8vGx0i5kc&e= This message is for the recipient’s use only, and may contain confidential, privileged or protected information. Any unauthorized use or dissemination of this communication is prohibited. If you received this message in error, please immediately notify the sender and destroy all copies of this message. The recipient should check this email and any attachments for the presence of viruses, as we accept no liability for any damage caused by any virus transmitted by this email. ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] Steps for gracefully handling bandwidth reduction during network maintenance
IIRC, maxMBpS isn't really a limit, but more of a hint for how GPFS should use its in-memory buffers for read prefetches and dirty writes. On Mon, Jun 17, 2019 at 09:31:38AM -0700, Alex Chekholko wrote: > Hi Chris, > > I think the next thing to double-check is when the maxMBpS change takes > effect. You may need to restart the nsds. Otherwise I think your plan is > sound. > > Regards, > Alex > > > On Mon, Jun 17, 2019 at 9:24 AM Christopher Black > wrote: > > > Our network team sometimes needs to take down sections of our network for > > maintenance. Our systems have dual paths thru pairs of switches, but often > > the maintenance will take down one of the two paths leaving all our nsd > > servers with half bandwidth. > > > > Some of our systems are transmitting at a higher rate than can be handled > > by half network (2x40Gb hosts with tx of 50Gb+). > > > > What can we do to gracefully handle network maintenance reducing bandwidth > > in half? > > > > Should we set maxMBpS for affected nodes to a lower value? (default on our > > ess appears to be maxMBpS = 3, would I reduce this to ~4000 for 32Gbps?) > > > > Any other ideas or comments? > > > > Our hope is that metadata operations are not affected much and users just > > see jobs and processes read or write at a slower rate. > > > > > > > > Best, > > > > Chris > > -- > > This message is for the recipient???s use only, and may contain > > confidential, privileged or protected information. Any unauthorized use or > > dissemination of this communication is prohibited. If you received this > > message in error, please immediately notify the sender and destroy all > > copies of this message. The recipient should check this email and any > > attachments for the presence of viruses, as we accept no liability for any > > damage caused by any virus transmitted by this email. > > ___ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > ___ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- -- Skylar Thompson (skyl...@u.washington.edu) -- Genome Sciences Department, System Administrator -- Foege Building S046, (206)-685-7354 -- University of Washington School of Medicine ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] Steps for gracefully handling bandwidth reduction during network maintenance
Hi I would really look into QoS instead. -- Cheers > On 17 Jun 2019, at 19.33, Alex Chekholko wrote: > > Hi Chris, > > I think the next thing to double-check is when the maxMBpS change takes > effect. You may need to restart the nsds. Otherwise I think your plan is > sound. > > Regards, > Alex > > >> On Mon, Jun 17, 2019 at 9:24 AM Christopher Black >> wrote: >> Our network team sometimes needs to take down sections of our network for >> maintenance. Our systems have dual paths thru pairs of switches, but often >> the maintenance will take down one of the two paths leaving all our nsd >> servers with half bandwidth. >> >> Some of our systems are transmitting at a higher rate than can be handled by >> half network (2x40Gb hosts with tx of 50Gb+). >> >> What can we do to gracefully handle network maintenance reducing bandwidth >> in half? >> >> Should we set maxMBpS for affected nodes to a lower value? (default on our >> ess appears to be maxMBpS = 3, would I reduce this to ~4000 for 32Gbps?) >> >> Any other ideas or comments? >> >> Our hope is that metadata operations are not affected much and users just >> see jobs and processes read or write at a slower rate. >> >> >> >> Best, >> >> Chris >> >> This message is for the recipient’s use only, and may contain confidential, >> privileged or protected information. Any unauthorized use or dissemination >> of this communication is prohibited. If you received this message in error, >> please immediately notify the sender and destroy all copies of this message. >> The recipient should check this email and any attachments for the presence >> of viruses, as we accept no liability for any damage caused by any virus >> transmitted by this email. >> ___ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss Ellei edellä ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] Steps for gracefully handling bandwidth reduction during network maintenance
Hi Chris, I think the next thing to double-check is when the maxMBpS change takes effect. You may need to restart the nsds. Otherwise I think your plan is sound. Regards, Alex On Mon, Jun 17, 2019 at 9:24 AM Christopher Black wrote: > Our network team sometimes needs to take down sections of our network for > maintenance. Our systems have dual paths thru pairs of switches, but often > the maintenance will take down one of the two paths leaving all our nsd > servers with half bandwidth. > > Some of our systems are transmitting at a higher rate than can be handled > by half network (2x40Gb hosts with tx of 50Gb+). > > What can we do to gracefully handle network maintenance reducing bandwidth > in half? > > Should we set maxMBpS for affected nodes to a lower value? (default on our > ess appears to be maxMBpS = 3, would I reduce this to ~4000 for 32Gbps?) > > Any other ideas or comments? > > Our hope is that metadata operations are not affected much and users just > see jobs and processes read or write at a slower rate. > > > > Best, > > Chris > -- > This message is for the recipient’s use only, and may contain > confidential, privileged or protected information. Any unauthorized use or > dissemination of this communication is prohibited. If you received this > message in error, please immediately notify the sender and destroy all > copies of this message. The recipient should check this email and any > attachments for the presence of viruses, as we accept no liability for any > damage caused by any virus transmitted by this email. > ___ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
[gpfsug-discuss] Steps for gracefully handling bandwidth reduction during network maintenance
Our network team sometimes needs to take down sections of our network for maintenance. Our systems have dual paths thru pairs of switches, but often the maintenance will take down one of the two paths leaving all our nsd servers with half bandwidth. Some of our systems are transmitting at a higher rate than can be handled by half network (2x40Gb hosts with tx of 50Gb+). What can we do to gracefully handle network maintenance reducing bandwidth in half? Should we set maxMBpS for affected nodes to a lower value? (default on our ess appears to be maxMBpS = 3, would I reduce this to ~4000 for 32Gbps?) Any other ideas or comments? Our hope is that metadata operations are not affected much and users just see jobs and processes read or write at a slower rate. Best, Chris This message is for the recipient’s use only, and may contain confidential, privileged or protected information. Any unauthorized use or dissemination of this communication is prohibited. If you received this message in error, please immediately notify the sender and destroy all copies of this message. The recipient should check this email and any attachments for the presence of viruses, as we accept no liability for any damage caused by any virus transmitted by this email. ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
