Hi Luis, Thanks for your quick response. I was referring to fabric QoS, not Spectrum Scale QoS (which I have some experience with). The reference to slide 10 of said presentation says to configure the RDMA device as follows to use QoS
verbsPorts = <device>/<port>/<fabric>/<service level> − List of RDMA ports to be used − <device> : RDMA device, required, e.g. mlx5_0, mlx5_1 − <port>: RDMA port on device, default 1, valid values are 1 or 2 − <fabric>: virtual fabric number, default 0, valid values are >= 0 Only verbsPorts using a common <fabric> are connected − <service level>: QoS level, default is 0, valid values defined in SM configuration The references to SM and verbsPorts being part of RDMA configuration indicates (to me) that this is fabric QoS … -- Best regards / Mit freundlichem Gruß Thomas Bernecker ------------------------------------------------------------------------------------------------------------------ Manager System Integration and Support, HPCE Division Mobile: +49 (1522) 2851523, Fax: +49 (211) 5369-199, Home Office: +49 (38821) 65091 NEC Deutschland GmbH, Fritz-Vomfelde-Straße 14-16, 40547 Düsseldorf, Germany Geschäftsführer: Christopher Richard Jackson – Handelsregister Düsseldorf HRB 57941 ------------------------------------------------------------------------------------------------------------------- From: gpfsug-discuss <[email protected]> On Behalf Of Luis Bolinches Sent: Thursday, September 7, 2023 10:51 AM To: gpfsug main discussion list <[email protected]> Subject: Re: [gpfsug-discuss] Spectrum Scale, InfiniBand and QoS Hi Assuming you are talking about GPFS QoS and not any fabric QoS. It would work with any supported fabric, ESS or not. It is a filesystem feature well above all the fabrics and HW. Limits IOPS per class https: //www. ibm. com/docs/en/storage-scale/5. 1. 8?topic=reference-mmchqos-command ZjQcmQRYFpfptBannerStart Be Careful With This Message The sender's identity could not be verified and someone may be impersonating the sender. Report Suspicious <https://us-phishalarm-ewt.proofpoint.com/EWT/v1/BQNorrFsuw!1ltCYpPUsLPrqvNt6ouhTp6NGkpGHjneRgeZQ4EL8X29-niUBGCBdKKkDls1qqoVuPo1KL8kYyJpf4yhW6ztYNlnYstlXTWSM5mLl3UqEvexW0whUlE-4P3Yj_DDkWnTV-cejP_cc5NNq29-2g$> ZjQcmQRYFpfptBannerEnd Hi Assuming you are talking about GPFS QoS and not any fabric QoS. It would work with any supported fabric, ESS or not. It is a filesystem feature well above all the fabrics and HW. Limits IOPS per class https://www.ibm.com/docs/en/storage-scale/5.1.8?topic=reference-mmchqos-command [ibm.com]<https://urldefense.com/v3/__https:/www.ibm.com/docs/en/storage-scale/5.1.8?topic=reference-mmchqos-command__;!!BQNorrFsuw!kLyDPEkaxGCP9yaFDTa9ncFU_wIG_irxMgBkJIrwT1H-30N9Wziwk81U93JepyFgVaBXCY7echVIaxaaWDoEvxGXUSF5bwZd$> I recommend (depending on number of nodes and seconds) fine-stats so then you can visualize which PID on which client node is doing what. I believe I did something of that on SSUG London 18 or 19 -- Ystävällisin terveisin/Regards/Saludos/Salutations/Salutacions Luis Bolinches Executive IT Specialist IBM Storage Scale Server (formerly ESS) developer Phone: +358503112585 Ab IBM Finland Oy Toinen linja 7 00530 Helsinki Uusimaa - Finland Visitors entrance: Siltasaarenkatu 22 "If you always give you will always have" -- Anonymous https://www.credly.com/users/luis-bolinches/badges [credly.com]<https://urldefense.com/v3/__https:/www.credly.com/users/luis-bolinches/badges__;!!BQNorrFsuw!kLyDPEkaxGCP9yaFDTa9ncFU_wIG_irxMgBkJIrwT1H-30N9Wziwk81U93JepyFgVaBXCY7echVIaxaaWDoEvxGXUVT_rW2x$> From: gpfsug-discuss <[email protected]<mailto:[email protected]>> On Behalf Of Thomas Bernecker Sent: Thursday, 7 September 2023 11.36 To: [email protected]<mailto:[email protected]> Subject: [EXTERNAL] [gpfsug-discuss] Spectrum Scale, InfiniBand and QoS Dear all, we recently found this presentation from 2021: https: //www. spectrumscaleug. org/wp-content/uploads/2021/05/SSSD21DE-06-Improving-Spectrum-Scale-performance-using-RDMA. pdf On page 10 it explains by setting the RDMA device in a specific ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. Report Suspicious <https://us-phishalarm-ewt.proofpoint.com/EWT/v1/PjiDSg!1e-v7R5zRvmblYv7uGFl7ra7I0IqiHHRBBi-CjVQYUsssj6dhwnB7Lo2ZohM0He5_MhQUpUfgGfOHPXlTwIhxLUDAE1nbLv0ArY1Ygr3eS1cGzhvDzAy1DtrxLl5hHS5FHw6KsNKzpBBnc2d$> ZjQcmQRYFpfptBannerEnd Dear all, we recently found this presentation from 2021: https://www.spectrumscaleug.org/wp-content/uploads/2021/05/SSSD21DE-06-Improving-Spectrum-Scale-performance-using-RDMA.pdf [spectrumscaleug.org]<https://urldefense.com/v3/__https:/www.spectrumscaleug.org/wp-content/uploads/2021/05/SSSD21DE-06-Improving-Spectrum-Scale-performance-using-RDMA.pdf__;!!BQNorrFsuw!kLyDPEkaxGCP9yaFDTa9ncFU_wIG_irxMgBkJIrwT1H-30N9Wziwk81U93JepyFgVaBXCY7echVIaxaaWDoEvxGXUaGQ5_uh$> On page 10 it explains by setting the RDMA device in a specific way one could employ QoS with InfiniBand, which is something we would like to achieve as well. Has anyone a working environment using QoS with InfiniBand outside of the ESS domain? If so, would you share your experience? Sorry for asking a rather broad question, but it seems that this is not well-known stuff … -- Best regards / Mit freundlichem Gruß Thomas Bernecker ------------------------------------------------------------------------------------------------------------------ Manager System Integration and Support, HPCE Division Mobile: +49 (1522) 2851523, Fax: +49 (211) 5369-199, Home Office: +49 (38821) 65091 NEC Deutschland GmbH, Fritz-Vomfelde-Straße 14-16, 40547 Düsseldorf, Germany Geschäftsführer: Christopher Richard Jackson – Handelsregister Düsseldorf HRB 57941 ------------------------------------------------------------------------------------------------------------------- Unless otherwise stated above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland
_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org
