dear colleagues,
I don't think, that we can cover a specific customer setup .. .you may open a PMR to address your issues or ask your local IBM /business partner account
but to answer your questions...
(1) pagepool can be up to 75% (by default) and it works well with RDMA , even for pagepool > 64GB
(2) the reason for high load is hard to analyze by email ... and I doubt, that reducing this high load will be done by lowering pagepool
(3) should nt be to tune any more... since we have recent MLX adapter/Mofed, this should be obsolete
(4) most likely on that node this verbsPort is not existing or active (check by mmlsconfig and ibdev2netdev)
Mit freundlichen Grüßen / Kind regards
Olaf Weiser
IBM Systems, SpectrumScale Client Adoption
-------------------------------------------------------------------------------------------------------------------------------------------
IBM Deutschland
IBM Allee 1
71139 Ehningen
Phone: +49-170-579-44-66
E-Mail: [email protected]
-------------------------------------------------------------------------------------------------------------------------------------------
IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter
Geschäftsführung: Gregor Pillen (Vorsitzender), Agnes Heftberger, Norbert Janzen, Markus Koerner, Christian Noll, Nicole Reimer
Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 14562 / WEEE-Reg.-Nr. DE 99369940
----- Original message -----
From: "Frederick Stock" <[email protected]>
Sent by: [email protected]
To: [email protected]
Cc: [email protected]
Subject: [EXTERNAL] Re: [gpfsug-discuss] Spectrum Scale pagepool size with RDMA
Date: Thu, Jul 23, 2020 1:15 PM
And what version of ESS/Scale are you running on your systems (mmdiag --version)?
Fred
__________________________________________________
Fred Stock | IBM Pittsburgh Lab | 720-430-8821
[email protected]----- Original message -----
From: "Yaron Daniel" <[email protected]>
Sent by: [email protected]
To: gpfsug main discussion list <[email protected]>
Cc:
Subject: [EXTERNAL] Re: [gpfsug-discuss] Spectrum Scale pagepool size with RDMA
Date: Thu, Jul 23, 2020 3:09 AM
Hi
What is the output for:
#mmlsconfig |grep -i verbs
#ibstat
Regards
Yaron Daniel 94 Em Ha'Moshavot Rd Storage Architect – IL Lab Services (Storage) Petach Tiqva, 49527 IBM Global Markets, Systems HW Sales Israel Phone: +972-3-916-5672 Fax: +972-3-916-5672 Mobile: +972-52-8395593 e-mail: [email protected] Webex: https://ibm.webex.com/meet/yard
IBM Israel
![]()
From: Prasad Surampudi <[email protected]>
To: "[email protected]" <[email protected]>
Date: 07/23/2020 03:34 AM
Subject: [EXTERNAL] [gpfsug-discuss] Spectrum Scale pagepool size with RDMA
Sent by: [email protected]
Hi,
We have an ESS clusters with two CES nodes. The pagepool is set to 128 GB ( Real Memory is 256 GB ) on both ESS NSD servers and CES nodes as well. Occasionally we see the mmfsd process memory usage reaches 90% on NSD servers and CES nodes and stays there until GPFS is recycled. I have couple of questions in this scenario:
- What are the general recommendations of pagepool size for nodes with RDMA enabled? On, IBM knowledge center for RDMA tuning says "If the GPFS pagepool is set to 32 GB, then the mapping of the RDMA for this pagepool must be at least 64 GB." So, does this mean that the pagepool can't be more than half of real memory with RDMA enabled? Also, Is this the reason why mmfsd memory usage exceeds pagepool size and spikes to almost 90%?
- If we dont want to see high mmfsd process memory usage on NSD/CES nodes, should we decrease the pagepool size?
- Can we tune log_num_mtt parameter to limit the memory usage? Currently its set to 0 for both NSD (ppc64_le) and CES (x86_64).
- We also see messages like "Verbs RDMA disabled for xx.xx.xx.xx due to no matching port found" . Any idea what this message indicate? I dont see any Verbs RDMA enabled message after these warning messages. Does it get enabled automatically?
Prasad Surampudi|Sr. Systems Engineer|ATS Group, LLC
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
