The discrepancy between the mmlsconfig view and mmdiag has been fixed in GFPS 4.2.3 version. Note, mmdiag reports the correct default value.
Tru. From: [email protected] To: [email protected] Date: 09/01/2017 06:43 PM Subject: gpfsug-discuss Digest, Vol 68, Issue 2 Sent by: [email protected] Send gpfsug-discuss mailing list submissions to [email protected] To subscribe or unsubscribe via the World Wide Web, visit https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=HQmkdQWQHoc1Nu6Mg_g8NVugim3OiUUy5n0QgLQcbkM&m=yK4FkYvJ21ubvurR6W1Pi3qvNw9ydj2XP0ghXPc7DUw&s=xZHUN9ZlFjvgBmBB8wnX2cQDQQV42R_q-xHubNA3JBM&e= or, via email, send a message with subject or body 'help' to [email protected] You can reach the person managing the list at [email protected] When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Re: GPFS GUI Nodes > NSD no data (Sobey, Richard A) 2. Change to default for verbsRdmaMinBytes? (Edward Wahl) 3. Quorum managers (Joshua Akers) 4. Re: Change to default for verbsRdmaMinBytes? (Sven Oehme) ---------------------------------------------------------------------- Message: 1 Date: Fri, 1 Sep 2017 13:36:56 +0000 From: "Sobey, Richard A" <[email protected]> To: gpfsug main discussion list <[email protected]> Subject: Re: [gpfsug-discuss] GPFS GUI Nodes > NSD no data Message-ID: <he1pr0602mb3225b01281871606c60c6333df...@he1pr0602mb3225.eurprd06.prod.outlook.com> Content-Type: text/plain; charset="us-ascii" Resolved this, guessed at changing GPFSNSDDisk.period to 5. From: [email protected] [ mailto:[email protected]] On Behalf Of Sobey, Richard A Sent: 01 September 2017 09:45 To: '[email protected]' <[email protected]> Subject: [gpfsug-discuss] GPFS GUI Nodes > NSD no data For some time now if I go into the GUI, select Monitoring > Nodes > NSD Server Nodes, the only columns with good data are Name, State and NSD Count. Everything else e.g. Avg Disk Wait Read is listed "N/A". Is this another config option I need to enable? It's been bugging me for a while, I don't think I've seen it work since 4.2.1 which was the first time I saw the GUI. Cheers Richard -------------- next part -------------- An HTML attachment was scrubbed... URL: < https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_attachments_20170901_2a4162e9_attachment-2D0001.html&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=HQmkdQWQHoc1Nu6Mg_g8NVugim3OiUUy5n0QgLQcbkM&m=yK4FkYvJ21ubvurR6W1Pi3qvNw9ydj2XP0ghXPc7DUw&s=jcPGl5zwtQFMbnEmBpNErsD43uwoVeKgKk_8j7ZeCJY&e= > ------------------------------ Message: 2 Date: Fri, 1 Sep 2017 16:56:25 -0400 From: Edward Wahl <[email protected]> To: gpfsug main discussion list <[email protected]> Subject: [gpfsug-discuss] Change to default for verbsRdmaMinBytes? Message-ID: <[email protected]> Content-Type: text/plain; charset="US-ASCII" Howdy. Just noticed this change to min RDMA packet size and I don't seem to see it in any patch notes. Maybe I just skipped the one where this changed? mmlsconfig verbsRdmaMinBytes verbsRdmaMinBytes 16384 (in case someone thinks we changed it) [root@proj-nsd01 ~]# mmlsconfig |grep verbs verbsRdma enable verbsRdma disable verbsRdmasPerConnection 14 verbsRdmasPerNode 1024 verbsPorts mlx5_3/1 verbsPorts mlx4_0 verbsPorts mlx5_0 verbsPorts mlx5_0 mlx5_1 verbsPorts mlx4_1/1 verbsPorts mlx4_1/2 Oddly I also see this in config, though I've seen these kinds of things before. mmdiag --config |grep verbsRdmaMinBytes verbsRdmaMinBytes 8192 We're on a recent efix. Current GPFS build: "4.2.2.3 efix21 (1028007)". -- Ed Wahl Ohio Supercomputer Center 614-292-9302 ------------------------------ Message: 3 Date: Fri, 01 Sep 2017 21:06:15 +0000 From: Joshua Akers <[email protected]> To: "[email protected]" <[email protected]> Subject: [gpfsug-discuss] Quorum managers Message-ID: <CAHO5rBG+PkntpshV105j54 [email protected]> Content-Type: text/plain; charset="utf-8" Hi all, I was wondering how most people set up quorum managers. We historically had physical admin nodes be the quorum managers, but are switching to a virtualized admin services infrastructure. We have been choosing a few compute nodes to act as quorum managers in our client clusters, but have considered using virtual machines instead. Has anyone else done this? Regards, Josh -- *Joshua D. Akers* *HPC Team Lead* NI&S Systems Support (MC0214) 1700 Pratt Drive Blacksburg, VA 24061 540-231-9506 -------------- next part -------------- An HTML attachment was scrubbed... URL: < https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_attachments_20170901_a49947db_attachment-2D0001.html&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=HQmkdQWQHoc1Nu6Mg_g8NVugim3OiUUy5n0QgLQcbkM&m=yK4FkYvJ21ubvurR6W1Pi3qvNw9ydj2XP0ghXPc7DUw&s=Gag0raQbp7KZAyINlnmuxlnpjboo9XOWO3dDL2HCsZo&e= > ------------------------------ Message: 4 Date: Fri, 01 Sep 2017 22:42:55 +0000 From: Sven Oehme <[email protected]> To: gpfsug main discussion list <[email protected]> Subject: Re: [gpfsug-discuss] Change to default for verbsRdmaMinBytes? Message-ID: <CALssuR3-KmWd4LqKZwQkP7_F6+aX3JM-KqRnc= [email protected]> Content-Type: text/plain; charset="utf-8" Hi Ed, yes the defaults for that have changed for customers who had not overridden the default settings. the reason we did this was that many systems in the field including all ESS systems that come pre-tuned where manually changed to 8k from the 16k default due to better performance that was confirmed in multiple customer engagements and tests with various settings , therefore we change the default to what it should be in the field so people are not bothered to set it anymore (simplification) or get benefits by changing the default to provides better performance. all this happened when we did the communication code overhaul that did lead to significant (think factors) of improved RPC performance for RDMA and VERBS workloads. there is another round of significant enhancements coming soon , that will make even more parameters either obsolete or change some of the defaults for better out of the box performance. i see that we should probably enhance the communication of this changes, not that i think this will have any negative effect compared to what your performance was with the old setting i am actually pretty confident that you get better performance with the new code, but by setting parameters back to default on most 'manual tuned' probably makes your system even faster. if you have a Scale Client on 4.2.3+ you really shouldn't have anything set beside maxfilestocache, pagepool, workerthreads and potential prefetch , if you are a protocol node, this and settings specific to an export (e.g. SMB, NFS set some special settings) , pretty much everything else these days should be set to default so the code can pick the correct parameters., if its not and you get better performance by manual tweaking something i like to hear about it. on the communication side in the next release will eliminate another set of parameters that are now 'auto set' and we plan to work on NSD next. i presented various slides about the communication and simplicity changes in various forums, latest public non NDA slides i presented are here --> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Manchester_08-5FResearch-5FTopics.pdf&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=HQmkdQWQHoc1Nu6Mg_g8NVugim3OiUUy5n0QgLQcbkM&m=yK4FkYvJ21ubvurR6W1Pi3qvNw9ydj2XP0ghXPc7DUw&s=8c_55Ld_iAC2sr_QU0cyGiOiyU7Z9NjcVknVuRpRIlk&e= hope this helps . Sven On Fri, Sep 1, 2017 at 1:56 PM Edward Wahl <[email protected]> wrote: > Howdy. Just noticed this change to min RDMA packet size and I don't seem > to > see it in any patch notes. Maybe I just skipped the one where this > changed? > > mmlsconfig verbsRdmaMinBytes > verbsRdmaMinBytes 16384 > > (in case someone thinks we changed it) > > [root@proj-nsd01 ~]# mmlsconfig |grep verbs > verbsRdma enable > verbsRdma disable > verbsRdmasPerConnection 14 > verbsRdmasPerNode 1024 > verbsPorts mlx5_3/1 > verbsPorts mlx4_0 > verbsPorts mlx5_0 > verbsPorts mlx5_0 mlx5_1 > verbsPorts mlx4_1/1 > verbsPorts mlx4_1/2 > > > Oddly I also see this in config, though I've seen these kinds of things > before. > mmdiag --config |grep verbsRdmaMinBytes > verbsRdmaMinBytes 8192 > > We're on a recent efix. > Current GPFS build: "4.2.2.3 efix21 (1028007)". > > -- > > Ed Wahl > Ohio Supercomputer Center > 614-292-9302 <(614)%20292-9302> > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=HQmkdQWQHoc1Nu6Mg_g8NVugim3OiUUy5n0QgLQcbkM&m=yK4FkYvJ21ubvurR6W1Pi3qvNw9ydj2XP0ghXPc7DUw&s=xZHUN9ZlFjvgBmBB8wnX2cQDQQV42R_q-xHubNA3JBM&e= > -------------- next part -------------- An HTML attachment was scrubbed... URL: < https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_attachments_20170901_b75cfc74_attachment.html&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=HQmkdQWQHoc1Nu6Mg_g8NVugim3OiUUy5n0QgLQcbkM&m=yK4FkYvJ21ubvurR6W1Pi3qvNw9ydj2XP0ghXPc7DUw&s=LpVpXMgqE_LD-t_J7yfNwURUrdUR29TzWvjVTi18kpA&e= > ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=HQmkdQWQHoc1Nu6Mg_g8NVugim3OiUUy5n0QgLQcbkM&m=yK4FkYvJ21ubvurR6W1Pi3qvNw9ydj2XP0ghXPc7DUw&s=xZHUN9ZlFjvgBmBB8wnX2cQDQQV42R_q-xHubNA3JBM&e= End of gpfsug-discuss Digest, Vol 68, Issue 2 *********************************************
_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
