A few years ago, Yuri noted that "dmm" has a special meaning to GPFS code. SCSI3 PR code keys off this disk type.
https://www.ibm.com/developerworks/community/forums/html/topic?id=77777777-0000-0000-0000-000014924464#77777777-0000-0000-0000-000014924464 https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq.html#scsi3 Jon On Thu, Jan 18, 2018 at 3:22 PM, Eric Ross <[email protected]> wrote: > I would say that's what it *appears* to be doing. The sys admin in me > would still want some "official" word from IBM, as I wouldn't my > assumption to be the cause of a meltdown at 16.50 on a Friday when > everyone is heading out for the weekend ;) > > Cheers, > -Eric > > On Thu, Jan 18, 2018 at 2:09 PM, Bryan Banister > <[email protected]> wrote: > > Great! So this is just for selecting devices and according to my > `mmlsnsd -X` output it's using the correct ones, so I can probably return > to ignoring this parameter! > > -Bryan > > > > -----Original Message----- > > From: [email protected] [mailto:gpfsug-discuss- > [email protected]] On Behalf Of Eric Ross > > Sent: Thursday, January 18, 2018 2:01 PM > > To: gpfsug main discussion list <[email protected]> > > Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, > nsddevices file > > > > Note: External Email > > ------------------------------------------------- > > > > Bryan, > > > > While waiting on the "official" word as to what each setting does > > differently, I remembered there was a brief explanation of the > > differences (at least of dmm vs generic) in the > > /var/mmfs/etc/nsddevices included with the GPFS-goodies toolkit > > (https://github.com/finley/GPFS-Goodies) I use to use when I was at > > IBM. > > > > //snip > > > > dmm vs. generic is used by GPFS to prioritize internal order of > > searching through available disks, then later GPFS discards other > > disk device names that it finds that match as the same NSD device > > by a different path. For this reason, dmm vs. generic is an > > important distinction if you are not explicitly producing the > > entire and exclusive set of disks that GPFS should use, as output > > from this script (nsddevices) _and_ exiting this script with a > > "return 0". -Brian Finley > > > > //end snip > > > > If I read that correctly, it would indicate GPFS uses those additional > > labels (at least dmm vs generic) as a mechanism to determine which > > device files to prefer when scanning a system and finding the same NSD > > available via different devices (i.e. /dev/mapper/foo vs > > /dev/sdq,/dev/sdx). By associating dmm with the dm-multipathed > > device, I guess it would just ignore the /dev/sd${foo} devices when it > > scans them. Also, the difference only seems to matter if you're not > > explicitly creating a list a Brian F. indicates. If you simply > > generate the list and exit (via return 0), GPFS wouldn't continue > > scanning, and then find the associated /dev/sd${foo} devices to begin > > with, and therefore wouldn't need a label like dmm to prioritize it > > over say a generic device. > > > > > > - Eric > > > > On Thu, Jan 18, 2018 at 10:59 AM, Bryan Banister > > <[email protected]> wrote: > >> Yes, just change the /var/mmfs/etc/nsddevices file so that it sets the > >> Devtype to the “correct” setting, for example: > >> > >> > >> > >> if [[ $osName = Linux ]] > >> > >> then > >> > >> : # Add function to discover disks in the Linux environment. > >> > >> ls -l /dev/mpath/ 2>/dev/null | awk '{print "mpath/"$9 " generic"}' > >> > >> ls -l /dev/mapper/ 2>/dev/null | awk '{print "mapper/"$9 " dmm"}' > >> > >> ls -1 /dev/vd* 2>/dev/null | awk -F '/' '{print ""$3 " generic"}' > >> > >> fi > >> > >> > >> > >> My question is what is the correct setting? > >> > >> > >> > >> And what does GPFS do differently with each setting? > >> > >> > >> > >> Thanks, > >> > >> -Bryan > >> > >> > >> > >> From: [email protected] > >> [mailto:[email protected]] On Behalf Of Jeffrey > R. > >> Lang > >> Sent: Thursday, January 18, 2018 9:20 AM > >> To: gpfsug main discussion list <[email protected]> > >> Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, > >> nsddevices file > >> > >> > >> > >> Note: External Email > >> > >> ________________________________ > >> > >> So is there a way to change it if it’s set incorrectly? > >> > >> > >> > >> jeff > >> > >> > >> > >> From: [email protected] > >> [mailto:[email protected]] On Behalf Of Jim > Doherty > >> Sent: Wednesday, January 17, 2018 6:28 PM > >> To: gpfsug main discussion list <[email protected]> > >> Subject: Re: [gpfsug-discuss] Question about NSD "Devtype" setting, > >> nsddevices file > >> > >> > >> > >> > >> > >> Run a mmlsnsd -X I suspect you will see that GPFS is using one of the > >> /dev/sd* "generic" paths to the LUN, not the /dev/mapper/ path. In > our > >> case the device is setup as dmm > >> > >> > >> > >> [root@service5 ~]# mmlsnsd -X > >> > >> Disk name NSD volume ID Device Devtype Node name > >> Remarks > >> ------------------------------------------------------------ > --------------------------------------- > >> volume1 0972B6CD587CD8E0 /dev/dm-0 dmm > >> service5.pok.stglabs.ibm.com server node > >> volume1 0972B6CD587CD8E0 /dev/dm-0 dmm > >> service6.pok.stglabs.ibm.com server node > >> volume2 0972B6CE587CD8E4 /dev/dm-4 dmm > >> service5.pok.stglabs.ibm.com server node > >> volume2 0972B6CE587CD8E4 /dev/dm-3 dmm > >> service6.pok.stglabs.ibm.com server node > >> volume3 0972B6CD587CD8E7 /dev/dm-1 dmm > >> service5.pok.stglabs.ibm.com server node > >> volume3 0972B6CD587CD8E7 /dev/dm-2 dmm > >> service6.pok.stglabs.ibm.com server node > >> volume4 0972B6CE587CF625 /dev/dm-3 dmm > >> service5.pok.stglabs.ibm.com server node > >> volume4 0972B6CE587CF625 /dev/dm-4 dmm > >> service6.pok.stglabs.ibm.com server node > >> > >> [root@service5 ~]# grep volume1 /var/mmfs/gen/mmsdrfs | grep SG_DISK > >> %%home%%:60_SG_DISKS:gpfs5:1:volume1:0:5001:dataAndMetadata: > 0972B6CD587CD8E0:nsd:service5.pok.stglabs.ibm.com,service6. > pok.stglabs.ibm.com::other::dmm:user:::quorumDisk:ready::system: > service5.pok.stglabs.ibm.com,service6.pok.stglabs.ibm.com::::: > >> [root@service5 ~]# > >> > >> > >> > >> If you run an tspreparedisk -s it will show you all of the paths. > >> > >> > >> > >> [root@service5 ~]# tspreparedisk -s | grep 0972B6CD587CD8E0 > >> 0972B6CD587CD8E0 /dev/sda generic > >> 0972B6CD587CD8E0 /dev/sdk generic > >> 0972B6CD587CD8E0 /dev/sdu generic > >> 0972B6CD587CD8E0 /dev/sdah generic > >> 0972B6CD587CD8E0 /dev/dm-0 dmm > >> [root@service5 ~]# > >> > >> > >> > >> Jim > >> > >> > >> > >> > >> > >> > >> > >> Jim > >> > >> On Wednesday, January 17, 2018, 5:12:10 PM EST, Bryan Banister > >> <[email protected]> wrote: > >> > >> > >> > >> > >> > >> Hi all, > >> > >> > >> > >> We are reviewing some of our configurations and were not sure what to > make > >> of the NSD Device Types that GPFS uses and what, if anything, do they > change > >> about how GPFS accesses/recovers/manages/etc the underlying storage > based on > >> this setting. > >> > >> > >> > >> The documentation doesn’t say much about it other than to consult the > >> /usr/lpp/mmfs/bin/mmdevdiscover command (no man page), which has this > >> section: > >> > >> > >> > >> # Known disk types currently are: > >> > >> # > >> > >> # powerdisk - EMC power path disk > >> > >> # vpath - IBM virtual path disk > >> > >> # dmm - Device-Mapper Multipath (DMM) > >> > >> # dlmfdrv - Hitachi dlm > >> > >> # hdisk - AIX hard disk > >> > >> # lv - AIX logical volume. Historical usage only. > >> > >> # Not allowed as a new device to mmcrnsd. > >> > >> # gpt - GPFS partition on Windows disk > >> > >> # generic - Device having no unique failover or multipathing > >> > >> # characteristic (predominantly Linux devices). > >> > >> # dasd - DASD device (for Linux on z Systems) > >> > >> > >> > >> We have our storage under Linux Device-Mapper Multipath control (two > device > >> paths to all storage, active/passive) and are accessible under > /dev/mapper, > >> but the NSD types are current set to ‘generic’ not ‘dmm’. This is > >> configured in the /var/mmfs/etc/nsddevices file: > >> > >> > >> > >> if [[ $osName = Linux ]] > >> > >> then > >> > >> : # Add function to discover disks in the Linux environment. > >> > >> ls -l /dev/mpath/ 2>/dev/null | awk '{print "mpath/"$9 " generic"}' > >> > >> ls -l /dev/mapper/ 2>/dev/null | awk '{print "mapper/"$9 " generic"}' > >> > >> ls -1 /dev/vd* 2>/dev/null | awk -F '/' '{print ""$3 " generic"}' > >> > >> fi > >> > >> > >> > >> Can somebody from IBM explain what the correct setting should be and > what > >> differences GPFS does with ‘generic’ vs. ‘dmm’ vs. others? > >> > >> > >> > >> Thanks in advance! > >> > >> -Bryan > >> > >> > >> > >> ________________________________ > >> > >> > >> Note: This email is for the confidential use of the named addressee(s) > only > >> and may contain proprietary, confidential or privileged information. If > you > >> are not the intended recipient, you are hereby notified that any review, > >> dissemination or copying of this email is strictly prohibited, and to > please > >> notify the sender immediately and destroy this email and any > attachments. > >> Email transmission cannot be guaranteed to be secure or error-free. The > >> Company, therefore, does not make any guarantees as to the completeness > or > >> accuracy of this email or any attachments. This email is for > informational > >> purposes only and does not constitute a recommendation, offer, request > or > >> solicitation of any kind to buy, sell, subscribe, redeem or perform any > type > >> of transaction of a financial product. > >> > >> _______________________________________________ > >> gpfsug-discuss mailing list > >> gpfsug-discuss at spectrumscale.org > >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > >> > >> > >> ________________________________ > >> > >> Note: This email is for the confidential use of the named addressee(s) > only > >> and may contain proprietary, confidential or privileged information. If > you > >> are not the intended recipient, you are hereby notified that any review, > >> dissemination or copying of this email is strictly prohibited, and to > please > >> notify the sender immediately and destroy this email and any > attachments. > >> Email transmission cannot be guaranteed to be secure or error-free. The > >> Company, therefore, does not make any guarantees as to the completeness > or > >> accuracy of this email or any attachments. This email is for > informational > >> purposes only and does not constitute a recommendation, offer, request > or > >> solicitation of any kind to buy, sell, subscribe, redeem or perform any > type > >> of transaction of a financial product. > >> > >> _______________________________________________ > >> gpfsug-discuss mailing list > >> gpfsug-discuss at spectrumscale.org > >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > >> > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > ________________________________ > > > > Note: This email is for the confidential use of the named addressee(s) > only and may contain proprietary, confidential or privileged information. > If you are not the intended recipient, you are hereby notified that any > review, dissemination or copying of this email is strictly prohibited, and > to please notify the sender immediately and destroy this email and any > attachments. Email transmission cannot be guaranteed to be secure or > error-free. The Company, therefore, does not make any guarantees as to the > completeness or accuracy of this email or any attachments. This email is > for informational purposes only and does not constitute a recommendation, > offer, request or solicitation of any kind to buy, sell, subscribe, redeem > or perform any type of transaction of a financial product. > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss >
_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
