On 12/6/16 11:36 AM, Sven Oehme wrote: i am not sure i understand your comment with 'persistent' do you mean when you create a nsddevice on a nvme device it won't get recognized after a restart ? yes /dev/sdX may change after a reboot especially if you add devices. using udev rules makes sure the device is always the same. if thats what you mean there are 2 answers , short term you need to add a /var/mmfs/etc/nsddevices script to your node that simply adds an echo for the nvme device like :
echo nvme0n1 generic this will tell the daemon to include that device on top of all other discovered devices that we include by default (like dm-* , sd*, etc) the longer term answer is that we have a tracking item to ad nvme* to the automatically discovered devices. yes that is what I meant by modifying mmdevdiscover on your second question, given that GPFS does workload balancing across devices you don't want to add extra complexity and path length to anything , so stick with raw devices . K that is what I was thinking. sven On Tue, Dec 6, 2016 at 8:40 AM Matt Weil <[email protected]<mailto:[email protected]>> wrote: Hello all, Thanks for sharing that. I am setting this up on our CES nodes. In this example the nvme devices are not persistent. RHEL's default udev rules put them in /dev/disk/by-id/ persistently by serial number so I modified mmdevdiscover to look for them there. What are others doing? custom udev rules for the nvme devices? Also I have used LVM in the past to stitch multiple nvme together for better performance. I am wondering in the use case with GPFS that it may hurt performance by hindering the ability of GPFS to do direct IO or directly accessing memory. Any opinions there? Thanks Matt On 12/5/16 10:33 AM, Ulf Troppens wrote: FYI ... in case not seen .... benchmark for LROC with NVMe http://www.intel.com/content/dam/www/public/us/en/documents/white-papers/performance-gains-ibm-spectrum-scale.pdf -- IBM Spectrum Scale Development - Client Engagements & Solutions Delivery Consulting IT Specialist Author "Storage Networks Explained" IBM Deutschland Research & Development GmbH Vorsitzende des Aufsichtsrats: Martina Koederitz Geschäftsführung: Dirk Wittkopp Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org<http://spectrumscale.org> http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org<http://spectrumscale.org> http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.
_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
