Hi, Having mentioned nsddevices, I do not know how Scale treats different device types differently, so generic would be a fine choice unless development tells you differently. Currently known device types are listed in the comments of the script /usr/lpp/mmfs/bin/mmdevdiscover
Mit freundlichen Grüßen / Kind regards Dr. Uwe Falke IT Specialist High Performance Computing Services / Integrated Technology Services / Data Center Services ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Rathausstr. 7 09111 Chemnitz Phone: +49 371 6978 2165 Mobile: +49 175 575 2877 E-Mail: [email protected] ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Business & Technology Services GmbH / Geschäftsführung: Thomas Wolter, Sven Schooß Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 17122 From: Uwe Falke/Germany/IBM To: gpfsug main discussion list <[email protected]> Date: 16/11/2018 10:19 Subject: Re: [gpfsug-discuss] How to use RHEL 7 mdadm NVMe devices with Spectrum Scale 4.2.3.10? Hi Lance, you might need to use /var/mmfs/etc/nsddevices to tell GPFS about these devices (template in /usr/lpp/mmfs/samples/nsddevices.sample) Mit freundlichen Grüßen / Kind regards Dr. Uwe Falke IT Specialist High Performance Computing Services / Integrated Technology Services / Data Center Services ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Rathausstr. 7 09111 Chemnitz Phone: +49 371 6978 2165 Mobile: +49 175 575 2877 E-Mail: [email protected] ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Business & Technology Services GmbH / Geschäftsführung: Thomas Wolter, Sven Schooß Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 17122 From: Lance Nakata <[email protected]> To: [email protected] Cc: "Jon L. Bergman" <[email protected]> Date: 16/11/2018 04:14 Subject: [gpfsug-discuss] How to use RHEL 7 mdadm NVMe devices with Spectrum Scale 4.2.3.10? Sent by: [email protected] We have a Dell R740xd with 24 x 1TB NVMe SSDs in the internal slots. Since PERC RAID cards don't see these devices, we are using mdadm software RAID to build NSDs. We took 12 NVMe SSDs and used mdadm to create a 10 + 1 + 1 hot spare RAID 5 stripe named /dev/md101. We took the other 12 NVMe SSDs and created a similar /dev/md102. mmcrnsd worked without errors. The problem is that Spectrum Scale does not see the /dev/md10x devices as proper NSDs; the Device and Devtype columns are blank: host2:~> sudo mmlsnsd -X Disk name NSD volume ID Device Devtype Node name Remarks --------------------------------------------------------------------------------------------------- nsd0001 864FD12858A36E79 /dev/sdb generic host1.slac.stanford.edu server node nsd0002 864FD12858A36E7A /dev/sdc generic host1.slac.stanford.edu server node nsd0021 864FD1285956B0A7 /dev/sdd generic host1.slac.stanford.edu server node nsd0251a 864FD1545BD0CCDF /dev/dm-9 dmm host2.slac.stanford.edu server node nsd0251b 864FD1545BD0CCE0 /dev/dm-11 dmm host2.slac.stanford.edu server node nsd0252a 864FD1545BD0CCE1 /dev/dm-10 dmm host2.slac.stanford.edu server node nsd0252b 864FD1545BD0CCE2 /dev/dm-8 dmm host2.slac.stanford.edu server node nsd02nvme1 864FD1545BEC5D72 - - host2.slac.stanford.edu (not found) server node nsd02nvme2 864FD1545BEC5D73 - - host2.slac.stanford.edu (not found) server node I know we can access the internal NVMe devices by their individual /dev/nvmeXX paths, but non-ESS-based Spectrum Scale does not have built-in RAID functionality. Hence, the only option in that scenario is replication, which is expensive and won't give us enough usable space. Software Environment: RHEL 7.6 with kernel 3.10.0-862.14.4.el7.x86_64 Spectrum Scale 4.2.3.10 Spectrum Scale Support has implied we can't use mdadm for NVMe devices. Is that really true? Does anyone use an mdadm-based NVMe config? If so, did you have to do some kind of customization to get it working? Thank you, Lance Nakata SLAC National Accelerator Laboratory _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
