On Fri, 2016-03-11 at 13:19 +0000, Sean Killen wrote: > Hi all, > > So I have finally got my SpectrumScale system installed (well half of > it). But it wasn't without some niggles. > > We have purchased DELL MD3860i disk trays with dual controllers (each > with 2x 10Gbit NICs), to Linux this appears as 4 paths, I spent quite a > while getting a nice multipath setup in place with 'friendly' names set >
Oh dear. I guess it might work with 10Gb Ethernet but based on my personal experience iSCSI is spectacularly unsuited to GPFS. Either your NSD nodes can overwhelm the storage arrays or the storage arrays can overwhelm the NSD servers and performance falls through the floor. That is unless you have Data Center Ethernet at which point you might as well have gone Fibre Channel in the first place. Though unless you are going to have large physical separation between the storage and NSD servers 12Gb SAS is a cheaper option and you can still have four NSD servers hooked up to each MD3 based storage array. I have in the past implement GPFS on Dell MD3200i's. I did eventually get it working reliably but it was so suboptimal with so many compromises that as soon as the MD3600f came out we purchased these to replaced the MD3200i's. Lets say you have three storage arrays with two paths to each controller and four NSD servers. Basically what happens is that an NSD server issues a bunch of requests for blocks to the storage arrays. Then all 12 paths start answering to your two connections to the NSD server. At this point the Ethernet adaptors on your NSD servers are overwhelmed 802.1D PAUSE frames start being issued which just result in head of line blocking and performance falls through the floor. You need Data Center Ethernet to handle this properly, which is probably why FCoE never took off as you can't just use the Ethernet switches and adaptors you have. Both FC and SAS handle this sort of congestion gracefully unlike ordinary Ethernet. Now the caveat for all this is that it is much easier to overwhelm a 1Gbps link than a 10Gbps link. However with the combination of SSD and larger cache's I can envisage that a 10Gbps link could be overwhelmed and you would then see the same performance issues that I saw. Basically the only way out is a one to one correspondence between ports on the NSD's and the storage controllers. JAB. -- Jonathan A. Buzzard Email: jonathan (at) buzzard.me.uk Fife, United Kingdom. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
