On Thu, 2017-03-16 at 10:43 -0400, Aaron Knister wrote: > Perhaps an environment where one has OPA and IB fabrics. Taken from here > (https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq.html): > > RDMA is not supported on a node when both Mellanox HCAs and Intel > Omni-Path HFIs are enabled for RDMA. > > The alternative being a situation where multiple IB fabrics exist that > require different OFED versions from each other (and most likely from > ESS) for support reasons (speaking from experience). That is to say if > $VENDOR supports OFED version X on an IB fabric, and ESS/GSS ships with > version Y and there's a problem on the IB fabric $VENDOR may point at > the different OFED version on the ESS/GSS and say they don't support it > and then one is in a bad spot. >
Or just use Ethernet for the GPFS traffic everywhere. It's 2017, 10GbE to compute nodes is cheap enough to be the norm, and you can use 40GbE and 100GbE everywhere else as required, and note unless you are a pure SSD system (which must be vanishingly rare at the moment) the latency is all in the disks anyway. I can't imagine why you would use IB and/or Omni-Path on anything other than compute clusters, so using Ethernet for your storage has advantages too especially if you are using IB. One of the features of Omni-Path over IB is that it will prioritize MPI traffic over storage, but given current core counts in CPU's separating storage out onto Ethernet is not a bad thing anyway as it keeps the MPI bandwidth per core up. My feeling is that in a compute cluster 10Gb for storage is more than enough for a node because much more and it would only take a handful of nodes to denial of service your storage, which is not good. A 500 node compute cluster with 10GbE for storage can in theory try and hit your storage for 600GB/s which very very few storage systems will be able to keep up with. On a storage GPFS cluster then you can up to 40/100GbE and put in multiple links for redundancy which is not possible with IB/Omni-path as far as I am aware which makes it a better solution. Even in a compute cluster with Ethernet I can stick in multiple connections for my NSD nodes for redundancy which is an improvement over the IB/Omni-path options. Especially given in my experience IB links go wonky orders of magnitude more often than Ethernet ones do. Loose a link to a compute node I loose one job on the cluster. Loose a link to the NSD server and I face loosing all the jobs on the cluster... JAB. -- Jonathan A. Buzzard Email: jonathan (at) buzzard.me.uk Fife, United Kingdom. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
