On Mon, Mar 28, 2011 at 09:36:43AM -0400, Jason Hill wrote: > Hi Brian, > > Here at ORNL we don't sepearate the IB SAN from the Lustre fabric
I misspoke yesterday. Our OSSes have a dual-port card used for storage, and a single port card used for LNET. The storage has half of its ports connected to the in-rack switch, and the other half are directly connected to the OSS via one port on the dual port card, with the other going to the switch. This is a legacy of risk mitigation during acceptance, but as the storage traffic is local it does not traverse the uplinks to the core network, so it does not congest/interfere with the LNET traffic. As others have said the things you have to be aware of are what failure modes in your network affect your connections to the storage and if any of the MPI traffic from a compute cluster could interfere with your connections to the storage as well. I would definately suggest putting your LNET and SRP connections on different physical HCA's to keep the traffic at least isolated on the OSS side. -- -Jason _______________________________________________ Lustre-discuss mailing list [email protected] http://lists.lustre.org/mailman/listinfo/lustre-discuss
