Hi, Thx for your answer!
Particularly the explication between connected and datagram mode (I see that with the IMB1 benchmarks of mpi)! The hardware we are using in details: - on the master: Mellanox MHGH18-XTC ConnectX with VPI adapter, single port 20Gb/s, PCIe2.0 x8 2.5GT/s - on the nodes: Integrated Mellanox DDR Infiniband 20Gbs ConnectX with QSFP Connector. How can I know the limit of the MTU size ? On the Infiniband we are just using mpi with different CFD programs. But always with mpi (intel mpi or openmpi). Sould I use QoS ? Thx for your help! Le 07/12/2010 16:09, Richard Croucher a écrit : > Connected mode will provide more throughput but datagram mode will provide > lower latency. > You don't say what HCA's you are using. Some of the optimizations for > Connected mode are only available for the newer ConnectX QDR HCA's. > > Your HCA will probably limit the MTU size. Leave this as large as possible. > > If you are only running a single application on the InfiniBand you need not > bother with QoS. If you are running multiple, then you do need to set > this. This is quite complex since you need to define V'L's, their > arbitration policies and assign SL's to them. This is described in the > OpenSM docs. This is relevant even if you are using the embedded SM in the > switch. > > AS a newbie, take a look in the ../OFED/docs > There is probably all you need there. Mellanox also have some useful docs on > their website. > > -----Original Message----- > From: [email protected] > [mailto:[email protected]] On Behalf Of giggzounet > Sent: 07 December 2010 14:01 > To: [email protected] > Subject: [ewg] Need help for Infiniband optimisation for our cluster > (MTU...) > > Hi, > > I'm new on this list. We have in our laboratory a little cluster: > - master 8 cores > - 8 nodes with 12 cores > - DDR infiniband switch Mellanox MTS3600R > > On these machines we have an oscar cluster with CentOS 5.5. We have > installed the ofed packages 1.5.1. The default config for the infiniband > is used. So infiniband is running in connected mode. > > Our cluster is used to solve CFD (Computational Fluid Dynamics) > problems. And I'm trying to optimize the infiniband network and so I > have several questions: > > - Is it the right mailing list to ask ? (if not...where should I post ?) > > - Is there a how-to on infiniband optimisation ? > > - CFD computations need a lot of bandwidth. There are a lot of data > exchange through MPI (we are using intel mpi). Has the infiniband mode > (connected or datagram) influence in this case ? What is the "best" MTU > for those computation ? > > > Best regards, > Guillaume > > _______________________________________________ > ewg mailing list > [email protected] > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg _______________________________________________ ewg mailing list [email protected] http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
