When using “mpirun” we can specify “-iface ib0” this is true, and the exact syntax depends on your MPI of choice, as noted above.
However, don't get confused between IPOIB and Infiniband itself. IPOIB is of course sending IP traffic over Infiniband. An Infiniband network can perfectly happily function without any IP addresses being assigned. The point I am getting at is that the IP connections should be used to set up/launch the job, depending on the launcher used by that MPI (eg Hydra as above) What I am saying is do not get confused between the activity which sets up the MPI processes on the remote nodes, and the actual MPI traffic. Unless you really want to use IPOIB for your MPI traffic (maybe for doing a benchmark comparison) I would say just run the srun with the ethernet IPs and let your MPI choose the best bit transport layer (BTL in OpenMPI speak). What I would do is tag the Infiniband equipped nodes with a feature called 'IB' or 'nonIB' for the others, and choose those nodes. (Sorry - my head is in PBSPro world these days so that would be a resources_available in that world) My advice - schedule just on the IB equipped nodes. Run your mpi with a verbose flag and see which BTL it is choosing. You may be pleasantly surprised! I would say 'take down the ib0 interface' but that may be a bad move - it is probably used for storage mounts at least. If I have misunderstood the point, and have been a bit rude here I apologise in advance. Someone with a clue will come along and slap me rounfd the head I am sure. On 25 October 2017 at 17:03, Le Biot, Pierre-Marie < pierre-marie.leb...@hpe.com> wrote: > Hi Sebastian, > > > > Another solution could be to change the configuration of nodes in > slurm.conf, making use of NodeName and NodeHostname (and NodeAddr if > needed) : > > > > “ > > NodeName > > Name that Slurm uses to refer to a node[...]. Typically this would be the > string that "/bin/hostname -s" returns.[...]It may also be an arbitrary > string if NodeHostname is specified.[...] > > > > NodeHostname > > Typically this would be the string that "/bin/hostname -s" returns.[...]By > default, the NodeHostname will be identical in value to NodeName. > > > > NodeAddr > > Name that a node should be referred to in establishing a communications > path.[...] NodeAddr may also contain IP addresses. By default, the NodeAddr > will be identical in value to NodeHostname. > > “ > > > > For the nodes having an infiniband interface declare the associated name > in NodeName and the regular hostname in NodeHostname. > > SLURM_NODELIST will contain the names declared in NodeName. > > > > Regards, > > Pierre-Marie Le Biot > > > > *From:* Sebastian Eastham [mailto:seast...@mit.edu] > *Sent:* Tuesday, October 24, 2017 10:02 PM > *To:* slurm-dev <slurm-dev@schedmd.com> > *Subject:* [slurm-dev] Selecting a network interface with srun > > > > Dear Slurm Developers mailing list, > > > > When calling the “srun” command, is there any way to specify the desired > network interface? Our network is a mix of ethernet and inifiniband, such > that only a subset of the nodes have an infiniband interface. When using > “mpirun” we can specify “-iface ib0”, but there does not appear to be a > similar option for “srun”. Although we can successfully run our > applications with “srun”, we can see from “iftop” that the application is > communicating purely through the ethernet interface. > > > > Once again, I appreciate any help or guidance that you can give me! > > > > Regards, > > > > Seb > > > > =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= > > Dr. Sebastian D. Eastham > > Research Scientist > > Laboratory for Aviation and the Environment > > Massachusetts Institute of Technology > > =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= > > > > >
