Re: [OMPI users] Q: Basic invoking of InfiniBand with OpenMPI

2017-07-17 Thread Russell Dekema
Since these ports are running in actual Ethernet mode (as opposed to IPoIB), I do not think the interface names will be of the ibN (ib0, ib1, etc) format. It is more likely that the interface names will be of the form ethN or enPApBsCfD. It would be best to check with your system administrator,

Re: [OMPI users] Q: Basic invoking of InfiniBand with OpenMPI

2017-07-17 Thread Boris M. Vulovic
Thanks Gus. I'll try and post results. I am newbie in this and appreciate any advice very much. Cheers --Boris On Mon, Jul 17, 2017 at 10:09 AM, Gus Correa wrote: > On 07/17/2017 01:06 PM, Gus Correa wrote: > >> Hi Boris >> >> The nodes may have standard Gigabit

Re: [OMPI users] Q: Basic invoking of InfiniBand with OpenMPI

2017-07-17 Thread Gus Correa
On 07/17/2017 01:06 PM, Gus Correa wrote: Hi Boris The nodes may have standard Gigabit Ethernet interfaces, besides the Infiniband (RoCE). You may want to direct OpenMPI to use the Infiniband interfaces, not Gigabit Ethernet, by adding something like this to "--mca btl self,vader,self": Oops!

Re: [OMPI users] Q: Basic invoking of InfiniBand with OpenMPI

2017-07-17 Thread Gus Correa
Hi Boris The nodes may have standard Gigabit Ethernet interfaces, besides the Infiniband (RoCE). You may want to direct OpenMPI to use the Infiniband interfaces, not Gigabit Ethernet, by adding something like this to "--mca btl self,vader,self": "--mca btl_tcp_if_include ib0,ib1" (Where the

Re: [OMPI users] Q: Basic invoking of InfiniBand with OpenMPI

2017-07-17 Thread Boris M. Vulovic
Gus, Gilles, Russell, John: Thanks very much for the replies and the help. I got confirmation from the "root" that it is indeed RoCE with 100G. I'll go over the info in the link Russell provided, but have a quick question: if I run the "*mpiexec*" with "*-mca btl tcp,self*" do I get the benefit

Re: [OMPI users] Q: Basic invoking of InfiniBand with OpenMPI

2017-07-17 Thread Russell Dekema
It looks like you have two dual-port Mellanox VPI cards in this machine. These cards can be set to run InfiniBand or Ethernet on a port-by-port basis, and all four of your ports are set to Ethernet mode. Two of your ports have active 100 gigabit Ethernet links, and the other two have no link up at

Re: [OMPI users] Q: Basic invoking of InfiniBand with OpenMPI

2017-07-17 Thread Gilles Gouaillardet
Boris, these logs seem a bit odd to me. as far as i remember, the state is POLLING when there is no subnet manager. and when there is one, the state is ACTIVE *but* both Base and SM lid are non zero btw, is IPoIB configured ? if yes, then can your hosts ping each other with this interface. i

Re: [OMPI users] Q: Basic invoking of InfiniBand with OpenMPI

2017-07-17 Thread John Hearns via users
Boris, do you have a Subnet Manager running on your fabric? I am sorry if there have bene other replies ot this over the weekend. On 14 July 2017 at 18:34, Boris M. Vulovic wrote: > Gus, Gilles and John, > > Thanks for the help. Let me first post (below) the output