Re: [OMPI users] Whether to use the IB BTL or not

[Jeff Squyres (jsquyres)]

In addition to what Howard said, there's actually two other metrics that are 
used, as well.  Each BTL exports a priority and an exclusivity value.

IIRC (it's been a while since I've looked at this code), the PML gathers up all 
BTL modules that claim that they can communicate between a pair of peers, and 
uses all the modules that share the same, highest exclusivity rating.

TCP has a lower exclusivity rating than OS-bypass BTLs (such as usnic and 
openib).  Hence, even if TCP is available between a pair of peers, OS-bypass 
transports will be preferred over TCP.



On Dec 23, 2014, at 4:04 PM, Howard Pritchard  wrote:

> HI Gary,
> 
> The decision occurs within the MPI processes themselves (during the call to 
> MPI_Init) - so
> after the orte daemons have started on the nodes.
> 
> The BTL's report their "latency" and "bandwidth" - up the stack to the 
> PML/BML layer which
> then decides based on these metrics which BTL to use to send/recv messages 
> between
> any given pair of MPI ranks. 
> 
> Hope this helps,
> 
> Howard
> 
> 
> 
> 
> 2014-12-23 13:29 GMT-07:00 Gary Jackson :
> 
> I'm not having any trouble getting it to start, and it's definitely using the 
> openib btl. I was just wondering how it decided whether the openib btl was 
> appropriate before going down the btl list to tcp when all mpirun gets is a 
> hostname and no other information about connectivity on the remote end. For 
> instance, is this determined before or after orted runs on the remote end?
> 
> On 12/23/14, 2:18 PM, Howard Pritchard wrote:
> Hello Gary,
> 
> It depends on how the Open MPI was built, and on mca parameters passed
> to the job either via settings in an mca params conf
> file or the mpirun command line or env. variables.  If you have mxm
> (MLNX) or psm (qlogic/intel) installed on the system
> where your open mpi was built, you may actually be using one of those
> via the MTL path.
> 
> Try
> 
> mpirun -np 2 -H hosta,hostb -mca btl self,vader,openib ./your_favorite_test
> 
> That should force open mpi to try using openib between the pair of
> hosts.  Note you don't need "vader" on the command line
> if you are running only one mpi rank/node.
> 
> Howard
> 
> 
> 
> 
> 2014-12-23 11:48 GMT-07:00 Gary Jackson  >:
> 
> 
> How does OpenMPI decide whether to use the IB BTL between a given
> pair of hosts, assuming there is an IB interface available?
> 
> --
> Gary
> _
> users mailing list
> us...@open-mpi.org 
> Subscription: http://www.open-mpi.org/__mailman/listinfo.cgi/users
> 
> Link to this post:
> http://www.open-mpi.org/__community/lists/users/2014/12/__26063.php
> 
> 
> 
> 
> 
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2014/12/26064.php
> 
> 
> 
> -- 
> Gary
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2014/12/26065.php
> 
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2014/12/26066.php


-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to: 
http://www.cisco.com/web/about/doing_business/legal/cri/

Re: [OMPI users] Whether to use the IB BTL or not

[Howard Pritchard]

HI Gary,

The decision occurs within the MPI processes themselves (during the call to
MPI_Init) - so
after the orte daemons have started on the nodes.

The BTL's report their "latency" and "bandwidth" - up the stack to the
PML/BML layer which
then decides based on these metrics which BTL to use to send/recv messages
between
any given pair of MPI ranks.

Hope this helps,

Howard




2014-12-23 13:29 GMT-07:00 Gary Jackson :

>
> I'm not having any trouble getting it to start, and it's definitely using
> the openib btl. I was just wondering how it decided whether the openib btl
> was appropriate before going down the btl list to tcp when all mpirun gets
> is a hostname and no other information about connectivity on the remote
> end. For instance, is this determined before or after orted runs on the
> remote end?
>
> On 12/23/14, 2:18 PM, Howard Pritchard wrote:
>
>> Hello Gary,
>>
>> It depends on how the Open MPI was built, and on mca parameters passed
>> to the job either via settings in an mca params conf
>> file or the mpirun command line or env. variables.  If you have mxm
>> (MLNX) or psm (qlogic/intel) installed on the system
>> where your open mpi was built, you may actually be using one of those
>> via the MTL path.
>>
>> Try
>>
>> mpirun -np 2 -H hosta,hostb -mca btl self,vader,openib
>> ./your_favorite_test
>>
>> That should force open mpi to try using openib between the pair of
>> hosts.  Note you don't need "vader" on the command line
>> if you are running only one mpi rank/node.
>>
>> Howard
>>
>>
>>
>>
>> 2014-12-23 11:48 GMT-07:00 Gary Jackson > >:
>>
>>
>> How does OpenMPI decide whether to use the IB BTL between a given
>> pair of hosts, assuming there is an IB interface available?
>>
>> --
>> Gary
>> _
>> users mailing list
>> us...@open-mpi.org 
>> Subscription: http://www.open-mpi.org/__mailman/listinfo.cgi/users
>> 
>> Link to this post:
>> http://www.open-mpi.org/__community/lists/users/2014/12/__26063.php
>> 
>>
>>
>>
>>
>> ___
>> users mailing list
>> us...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> Link to this post: http://www.open-mpi.org/community/lists/users/2014/12/
>> 26064.php
>>
>>
>
> --
> Gary
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: http://www.open-mpi.org/community/lists/users/2014/12/
> 26065.php
>

Re: [OMPI users] Whether to use the IB BTL or not

[Gary Jackson]

I'm not having any trouble getting it to start, and it's definitely 
using the openib btl. I was just wondering how it decided whether the 
openib btl was appropriate before going down the btl list to tcp when 
all mpirun gets is a hostname and no other information about 
connectivity on the remote end. For instance, is this determined before 
or after orted runs on the remote end?


On 12/23/14, 2:18 PM, Howard Pritchard wrote:

Hello Gary,

It depends on how the Open MPI was built, and on mca parameters passed
to the job either via settings in an mca params conf
file or the mpirun command line or env. variables.  If you have mxm
(MLNX) or psm (qlogic/intel) installed on the system
where your open mpi was built, you may actually be using one of those
via the MTL path.

Try

mpirun -np 2 -H hosta,hostb -mca btl self,vader,openib ./your_favorite_test

That should force open mpi to try using openib between the pair of
hosts.  Note you don't need "vader" on the command line
if you are running only one mpi rank/node.

Howard




2014-12-23 11:48 GMT-07:00 Gary Jackson >:


How does OpenMPI decide whether to use the IB BTL between a given
pair of hosts, assuming there is an IB interface available?

--
Gary
_
users mailing list
us...@open-mpi.org 
Subscription: http://www.open-mpi.org/__mailman/listinfo.cgi/users

Link to this post:
http://www.open-mpi.org/__community/lists/users/2014/12/__26063.php





___
users mailing list
us...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post: 
http://www.open-mpi.org/community/lists/users/2014/12/26064.php




--
Gary

Re: [OMPI users] Whether to use the IB BTL or not

[Howard Pritchard]

Hello Gary,

It depends on how the Open MPI was built, and on mca parameters passed to
the job either via settings in an mca params conf
file or the mpirun command line or env. variables.  If you have mxm (MLNX)
or psm (qlogic/intel) installed on the system
where your open mpi was built, you may actually be using one of those via
the MTL path.

Try

mpirun -np 2 -H hosta,hostb -mca btl self,vader,openib ./your_favorite_test

That should force open mpi to try using openib between the pair of hosts.
Note you don't need "vader" on the command line
if you are running only one mpi rank/node.

Howard




2014-12-23 11:48 GMT-07:00 Gary Jackson :

>
> How does OpenMPI decide whether to use the IB BTL between a given pair of
> hosts, assuming there is an IB interface available?
>
> --
> Gary
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: http://www.open-mpi.org/community/lists/users/2014/12/
> 26063.php
>