Re: [OMPI users] libfabric verb provider for iWARP RNIC

[dpchoudh .]

For the sake of completeness, let me put the answer here. I posted the
question on libfabric mailing list and with their input, installed
librdmacm-devel. After that and a rebuild, the issue went away.

Thanks
Durga

We learn from history that we never learn from history.

On Mon, Apr 4, 2016 at 3:20 PM, dpchoudh .  wrote:

> Hi Howard
>
> Thank you very much for your suggestions. All the installation location in
> my case are the default ones, so that is likely not the issue.
>
> What I find a bit confusing is this:
>
> As I mentioned, my cluster has both Qlogic Infiniband and Chelsio iWARP
> (which are exposed to OpenMPI natively as well as an IP interface)
>
> With this configuration, if I build libfabric with configure options
> --with-psm=auto --with-verbs=auto, as I mentioned earlier, only the PSM
> interface shows up in the fi_info listing, and OpenMPI programs using ofi
> MTL *do* work. However, I do not know if the traffic is going through the
> Qlogic card or the Chelsio card; it is likely the former.
>
> I am going to ask this on the libfabric list, but perhaps the following
> question is relevant in the OpenMPI list:
>
> My understanding about the OFI MTL is the following: please correct me
> where I am wrong:
> As I understand, ANY type of transport that exposes a verbs interface
> (iWARP, RoCE, Infiniband from any manufacturer) can become a libfabric
> provider (when libfabric is compiled with --with-verbs option) and thus
> support the OFI MTL (and thus the cm PML?)
>
> Is the above true?
>
> Best regards
> Durga
>
> We learn from history that we never learn from history.
>
> On Mon, Apr 4, 2016 at 7:29 AM, Howard Pritchard 
> wrote:
>
>> Hi Durga,
>>
>> I'd suggest reposting this to the libfabric-users mail list.
>> You can join that list at
>> http://lists.openfabrics.org/mailman/listinfo/libfabric-users
>>
>> I'd suggest including the output of config.log.  If you installed
>> ofed in non-canonical location, you may need to give an explicit
>> path as an argument to the --enable-verbs configury option.
>>
>> Note if you're trying to use libfabric with the Open MPI ofi
>> mtl, you will need to get literally the freshest version of
>> libfabric, either at github or the 1.3rc2 tarball at
>>
>> http://www.openfabrics.org/downloads/ofi/
>>
>> Good luck,
>>
>> Howard
>>
>>
>> 2016-04-02 13:41 GMT-06:00 dpchoudh . :
>>
>>> Hello all
>>>
>>> My machine has 3 network cards:
>>>
>>> 1. Broadcom GbE (vanilla type, with some offload capability)
>>> 2. Chelsion S310 10Gb iWARP
>>> 3. Qlogic DDR 4X Infiniband.
>>>
>>> With this setup, I built libfabric like this:
>>>
>>> ./configure --enable-udp=auto --enable-gni=auto --enable-mxm=auto
>>> --enable-usnic=auto --enable-verbs=auto --enable-sockets=auto
>>> --enable-psm2=auto --enable-psm=auto && make && sudo make install
>>>
>>> However, in the built libfabric, I do not see a verb provider, which I'd
>>> expect for the iWARP card, at least.
>>>
>>> [durga@smallMPI libfabric]$ fi_info
>>> psm: psm
>>> version: 0.9
>>> type: FI_EP_RDM
>>> protocol: FI_PROTO_PSMX
>>> UDP: UDP-IP
>>> version: 1.0
>>> type: FI_EP_DGRAM
>>> protocol: FI_PROTO_UDP
>>> sockets: IP
>>> version: 1.0
>>> type: FI_EP_MSG
>>> protocol: FI_PROTO_SOCK_TCP
>>> sockets: IP
>>> version: 1.0
>>> type: FI_EP_DGRAM
>>> protocol: FI_PROTO_SOCK_TCP
>>> sockets: IP
>>> version: 1.0
>>> type: FI_EP_RDM
>>> protocol: FI_PROTO_SOCK_TCP
>>>
>>>
>>> Am I doing something wrong or misunderstanding how libfabric works?
>>>
>>> Thanks in advance
>>> Durga
>>>
>>> We learn from history that we never learn from history.
>>>
>>> ___
>>> users mailing list
>>> us...@open-mpi.org
>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>>> Link to this post:
>>> http://www.open-mpi.org/community/lists/users/2016/04/28870.php
>>>
>>
>>
>> ___
>> users mailing list
>> us...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> Link to this post:
>> http://www.open-mpi.org/community/lists/users/2016/04/28884.php
>>
>
>

Re: [OMPI users] resolution of MPI_Wtime

[George Bosilca]

In opal/mca/linux/timer_linux_component.c. The timer is a special component
which is statically included during the build process (via
the MCA_timer_IMPLEMENTATION_HEADER define in opal/mca/timer/base/base.h
line 48). Thus the symbol should appear directly in the libmpi.a

  george.


On Tue, Apr 5, 2016 at 12:31 PM, Dave Love  wrote:

> Aurélien Bouteiller  writes:
>
> > Open MPI uses clock_gettime when it is available, and defaults to
> > gettimeofday only when this better option can't be found. Check that
> > your system has clock_gettime and the resolution of this timer.
>
> That's what I thought after I raised this originally, but where
> (assuming I'm not using these components)?
>
>   openmpi-1.10.2]$ grep -r -l clock_gettime | egrep -v /libevent\|/vt/
>   ompi/mca/common/cuda/common_cuda.c
>   ompi/mca/btl/scif/btl_scif_get.c
>   ompi/mca/btl/scif/btl_scif_send.c
>   ompi/mca/btl/scif/btl_scif_put.c
>   ompi/mca/btl/scif/btl_scif.h
>
> This is on RHEL6, but I doubt there's a current distribution without
> clock_gettime.
>
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2016/04/28890.php
>

Re: [OMPI users] resolution of MPI_Wtime

[Dave Love]

Aurélien Bouteiller  writes:

> Open MPI uses clock_gettime when it is available, and defaults to
> gettimeofday only when this better option can't be found. Check that
> your system has clock_gettime and the resolution of this timer.

That's what I thought after I raised this originally, but where
(assuming I'm not using these components)?

  openmpi-1.10.2]$ grep -r -l clock_gettime | egrep -v /libevent\|/vt/
  ompi/mca/common/cuda/common_cuda.c
  ompi/mca/btl/scif/btl_scif_get.c
  ompi/mca/btl/scif/btl_scif_send.c
  ompi/mca/btl/scif/btl_scif_put.c
  ompi/mca/btl/scif/btl_scif.h

This is on RHEL6, but I doubt there's a current distribution without
clock_gettime.

Re: [OMPI users] resolution of MPI_Wtime

[Mark Santcroos]

> On 05 Apr 2016, at 16:46 , Aurélien Bouteiller  wrote:
> Open MPI uses clock_gettime when it is available, and defaults to 
> gettimeofday only when this better option can't be found. Check that your 
> system has clock_gettime and the resolution of this timer. 

Depending on what you mean, I don't this is universally true, as I recently ran 
into this issue in another part of the code.
AFAIK, some parts of the code have a macro that check for clock_gettime and use 
it, but in other places there is just direct usage of gettimeofday.

If you commented on MPI_Wtime specifically, then I don't know :-)

Mark

Re: [OMPI users] resolution of MPI_Wtime

[Aurélien Bouteiller]

Open MPI uses clock_gettime when it is available, and defaults to gettimeofday 
only when this better option can't be found. Check that your system has 
clock_gettime and the resolution of this timer. 

Aurélien 
--
Aurélien Bouteiller, Ph.D. ~~ https://icl.cs.utk.edu/~bouteill/

> Le 5 avr. 2016 à 10:29, Dave Love  a écrit :
> 
> I can't immediately find a reference, but I thought it had been agreed
> some time ago that MPI_Wtime should use high resolution clocks where
> they're available.  However, 1.10 is still using gettimeofday on
> GNU/Linux.  Is there a good reason for that?
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2016/04/28887.php



smime.p7s
Description: S/MIME cryptographic signature

[OMPI users] resolution of MPI_Wtime

[Dave Love]

I can't immediately find a reference, but I thought it had been agreed
some time ago that MPI_Wtime should use high resolution clocks where
they're available.  However, 1.10 is still using gettimeofday on
GNU/Linux.  Is there a good reason for that?