Re: [OMPI users] EXTERNAL: Re: What's the status of OpenMPI and thread safety?

[Ralph Castain]

Hi Ed

FWIW: Intel MPI has better thread support, though you'll lose some features. I 
don't know what NICs you have, but have you tried running with the MTL's 
instead of the openib BTL? Both the psm and mxm MTLs are supposed to be thread 
safe, and will outperform the openib BTL anyway.

Not sure if Intel supports mxm or not - should handle psm as Intel now owns 
Truescale.

HTH
Ralph

On Dec 19, 2013, at 7:23 AM, Blosch, Edwin L  wrote:

> Thanks Ralph,
>  
> We are attempting to use 1.6.4 with an application that requires 
> multi-threading, and it is hanging most of the time; it is using openib.  
> They steered us to try Intel MPI for now.  If you lack drivers/testers for 
> improved thread safety on openib, let me know and I’ll encourage the 
> developers of the application to support you.
>  
> Ed
>  
> From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Ralph Castain
> Sent: Wednesday, December 18, 2013 6:50 PM
> To: Open MPI Users
> Subject: EXTERNAL: Re: [OMPI users] What's the status of OpenMPI and thread 
> safety?
>  
> This was, in fact, a primary point of discussion at last week's OMPI 
> developer's conference. Bottom line is that we are only a little further 
> along than we used to be, but are focusing on improving it. You'll find good 
> thread support for some transports (some of the MTLs and at least the TCP 
> BTL), not so good for others (e.g., openib is flat-out not thread safe).
>  
>  
> On Dec 18, 2013, at 3:57 PM, Blosch, Edwin L  wrote:
> 
> 
> I was wondering if the FAQ entry below is considered current opinion or 
> perhaps a little stale.  Is multi-threading still considered to be ‘lightly 
> tested’?  Are there known open bugs?
>  
> Thank you,
>  
> Ed
>  
>  
> 7. Is Open MPI thread safe?
>  
> Support for MPI_THREAD_MULTIPLE (i.e., multiple threads executing within the 
> MPI library) and asynchronous message passing progress (i.e., continuing 
> message passing operations even while no user threads are in the MPI library) 
> has been designed into Open MPI from its first planning meetings.
>  
> Support for MPI_THREAD_MULTIPLE is included in the first version of Open MPI, 
> but it is only lightly tested and likely still has some bugs. Support for 
> asynchronous progress is included in the TCP point-to-point device, but it, 
> too, has only had light testing and likely still has bugs.
>  
> Completing the testing for full support of MPI_THREAD_MULTIPLE and 
> asynchronous progress is planned in the near future.
>  
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>  
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] EXTERNAL: Re: What's the status of OpenMPI and thread safety?

[Blosch, Edwin L]

Thanks Ralph,

We are attempting to use 1.6.4 with an application that requires 
multi-threading, and it is hanging most of the time; it is using openib.  They 
steered us to try Intel MPI for now.  If you lack drivers/testers for improved 
thread safety on openib, let me know and I'll encourage the developers of the 
application to support you.

Ed

From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Ralph Castain
Sent: Wednesday, December 18, 2013 6:50 PM
To: Open MPI Users
Subject: EXTERNAL: Re: [OMPI users] What's the status of OpenMPI and thread 
safety?

This was, in fact, a primary point of discussion at last week's OMPI 
developer's conference. Bottom line is that we are only a little further along 
than we used to be, but are focusing on improving it. You'll find good thread 
support for some transports (some of the MTLs and at least the TCP BTL), not so 
good for others (e.g., openib is flat-out not thread safe).


On Dec 18, 2013, at 3:57 PM, Blosch, Edwin L 
> wrote:


I was wondering if the FAQ entry below is considered current opinion or perhaps 
a little stale.  Is multi-threading still considered to be 'lightly tested'?  
Are there known open bugs?

Thank you,

Ed


7. Is Open MPI thread safe?

Support for MPI_THREAD_MULTIPLE (i.e., multiple threads executing within the 
MPI library) and asynchronous message passing progress (i.e., continuing 
message passing operations even while no user threads are in the MPI library) 
has been designed into Open MPI from its first planning meetings.

Support for MPI_THREAD_MULTIPLE is included in the first version of Open MPI, 
but it is only lightly tested and likely still has some bugs. Support for 
asynchronous progress is included in the TCP point-to-point device, but it, 
too, has only had light testing and likely still has bugs.

Completing the testing for full support of MPI_THREAD_MULTIPLE and asynchronous 
progress is planned in the near future.

___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] [EXTERNAL] Re: What's the status of OpenMPI and thread safety?

[Barrett, Brian W]

Pablo -

As Ralph mentioned, it will be different, possibly not for the better, in
1.7.  This is an area of active work, so any help would be appreciated.

However, the one issue you brought up is going to be problematic, even
with threads.  Our design essentially makes it such that blocking MPI
calls never block internally (for any thread level).  It's one of the
trade-offs in our multi-device design.  Good: multi-device just works
without any complicated state sharing between devices.  Bad: it's hard for
us to block.  We've talked about making a blocking (or slow polling,
semi-blocking) option for when we can detect we only have one device, but
it hasn't been a high priority.

Like Ralph said, if you're interested in working on the threading or
blocking issues, please join the devel list and let us know.  We're always
willing to take new patches.

Thanks,

Brian

On 12/19/13 5:34 AM, "Pablo Barrio"  wrote:

>Hi all, this is the first time I post to the list (although I have read
>it for a while now). I hope this helps.
>
>I'm heavily using MPI_THREAD_MULTIPLE on multicores (sm BTL) and my
>programs work fine from a CORRECTNESS point of view. I use OpenMPI 1.6
>(SVN rev. 26429) and pthreads on Linux.
>
>This said, the performance is still very poor. Some of my programs become
>a thousand times slower. After some profiling/tracing, I found out that
>the Linux scheduler gave CPU time to threads stuck in blocking calls
>(Ssend, Recv, Wait, etcetera). It seems to
> me that the MPI implementation can be improved to avoid spending CPU
>time in threads waiting for messages.
>
>In short, my experience is that the implementation is correct but not
>very efficient so far.
>
>I have a few questions:
>
>1. My OpenMPI version is more than a year old. Have these performance
>issues been fixed in the latest versions?
>
>2. If not, perhaps I could contribute to OpenMPI multithreading
>support. Who takes care of this? How can I help?
>
>Thanks ahead.
>-- 
>Pablo Barrio
>Dpt. Electrical Engineering - Technical University of Madrid
>Office C-203
>Avda. Complutense s/n, 28040 Madrid
>Tel. (+34) 915495700 ext. 4234
>@: pbar...@die.upm.es
>On 19/12/13 01:49, Ralph Castain wrote:
>
>
>This was, in fact, a primary point of discussion at last week's OMPI
>developer's conference. Bottom line is that we are only a little further
>along than we used to be, but are focusing on improving it. You'll find
>good thread support for some transports (some
> of the MTLs and at least the TCP BTL), not so good for others (e.g.,
>openib is flat-out not thread safe).
>
>
>
>On Dec 18, 2013, at 3:57 PM, Blosch, Edwin L 
>wrote:
>
>
>I was wondering if the FAQ entry below is considered current opinion or
>perhaps a little stale.  Is multi-threading still considered to be
>Œlightly tested¹?  Are there known open bugs?
> 
>Thank you,
> 
>Ed
> 
> 
>7. Is Open MPI thread safe?
> 
>Support for MPI_THREAD_MULTIPLE (i.e., multiple threads executing within
>the MPI library) and asynchronous message passing progress (i.e.,
>continuing message passing operations even while no user threads are in
>the MPI library) has been designed into Open MPI
> from its first planning meetings.
> 
>Support for MPI_THREAD_MULTIPLE is included in the first version of Open
>MPI, but it is only lightly tested and likely still has some bugs.
>Support for asynchronous progress is included in the TCP point-to-point
>device, but it, too, has only had light testing
> and likely still has bugs.
> 
>Completing the testing for full support of MPI_THREAD_MULTIPLE and
>asynchronous progress is planned in the near future.
> 
>
>___
>users mailing list
>us...@open-mpi.org
>http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>
>
>
>
>
> 
>___
>users mailing list
>users@open-mpi.orghttp://www.open-mpi.org/mailman/listinfo.cgi/users
>
>


--
  Brian W. Barrett
  Scalable System Software Group
  Sandia National Laboratories