Re: [OMPI users] EXTERNAL: Re: What's the status of OpenMPI and thread safety?
Hi Ed FWIW: Intel MPI has better thread support, though you'll lose some features. I don't know what NICs you have, but have you tried running with the MTL's instead of the openib BTL? Both the psm and mxm MTLs are supposed to be thread safe, and will outperform the openib BTL anyway. Not sure if Intel supports mxm or not - should handle psm as Intel now owns Truescale. HTH Ralph On Dec 19, 2013, at 7:23 AM, Blosch, Edwin L <edwin.l.blo...@lmco.com> wrote: > Thanks Ralph, > > We are attempting to use 1.6.4 with an application that requires > multi-threading, and it is hanging most of the time; it is using openib. > They steered us to try Intel MPI for now. If you lack drivers/testers for > improved thread safety on openib, let me know and I’ll encourage the > developers of the application to support you. > > Ed > > From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Ralph Castain > Sent: Wednesday, December 18, 2013 6:50 PM > To: Open MPI Users > Subject: EXTERNAL: Re: [OMPI users] What's the status of OpenMPI and thread > safety? > > This was, in fact, a primary point of discussion at last week's OMPI > developer's conference. Bottom line is that we are only a little further > along than we used to be, but are focusing on improving it. You'll find good > thread support for some transports (some of the MTLs and at least the TCP > BTL), not so good for others (e.g., openib is flat-out not thread safe). > > > On Dec 18, 2013, at 3:57 PM, Blosch, Edwin L <edwin.l.blo...@lmco.com> wrote: > > > I was wondering if the FAQ entry below is considered current opinion or > perhaps a little stale. Is multi-threading still considered to be ‘lightly > tested’? Are there known open bugs? > > Thank you, > > Ed > > > 7. Is Open MPI thread safe? > > Support for MPI_THREAD_MULTIPLE (i.e., multiple threads executing within the > MPI library) and asynchronous message passing progress (i.e., continuing > message passing operations even while no user threads are in the MPI library) > has been designed into Open MPI from its first planning meetings. > > Support for MPI_THREAD_MULTIPLE is included in the first version of Open MPI, > but it is only lightly tested and likely still has some bugs. Support for > asynchronous progress is included in the TCP point-to-point device, but it, > too, has only had light testing and likely still has bugs. > > Completing the testing for full support of MPI_THREAD_MULTIPLE and > asynchronous progress is planned in the near future. > > ___ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users > > ___ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users
Re: [OMPI users] EXTERNAL: Re: What's the status of OpenMPI and thread safety?
Thanks Ralph, We are attempting to use 1.6.4 with an application that requires multi-threading, and it is hanging most of the time; it is using openib. They steered us to try Intel MPI for now. If you lack drivers/testers for improved thread safety on openib, let me know and I'll encourage the developers of the application to support you. Ed From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Ralph Castain Sent: Wednesday, December 18, 2013 6:50 PM To: Open MPI Users Subject: EXTERNAL: Re: [OMPI users] What's the status of OpenMPI and thread safety? This was, in fact, a primary point of discussion at last week's OMPI developer's conference. Bottom line is that we are only a little further along than we used to be, but are focusing on improving it. You'll find good thread support for some transports (some of the MTLs and at least the TCP BTL), not so good for others (e.g., openib is flat-out not thread safe). On Dec 18, 2013, at 3:57 PM, Blosch, Edwin L <edwin.l.blo...@lmco.com<mailto:edwin.l.blo...@lmco.com>> wrote: I was wondering if the FAQ entry below is considered current opinion or perhaps a little stale. Is multi-threading still considered to be 'lightly tested'? Are there known open bugs? Thank you, Ed 7. Is Open MPI thread safe? Support for MPI_THREAD_MULTIPLE (i.e., multiple threads executing within the MPI library) and asynchronous message passing progress (i.e., continuing message passing operations even while no user threads are in the MPI library) has been designed into Open MPI from its first planning meetings. Support for MPI_THREAD_MULTIPLE is included in the first version of Open MPI, but it is only lightly tested and likely still has some bugs. Support for asynchronous progress is included in the TCP point-to-point device, but it, too, has only had light testing and likely still has bugs. Completing the testing for full support of MPI_THREAD_MULTIPLE and asynchronous progress is planned in the near future. ___ users mailing list us...@open-mpi.org<mailto:us...@open-mpi.org> http://www.open-mpi.org/mailman/listinfo.cgi/users
Re: [OMPI users] [EXTERNAL] Re: What's the status of OpenMPI and thread safety?
Pablo - As Ralph mentioned, it will be different, possibly not for the better, in 1.7. This is an area of active work, so any help would be appreciated. However, the one issue you brought up is going to be problematic, even with threads. Our design essentially makes it such that blocking MPI calls never block internally (for any thread level). It's one of the trade-offs in our multi-device design. Good: multi-device just works without any complicated state sharing between devices. Bad: it's hard for us to block. We've talked about making a blocking (or slow polling, semi-blocking) option for when we can detect we only have one device, but it hasn't been a high priority. Like Ralph said, if you're interested in working on the threading or blocking issues, please join the devel list and let us know. We're always willing to take new patches. Thanks, Brian On 12/19/13 5:34 AM, "Pablo Barrio"wrote: >Hi all, this is the first time I post to the list (although I have read >it for a while now). I hope this helps. > >I'm heavily using MPI_THREAD_MULTIPLE on multicores (sm BTL) and my >programs work fine from a CORRECTNESS point of view. I use OpenMPI 1.6 >(SVN rev. 26429) and pthreads on Linux. > >This said, the performance is still very poor. Some of my programs become >a thousand times slower. After some profiling/tracing, I found out that >the Linux scheduler gave CPU time to threads stuck in blocking calls >(Ssend, Recv, Wait, etcetera). It seems to > me that the MPI implementation can be improved to avoid spending CPU >time in threads waiting for messages. > >In short, my experience is that the implementation is correct but not >very efficient so far. > >I have a few questions: > >1. My OpenMPI version is more than a year old. Have these performance >issues been fixed in the latest versions? > >2. If not, perhaps I could contribute to OpenMPI multithreading >support. Who takes care of this? How can I help? > >Thanks ahead. >-- >Pablo Barrio >Dpt. Electrical Engineering - Technical University of Madrid >Office C-203 >Avda. Complutense s/n, 28040 Madrid >Tel. (+34) 915495700 ext. 4234 >@: pbar...@die.upm.es >On 19/12/13 01:49, Ralph Castain wrote: > > >This was, in fact, a primary point of discussion at last week's OMPI >developer's conference. Bottom line is that we are only a little further >along than we used to be, but are focusing on improving it. You'll find >good thread support for some transports (some > of the MTLs and at least the TCP BTL), not so good for others (e.g., >openib is flat-out not thread safe). > > > >On Dec 18, 2013, at 3:57 PM, Blosch, Edwin L >wrote: > > >I was wondering if the FAQ entry below is considered current opinion or >perhaps a little stale. Is multi-threading still considered to be >Œlightly tested¹? Are there known open bugs? > >Thank you, > >Ed > > >7. Is Open MPI thread safe? > >Support for MPI_THREAD_MULTIPLE (i.e., multiple threads executing within >the MPI library) and asynchronous message passing progress (i.e., >continuing message passing operations even while no user threads are in >the MPI library) has been designed into Open MPI > from its first planning meetings. > >Support for MPI_THREAD_MULTIPLE is included in the first version of Open >MPI, but it is only lightly tested and likely still has some bugs. >Support for asynchronous progress is included in the TCP point-to-point >device, but it, too, has only had light testing > and likely still has bugs. > >Completing the testing for full support of MPI_THREAD_MULTIPLE and >asynchronous progress is planned in the near future. > > >___ >users mailing list >us...@open-mpi.org >http://www.open-mpi.org/mailman/listinfo.cgi/users > > > > > > > >___ >users mailing list >users@open-mpi.orghttp://www.open-mpi.org/mailman/listinfo.cgi/users > > -- Brian W. Barrett Scalable System Software Group Sandia National Laboratories