Re: [OMPI devel] Open MPI v4.0.1: Process is hanging inside MPI_Init() when debugged with TotalView

2019-11-13 Thread Ralph Castain via devel
"Ralph Castain via devel" ---11/12/2019 09:24:23 AM---> On Nov 11, 2019, at 4:53 PM, Gilles Gouaillardet via devel <mailto:devel@lists.open-mpi.org> wrote: > From: "Ralph Castain via devel" <mailto:devel@lists.open-mpi.org> To: "OpenMPI Devel" <

Re: [OMPI devel] Open MPI v4.0.1: Process is hanging inside MPI_Init() when debugged with TotalView

2019-11-13 Thread Ralph Castain via devel
gnore ---11/12/2019 02:25:14 PM---Hi Austen, Thanks for the reply. What I am seeing is consistent with your thought, in that when I se From: John DelSignore To: Open MPI Developers Cc: Austen W Lauria, devel Date: 11/12/2019 02:25 PM Subject: [EXTERNAL] Re: [OMPI devel] Open MPI v4.0.1: Process is ha

Re: [OMPI devel] Open MPI v4.0.1: Process is hanging inside MPI_Init() when debugged with TotalView

2019-11-13 Thread John DelSignore via devel
> To: Open MPI Developers <mailto:devel@lists.open-mpi.org> Cc: Austen W Lauria <mailto:awlau...@us.ibm.com>, devel <mailto:devel-boun...@lists.open-mpi.org> Date: 11/12/2019 02:25 PM Subject: [EXTERNAL] Re: [OMPI devel] Open MPI v4.0.1: Process is hanging inside MPI_Init() w

Re: [OMPI devel] Open MPI v4.0.1: Process is hanging inside MPI_Init() when debugged with TotalView

2019-11-12 Thread Ralph Castain via devel
Austen W Lauria , devel Date: 11/12/2019 02:25 PM Subject: [EXTERNAL] Re: [OMPI devel] Open MPI v4.0.1: Process is hanging inside MPI_Init() when debugged with TotalView Hi Austen, Thanks for the reply. What I am seeing is consistent with your thought, in

Re: [OMPI devel] Open MPI v4.0.1: Process is hanging inside MPI_Init() when debugged with TotalView

2019-11-12 Thread John DelSignore via devel
<mailto:devel-boun...@lists.open-mpi.org> Date: 11/12/2019 02:25 PM Subject: [EXTERNAL] Re: [OMPI devel] Open MPI v4.0.1: Process is hanging inside MPI_Init() when debugged with TotalView Hi Austen, Thanks for the reply. What I am seeing is consistent with your

Re: [OMPI devel] Open MPI v4.0.1: Process is hanging inside MPI_Init() when debugged with TotalView

2019-11-12 Thread Ralph Castain via devel
which I see as a good thing as well. "Ralph Castain via devel" ---11/12/2019 09:24:23 AM---> On Nov 11, 2019, at 4:53 PM, Gilles Gouaillardet via devel <mailto:devel@lists.open-mpi.org> wrote: > From: "Ralph Castain via devel" <mailto:devel@lists.open-mpi.org&

Re: [OMPI devel] Open MPI v4.0.1: Process is hanging inside MPI_Init() when debugged with TotalView

2019-11-12 Thread George Bosilca via devel
this variable to: > > pthread_mutex_lock(); > flg = new_val; > pthread_cond_signal(); >pthread_mutex_unlock(); > >This provides the memory barrier for the thread polling on >the

Re: [OMPI devel] Open MPI v4.0.1: Process is hanging inside MPI_Init() when debugged with TotalView

2019-11-12 Thread Austen W Lauria via devel
ood thing as well. "Ralph Castain via devel" ---11/12/2019 09:24:23 AM---> On Nov 11, 2019, at 4:53 PM, Gilles Gouaillardet via devel wrote: > From: "Ralph Castain via devel" To: "Ope

Re: [OMPI devel] Open MPI v4.0.1: Process is hanging inside MPI_Init() when debugged with TotalView

2019-11-12 Thread George Bosilca via devel
gt; pthread_cond_signal(); > pthread_mutex_unlock(); > > This provides the memory barrier for the thread polling on the flag to see > the update - something the volatile keyword doesn't do on its own. I think > it's also much cleaner as it eliminates an arbitrary sleep from the

Re: [OMPI devel] Open MPI v4.0.1: Process is hanging inside MPI_Init() when debugged with TotalView

2019-11-12 Thread Austen W Lauria via devel
devel" To: "OpenMPI Devel" Cc: "Ralph Castain" Date: 11/12/2019 09:24 AM Subject: [EXTERNAL] Re: [OMPI devel] Open MPI v4.0.1: Process is hanging inside MPI_Init() when debugged with TotalView Sent by

Re: [OMPI devel] Open MPI v4.0.1: Process is hanging inside MPI_Init() when debugged with TotalView

2019-11-12 Thread Austen W Lauria via devel
"Ralph Castain" Date: 11/12/2019 01:28 PM Subject: [EXTERNAL] Re: [OMPI devel] Open MPI v4.0.1: Process is hanging inside MPI_Init() when debugged with TotalView Sent by:"devel" Just to be clear as well: you cannot use the pthread method yo

Re: [OMPI devel] Open MPI v4.0.1: Process is hanging inside MPI_Init() when debugged with TotalView

2019-11-12 Thread John DelSignore via devel
> From: "Ralph Castain via devel" <mailto:devel@lists.open-mpi.org> To: "OpenMPI Devel" <mailto:devel@lists.open-mpi.org> Cc: "Ralph Castain" <mailto:r...@open-mpi.org> Date: 11/12/2019 09:24 AM Subject: [EXTERNAL] Re:

Re: [OMPI devel] Open MPI v4.0.1: Process is hanging inside MPI_Init() when debugged with TotalView

2019-11-12 Thread Ralph Castain via devel
devel mailto:devel@lists.open-mpi.org> > wrote: > From: "Ralph Castain via devel" mailto:devel@lists.open-mpi.org> > To: "OpenMPI Devel" mailto:devel@lists.open-mpi.org> > Cc: "Ralph Castain" mailto:r...@open-mpi.org> > Date: 11/12/2019 09

Re: [OMPI devel] Open MPI v4.0.1: Process is hanging inside MPI_Init() when debugged with TotalView

2019-11-12 Thread George Bosilca via devel
uot; ---11/12/2019 09:24:23 AM---> On Nov 11, 2019, at 4:53 > PM, Gilles Gouaillardet via devel wrote: > > > From: "Ralph Castain via devel" > To: "OpenMPI Devel" > Cc: "Ralph Castain" > Date: 11/12/2019 09:24 AM > Subject: [EXTE

Re: [OMPI devel] Open MPI v4.0.1: Process is hanging inside MPI_Init() when debugged with TotalView

2019-11-12 Thread Austen W Lauria via devel
11/12/2019 09:24 AM Subject: [EXTERNAL] Re: [OMPI devel] Open MPI v4.0.1: Process is hanging inside MPI_Init() when debugged with TotalView Sent by:"devel" > On Nov 11, 2019, at 4:53 PM, Gilles Gouaillardet via devel wrote: > > John, > > OMP

Re: [OMPI devel] Open MPI v4.0.1: Process is hanging inside MPI_Init() when debugged with TotalView

2019-11-12 Thread Ralph Castain via devel
> On Nov 11, 2019, at 4:53 PM, Gilles Gouaillardet via devel > wrote: > > John, > > OMPI_LAZY_WAIT_FOR_COMPLETION(active) > > > is a simple loop that periodically checks the (volatile) "active" condition, > that is expected to be updated by an other thread. > So if you set your breakpoint

Re: [OMPI devel] Open MPI v4.0.1: Process is hanging inside MPI_Init() when debugged with TotalView

2019-11-11 Thread Gilles Gouaillardet via devel
John, OMPI_LAZY_WAIT_FOR_COMPLETION(active) is a simple loop that periodically checks the (volatile) "active" condition, that is expected to be updated by an other thread. So if you set your breakpoint too early, and **all** threads are stopped when this breakpoint is hit, you might

Re: [OMPI devel] Open MPI v4.0.1: Process is hanging inside MPI_Init() when debugged with TotalView

2019-11-11 Thread Ralph Castain via devel
Hi John Sorry to say, but there is no way to really answer your question as the OMPI community doesn't actively test MPIR support. I haven't seen any reports of hangs during MPI_Init from any release series, including 4.x. My guess is that it may have something to do with the debugger

[OMPI devel] Open MPI v4.0.1: Process is hanging inside MPI_Init() when debugged with TotalView

2019-11-08 Thread John DelSignore via devel
Hi, An LLNL TotalView user on a Mac reported that their MPI job was hanging inside MPI_Init() when started under the control of TotalView. They were using Open MPI 4.0.1, and TotalView was using the MPIR Interface (sorry, we don't support the PMIx debugging hooks yet). I was able to reproduce