"MPI can get through your firewall, right?"
As far as I can tell the firewall is not the problem - have tried it with firewalls disabled, automatic fw polices based on port requests from MPI, and with manual exception policies. > From: users-requ...@open-mpi.org > Subject: users Digest, Vol 1911, Issue 4 > To: us...@open-mpi.org > Date: Fri, 20 May 2011 14:58:40 -0400 > > Send users mailing list submissions to > us...@open-mpi.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://www.open-mpi.org/mailman/listinfo.cgi/users > or, via email, send a message with subject or body 'help' to > users-requ...@open-mpi.org > > You can reach the person managing the list at > users-ow...@open-mpi.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of users digest..." > > > Today's Topics: > > 1. Re: v1.5.3-x64 does not work on Windows 7 workgroup (Damien) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Fri, 20 May 2011 12:58:21 -0600 > From: Damien <dam...@khubla.com> > Subject: Re: [OMPI users] v1.5.3-x64 does not work on Windows 7 > workgroup > To: Open MPI Users <us...@open-mpi.org> > Message-ID: <4dd6b9cd.8060...@khubla.com> > Content-Type: text/plain; charset="iso-8859-1"; Format="flowed" > > MPI can get through your firewall, right? > > Damien > > On 20/05/2011 12:53 PM, Jason Mackay wrote: > > I have verified that disabling UAC does not fix the problem. xhlp.exe > > starts, threads spin up on both machines, CPU usage is at 80-90% but > > no progress is ever made. > > > > >From this state, Ctrl-break on the head node yields the following output: > > > > [REMOTEMACHINE:02032] [[20816,1],0]-[[20816,0],0] > > mca_oob_tcp_msg_recv: readv failed: Unknown error (108) > > [REMOTEMACHINE:05064] [[20816,1],1]-[[20816,0],0] > > mca_oob_tcp_msg_recv: readv failed: Unknown error (108) > > [REMOTEMACHINE:05420] [[20816,1],2]-[[20816,0],0] > > mca_oob_tcp_msg_recv: readv failed: Unknown error (108) > > [REMOTEMACHINE:03852] [[20816,1],3]-[[20816,0],0] > > mca_oob_tcp_msg_recv: readv failed: Unknown error (108) > > [REMOTEMACHINE:05436] [[20816,1],4]-[[20816,0],0] > > mca_oob_tcp_msg_recv: readv failed: Unknown error (108) > > [REMOTEMACHINE:04416] [[20816,1],5]-[[20816,0],0] > > mca_oob_tcp_msg_recv: readv failed: Unknown error (108) > > [REMOTEMACHINE:02032] [[20816,1],0] routed:binomial: Connection to > > lifeline [[20816,0],0] lost > > [REMOTEMACHINE:05064] [[20816,1],1] routed:binomial: Connection to > > lifeline [[20816,0],0] lost > > [REMOTEMACHINE:05420] [[20816,1],2] routed:binomial: Connection to > > lifeline [[20816,0],0] lost > > [REMOTEMACHINE:03852] [[20816,1],3] routed:binomial: Connection to > > lifeline [[20816,0],0] lost > > [REMOTEMACHINE:05436] [[20816,1],4] routed:binomial: Connection to > > lifeline [[20816,0],0] lost > > [REMOTEMACHINE:04416] [[20816,1],5] routed:binomial: Connection to > > lifeline [[20816,0],0] lost > > > > > > > > > From: users-requ...@open-mpi.org > > > Subject: users Digest, Vol 1911, Issue 1 > > > To: us...@open-mpi.org > > > Date: Fri, 20 May 2011 08:14:13 -0400 > > > > > > Send users mailing list submissions to > > > us...@open-mpi.org > > > > > > To subscribe or unsubscribe via the World Wide Web, visit > > > http://www.open-mpi.org/mailman/listinfo.cgi/users > > > or, via email, send a message with subject or body 'help' to > > > users-requ...@open-mpi.org > > > > > > You can reach the person managing the list at > > > users-ow...@open-mpi.org > > > > > > When replying, please edit your Subject line so it is more specific > > > than "Re: Contents of users digest..." > > > > > > > > > Today's Topics: > > > > > > 1. Re: Error: Entry Point Not Found (Zhangping Wei) > > > 2. Re: Problem with MPI_Request, MPI_Isend/recv and > > > MPI_Wait/Test (George Bosilca) > > > 3. Re: v1.5.3-x64 does not work on Windows 7 workgroup (Jeff Squyres) > > > 4. Re: Error: Entry Point Not Found (Jeff Squyres) > > > 5. Re: openmpi (1.2.8 or above) and Intel composer XE 2011 (aka > > > 12.0) (Jeff Squyres) > > > 6. Re: Openib with > 32 cores per node (Jeff Squyres) > > > 7. Re: MPI_COMM_DUP freeze with OpenMPI 1.4.1 (Jeff Squyres) > > > 8. Re: Trouble with MPI-IO (Jeff Squyres) > > > 9. Re: Trouble with MPI-IO (Tom Rosmond) > > > 10. Re: Problem with MPI_Request, MPI_Isend/recv and > > > MPI_Wait/Test (David B?ttner) > > > 11. Re: Trouble with MPI-IO (Jeff Squyres) > > > 12. Re: MPI_Alltoallv function crashes when np > 100 (Jeff Squyres) > > > 13. Re: MPI_ERR_TRUNCATE with MPI_Allreduce() error, but only > > > sometimes... (Jeff Squyres) > > > 14. Re: Trouble with MPI-IO (Jeff Squyres) > > > > > > > > > ---------------------------------------------------------------------- > > > > > > Message: 1 > > > Date: Thu, 19 May 2011 09:13:53 -0700 (PDT) > > > From: Zhangping Wei <zhangping_...@yahoo.com> > > > Subject: Re: [OMPI users] Error: Entry Point Not Found > > > To: us...@open-mpi.org > > > Message-ID: <101342.7961...@web111818.mail.gq1.yahoo.com> > > > Content-Type: text/plain; charset="gb2312" > > > > > > Dear Paul, > > > > > > I checked the way 'mpirun -np N <cmd>' you mentioned, but it was the > > same > > > problem. > > > > > > I guess it may related to the system I used, because I have used it > > correctly in > > > another XP 32 bit system. > > > > > > I look forward to more advice.Thanks. > > > > > > Zhangping > > > > > > > > > > > > > > > ________________________________ > > > ???????? "users-requ...@open-mpi.org" <users-requ...@open-mpi.org> > > > ???????? us...@open-mpi.org > > > ?????????? 2011/5/19 (????) 11:00:02 ???? > > > ?? ???? users Digest, Vol 1910, Issue 2 > > > > > > Send users mailing list submissions to > > > us...@open-mpi.org > > > > > > To subscribe or unsubscribe via the World Wide Web, visit > > > http://www.open-mpi.org/mailman/listinfo.cgi/users > > > or, via email, send a message with subject or body 'help' to > > > users-requ...@open-mpi.org > > > > > > You can reach the person managing the list at > > > users-ow...@open-mpi.org > > > > > > When replying, please edit your Subject line so it is more specific > > > than "Re: Contents of users digest..." > > > > > > > > > Today's Topics: > > > > > > 1. Re: Error: Entry Point Not Found (Paul van der Walt) > > > 2. Re: Openib with > 32 cores per node (Robert Horton) > > > 3. Re: Openib with > 32 cores per node (Samuel K. Gutierrez) > > > > > > > > > ---------------------------------------------------------------------- > > > > > > Message: 1 > > > Date: Thu, 19 May 2011 16:14:02 +0100 > > > From: Paul van der Walt <p...@denknerd.nl> > > > Subject: Re: [OMPI users] Error: Entry Point Not Found > > > To: Open MPI Users <us...@open-mpi.org> > > > Message-ID: <banlktinjz0cntchqjczyhfgsnr51jpu...@mail.gmail.com> > > > Content-Type: text/plain; charset=UTF-8 > > > > > > Hi, > > > > > > On 19 May 2011 15:54, Zhangping Wei <zhangping_...@yahoo.com> wrote: > > > > 4, I use command window to run it in this way: ?mpirun ?n 4 > > ?**.exe ?,then I > > > > > > Probably not the problem, but shouldn't that be 'mpirun -np N <cmd>' ? > > > > > > Paul > > > > > > -- > > > O< ascii ribbon campaign - stop html mail - www.asciiribbon.org > > > > > > > > > > > > ------------------------------ > > > > > > Message: 2 > > > Date: Thu, 19 May 2011 16:37:56 +0100 > > > From: Robert Horton <r.hor...@qmul.ac.uk> > > > Subject: Re: [OMPI users] Openib with > 32 cores per node > > > To: Open MPI Users <us...@open-mpi.org> > > > Message-ID: <1305819476.9663.148.camel@moelwyn> > > > Content-Type: text/plain; charset="UTF-8" > > > > > > On Thu, 2011-05-19 at 08:27 -0600, Samuel K. Gutierrez wrote: > > > > Hi, > > > > > > > > Try the following QP parameters that only use shared receive queues. > > > > > > > > -mca btl_openib_receive_queues S,12288,128,64,32:S,65536,128,64,32 > > > > > > > > > > Thanks for that. If I run the job over 2 x 48 cores it now works and the > > > performance seems reasonable (I need to do some more tuning) but when I > > > go up to 4 x 48 cores I'm getting the same problem: > > > > > > > > [compute-1-7.local][[14383,1],86][../../../../../ompi/mca/btl/openib/connect/btl_openib_connect_oob.c:464:qp_create_one] > > > error creating qp errno says Cannot allocate memory > > > [compute-1-7.local:18106] *** An error occurred in MPI_Isend > > > [compute-1-7.local:18106] *** on communicator MPI_COMM_WORLD > > > [compute-1-7.local:18106] *** MPI_ERR_OTHER: known error not in list > > > [compute-1-7.local:18106] *** MPI_ERRORS_ARE_FATAL (your MPI job > > will now abort) > > > > > > Any thoughts? > > > > > > Thanks, > > > Rob > > > -- > > > Robert Horton > > > System Administrator (Research Support) - School of Mathematical > > Sciences > > > Queen Mary, University of London > > > r.hor...@qmul.ac.uk - +44 (0) 20 7882 7345 > > > > > > > > > > > > ------------------------------ > > > > > > Message: 3 > > > Date: Thu, 19 May 2011 09:59:13 -0600 > > > From: "Samuel K. Gutierrez" <sam...@lanl.gov> > > > Subject: Re: [OMPI users] Openib with > 32 cores per node > > > To: Open MPI Users <us...@open-mpi.org> > > > Message-ID: <b3e83138-9af0-48c0-871c-dbbb2e712...@lanl.gov> > > > Content-Type: text/plain; charset=us-ascii > > > > > > Hi, > > > > > > On May 19, 2011, at 9:37 AM, Robert Horton wrote > > > > > > > On Thu, 2011-05-19 at 08:27 -0600, Samuel K. Gutierrez wrote: > > > >> Hi, > > > >> > > > >> Try the following QP parameters that only use shared receive queues. > > > >> > > > >> -mca btl_openib_receive_queues S,12288,128,64,32:S,65536,128,64,32 > > > >> > > > > > > > > Thanks for that. If I run the job over 2 x 48 cores it now works > > and the > > > > performance seems reasonable (I need to do some more tuning) but > > when I > > > > go up to 4 x 48 cores I'm getting the same problem: > > > > > > > > > >[compute-1-7.local][[14383,1],86][../../../../../ompi/mca/btl/openib/connect/btl_openib_connect_oob.c:464:qp_create_one] > > > >] error creating qp errno says Cannot allocate memory > > > > [compute-1-7.local:18106] *** An error occurred in MPI_Isend > > > > [compute-1-7.local:18106] *** on communicator MPI_COMM_WORLD > > > > [compute-1-7.local:18106] *** MPI_ERR_OTHER: known error not in list > > > > [compute-1-7.local:18106] *** MPI_ERRORS_ARE_FATAL (your MPI job > > will now > > > >abort) > > > > > > > > Any thoughts? > > > > > > How much memory does each node have? Does this happen at startup? > > > > > > Try adding: > > > > > > -mca btl_openib_cpc_include rdmacm > > > > > > I'm not sure if your version of OFED supports this feature, but > > maybe using XRC > > > may help. I **think** other tweaks are needed to get this going, but > > I'm not > > > familiar with the details. > > > > > > Hope that helps, > > > > > > Samuel K. Gutierrez > > > Los Alamos National Laboratory > > > > > > > > > > > > > > Thanks, > > > > Rob > > > > -- > > > > Robert Horton > > > > System Administrator (Research Support) - School of Mathematical > > Sciences > > > > Queen Mary, University of London > > > > r.hor...@qmul.ac.uk - +44 (0) 20 7882 7345 > > > > > > > > _______________________________________________ > > > > users mailing list > > > > us...@open-mpi.org > > > > http://www.open-mpi.org/mailman/listinfo.cgi/users > > > > > > > > > > > > > > > > > > > > > ------------------------------ > > > > > > _______________________________________________ > > > users mailing list > > > us...@open-mpi.org > > > http://www.open-mpi.org/mailman/listinfo.cgi/users > > > > > > End of users Digest, Vol 1910, Issue 2 > > > ************************************** > > > -------------- next part -------------- > > > HTML attachment scrubbed and removed > > > > > > ------------------------------ > > > > > > Message: 2 > > > Date: Thu, 19 May 2011 08:48:03 -0800 > > > From: George Bosilca <bosi...@eecs.utk.edu> > > > Subject: Re: [OMPI users] Problem with MPI_Request, MPI_Isend/recv and > > > MPI_Wait/Test > > > To: Open MPI Users <us...@open-mpi.org> > > > Message-ID: <fcac66f9-fdb5-48bb-a800-263d8a4f9...@eecs.utk.edu> > > > Content-Type: text/plain; charset=iso-8859-1 > > > > > > David, > > > > > > I do not see any mechanism for protecting the accesses to the > > requests to a single thread? What is the thread model you're using? > > > > > > >From an implementation perspective, your code is correct only if > > you initialize the MPI library with MPI_THREAD_MULTIPLE and if the > > library accepts. Otherwise, there is an assumption that the > > application is single threaded, or that the MPI behavior is > > implementation dependent. Please read the MPI standard regarding to > > MPI_Init_thread for more details. > > > > > > Regards, > > > george. > > > > > > On May 19, 2011, at 02:34 , David B?ttner wrote: > > > > > > > Hello, > > > > > > > > I am working on a hybrid MPI (OpenMPI 1.4.3) and Pthread code. I > > am using MPI_Isend and MPI_Irecv for communication and > > MPI_Test/MPI_Wait to check if it is done. I do this repeatedly in the > > outer loop of my code. The MPI_Test is used in the inner loop to check > > if some function can be called which depends on the received data. > > > > The program regularly crashed (only when not using printf...) and > > after debugging it I figured out the following problem: > > > > > > > > In MPI_Isend I have an invalid read of memory. I fixed the problem > > with not re-using a > > > > > > > > MPI_Request req_s, req_r; > > > > > > > > but by using > > > > > > > > MPI_Request* req_s; > > > > MPI_Request* req_r > > > > > > > > and re-allocating them before the MPI_Isend/recv. > > > > > > > > The documentation says, that in MPI_Wait and MPI_Test (if > > successful) the request-objects are deallocated and set to > > MPI_REQUEST_NULL. > > > > It also says, that in MPI_Isend and MPI_Irecv, it allocates the > > Objects and associates it with the request object. > > > > > > > > As I understand this, this either means I can use a pointer to > > MPI_Request which I don't have to initialize for this (it doesn't work > > but crashes), or that I can use a MPI_Request pointer which I have > > initialized with malloc(sizeof(MPI_REQUEST)) (or passing the address > > of a MPI_Request req), which is set and unset in the functions. But > > this version crashes, too. > > > > What works is using a pointer, which I allocate before the > > MPI_Isend/recv and which I free after MPI_Wait in every iteration. In > > other words: It only uses if I don't reuse any kind of MPI_Request. > > Only if I recreate one every time. > > > > > > > > Is this, what is should be like? I believe that a reuse of the > > memory would be a lot more efficient (less calls to malloc...). Am I > > missing something here? Or am I doing something wrong? > > > > > > > > > > > > Let me provide some more detailed information about my problem: > > > > > > > > I am running the program on a 30 node infiniband cluster. Each > > node has 4 single core Opteron CPUs. I am running 1 MPI Rank per node > > and 4 threads per rank (-> one thread per core). > > > > I am compiling with mpicc of OpenMPI using gcc below. > > > > Some pseudo-code of the program can be found at the end of this > > e-mail. > > > > > > > > I was able to reproduce the problem using different amount of > > nodes and even using one node only. The problem does not arise when I > > put printf-debugging information into the code. This pointed me into > > the direction that I have some memory problem, where some write > > accesses some memory it is not supposed to. > > > > I ran the tests using valgrind with --leak-check=full and > > --show-reachable=yes, which pointed me either to MPI_Isend or MPI_Wait > > depending on whether I had the threads spin in a loop for MPI_Test to > > return success or used MPI_Wait respectively. > > > > > > > > I would appreciate your help with this. Am I missing something > > important here? Is there a way to re-use the request in the different > > iterations other than I thought it should work? > > > > Or is there a way to re-initialize the allocated memory before the > > MPI_Isend/recv so that I at least don't have to call free and malloc > > each time? > > > > > > > > Thank you very much for your help! > > > > Kind regards, > > > > David B?ttner > > > > > > > > _____________________ > > > > Pseudo-Code of program: > > > > > > > > MPI_Request* req_s; > > > > MPI_Request* req_w; > > > > OUTER-LOOP > > > > if(0 == threadid) > > > > { > > > > req_s = malloc(sizeof(MPI_Request)); > > > > req_r = malloc(sizeof(MPI_Request)); > > > > MPI_Isend(..., req_s) > > > > MPI_Irecv(..., req_r) > > > > } > > > > pthread_barrier > > > > INNER-LOOP (while NOT_DONE or RET) > > > > if(TRYLOCK && NOT_DONE) > > > > { > > > > if(MPI_TEST(req_r)) > > > > { > > > > Call_Function_A; > > > > NOT_DONE = 0; > > > > } > > > > > > > > } > > > > RET = Call_Function_B; > > > > } > > > > pthread_barrier_wait > > > > if(0 == threadid) > > > > { > > > > MPI_WAIT(req_s) > > > > MPI_WAIT(req_r) > > > > free(req_s); > > > > free(req_r); > > > > } > > > > _____________ > > > > > > > > > > > > -- > > > > David B?ttner, Informatik, Technische Universit?t M?nchen > > > > TUM I-10 - FMI 01.06.059 - Tel. 089 / 289-17676 > > > > > > > > _______________________________________________ > > > > users mailing list > > > > us...@open-mpi.org > > > > http://www.open-mpi.org/mailman/listinfo.cgi/users > > > > > > "To preserve the freedom of the human mind then and freedom of the > > press, every spirit should be ready to devote itself to martyrdom; for > > as long as we may think as we will, and speak as we think, the > > condition of man will proceed in improvement." > > > -- Thomas Jefferson, 1799 > > > > > > > > > > > > > > > ------------------------------ > > > > > > Message: 3 > > > Date: Thu, 19 May 2011 21:22:48 -0400 > > > From: Jeff Squyres <jsquy...@cisco.com> > > > Subject: Re: [OMPI users] v1.5.3-x64 does not work on Windows 7 > > > workgroup > > > To: Open MPI Users <us...@open-mpi.org> > > > Message-ID: <278274f0-bf00-4498-950f-9779e0083...@cisco.com> > > > Content-Type: text/plain; charset=us-ascii > > > > > > Unfortunately, our Windows guy (Shiqing) is off getting married and > > will be out for a little while. :-( > > > > > > All that I can cite is the README.WINDOWS.txt file in the top-level > > directory. I'm afraid that I don't know much else about Windows. :-( > > > > > > > > > On May 18, 2011, at 8:17 PM, Jason Mackay wrote: > > > > > > > Hi all, > > > > > > > > My thanks to all those involved for putting together this Windows > > binary release of OpenMPI! I am hoping to use it in a small Windows > > based OpenMPI cluster at home. > > > > > > > > Unfortunately my experience so far has not exactly been trouble > > free. It seems that, due to the fact that this release is using WMI, > > there are a number of settings that must be configured on the machines > > in order to get this to work. These settings are not documented in the > > distribution at all. I have been experimenting with it for over a week > > on and off and as soon as I solve one problem, another one arises. > > > > > > > > Currently, after much searching, reading, and tinkering with DCOM > > settings etc..., I can remotely start processes on all my machines > > using mpirun but those processes cannot access network shares (e.g. > > for binary distribution) and HPL (which works on any one node) does > > not seem to work if I run it across multiple nodes, also indicating a > > network issue (CPU sits at 100% in all processes with no network > > traffic and never terminates). To eliminate premission issues that may > > be caused by UAC I tried the same setup on two domain machines using > > an administrative account to launch and the behavior was the same. I > > have read that WMI processes cannot access network resources and I am > > at a loss for a solution to this newest of problems. If anyone knows > > how to make this work I would appreciate the help. I assume that > > someone has gotten this working and has the answers. > > > > > > > > I have searched the mailing list archives and I found other users > > with similar problems but no clear guidance on the threads. Some > > threads make references to Microsoft KB articles but do not explicitly > > tell the user what needs to be done, leaving each new user to > > rediscover the tricks on their own. One thread made it appear that > > testing had only been done on Windows XP. Needless to say, security > > has changed dramatically in Windows since XP! > > > > > > > > I would like to see OpenMPI for Windows be usable by a newcomer > > without all of this pain. > > > > > > > > What would be fantastic would be: > > > > 1) a step-by-step procedure for how to get OpenMPI 1.5 working on > > Windows > > > > a) preferably in a bare Windows 7 workgroup environment with > > nothing else (i.e. no Microsoft Cluster Compute Pack, no domain etc...) > > > > 2) inclusion of these steps in the binary distribution > > > > 3) bonus points for a script which accomplishes these things > > automatically > > > > > > > > If someone can help with (1), I would happily volunteer my time to > > work on (3). > > > > > > > > Regards, > > > > Jason > > > > _______________________________________________ > > > > users mailing list > > > > us...@open-mpi.org > > > > http://www.open-mpi.org/mailman/listinfo.cgi/users > > > > > > > > > -- > > > Jeff Squyres > > > jsquy...@cisco.com > > > For corporate legal information go to: > > > http://www.cisco.com/web/about/doing_business/legal/cri/ > > > > > > > > > > > > > > > ------------------------------ > > > > > > Message: 4 > > > Date: Thu, 19 May 2011 21:26:43 -0400 > > > From: Jeff Squyres <jsquy...@cisco.com> > > > Subject: Re: [OMPI users] Error: Entry Point Not Found > > > To: Open MPI Users <us...@open-mpi.org> > > > Message-ID: <f830ec35-fc9b-4801-b2a3-50f54d215...@cisco.com> > > > Content-Type: text/plain; charset=windows-1252 > > > > > > On May 19, 2011, at 10:54 AM, Zhangping Wei wrote: > > > > > > > 4, I use command window to run it in this way: ?mpirun ?n 4 **.exe > > ?,then I met the error: ?entry point not found: the procedure entry > > point inet_pton could not be located in the dynamic link library > > WS2_32.dll? > > > > > > Unfortunately our Windows developer/maintainer is out for a little > > while (he's getting married); he pretty much did the Windows stuff by > > himself, so none of the rest of us know much about it. :( > > > > > > inet_pton is a standard function call relating to IP addresses that > > we use in the internals of OMPI; I'm not sure why it wouldn't be found > > on Windows XP (Shiqing did cite that the OMPI Windows port should work > > on Windows XP). > > > > > > This post seems to imply that inet_ntop is only available on Vista > > and above: > > > > > > > > http://social.msdn.microsoft.com/Forums/en-US/vcgeneral/thread/e40465f2-41b7-4243-ad33-15ae9366f4e6/ > > > > > > So perhaps Shiqing needs to put in some kind of portability > > workaround for OMPI, and the current binaries won't actually work for > > XP...? > > > > > > I can't say that for sure because I really know very little about > > Windows; we'll unfortunately have to wait until he returns to get a > > definitive answer. :-( > > > > > > -- > > > Jeff Squyres > > > jsquy...@cisco.com > > > For corporate legal information go to: > > > http://www.cisco.com/web/about/doing_business/legal/cri/ > > > > > > > > > > > > > > > ------------------------------ > > > > > > Message: 5 > > > Date: Thu, 19 May 2011 21:37:49 -0400 > > > From: Jeff Squyres <jsquy...@cisco.com> > > > Subject: Re: [OMPI users] openmpi (1.2.8 or above) and Intel composer > > > XE 2011 (aka 12.0) > > > To: Open MPI Users <us...@open-mpi.org> > > > Cc: Giovanni Bracco <giovanni.bra...@enea.it>, Agostino Funel > > > <agostino.fu...@enea.it>, Fiorenzo Ambrosino > > > <fiorenzo.ambros...@enea.it>, Guido Guarnieri > > > <guido.guarni...@enea.it>, Roberto Ciavarella > > > <roberto.ciavare...@enea.it>, Salvatore Podda > > > <salvatore.po...@enea.it>, Giovanni Ponti <giovanni.po...@enea.it> > > > Message-ID: <45362608-b8b0-4ade-9959-b35c5690a...@cisco.com> > > > Content-Type: text/plain; charset=us-ascii > > > > > > Sorry for the late reply. > > > > > > Other users have seen something similar but we have never been able > > to reproduce it. Is this only when using IB? If you use "mpirun --mca > > btl_openib_cpc_if_include rdmacm", does the problem go away? > > > > > > > > > On May 11, 2011, at 6:00 PM, Marcus R. Epperson wrote: > > > > > > > I've seen the same thing when I build openmpi 1.4.3 with Intel 12, > > but only when I have -O2 or -O3 in CFLAGS. If I drop it down to -O1 > > then the collectives hangs go away. I don't know what, if anything, > > the higher optimization buys you when compiling openmpi, so I'm not > > sure if that's an acceptable workaround or not. > > > > > > > > My system is similar to yours - Intel X5570 with QDR Mellanox IB > > running RHEL 5, Slurm, and these openmpi btls: openib,sm,self. I'm > > using IMB 3.2.2 with a single iteration of Barrier to reproduce the > > hang, and it happens 100% of the time for me when I invoke it like this: > > > > > > > > # salloc -N 9 orterun -n 65 ./IMB-MPI1 -npmin 64 -iter 1 barrier > > > > > > > > The hang happens on the first Barrier (64 ranks) and each of the > > participating ranks have this backtrace: > > > > > > > > __poll (...) > > > > poll_dispatch () from [instdir]/lib/libopen-pal.so.0 > > > > opal_event_loop () from [instdir]/lib/libopen-pal.so.0 > > > > opal_progress () from [instdir]/lib/libopen-pal.so.0 > > > > ompi_request_default_wait_all () from [instdir]/lib/libmpi.so.0 > > > > ompi_coll_tuned_sendrecv_actual () from [instdir]/lib/libmpi.so.0 > > > > ompi_coll_tuned_barrier_intra_recursivedoubling () from > > [instdir]/lib/libmpi.so.0 > > > > ompi_coll_tuned_barrier_intra_dec_fixed () from > > [instdir]/lib/libmpi.so.0 > > > > PMPI_Barrier () from [instdir]/lib/libmpi.so.0 > > > > IMB_barrier () > > > > IMB_init_buffers_iter () > > > > main () > > > > > > > > The one non-participating rank has this backtrace: > > > > > > > > __poll (...) > > > > poll_dispatch () from [instdir]/lib/libopen-pal.so.0 > > > > opal_event_loop () from [instdir]/lib/libopen-pal.so.0 > > > > opal_progress () from [instdir]/lib/libopen-pal.so.0 > > > > ompi_request_default_wait_all () from [instdir]/lib/libmpi.so.0 > > > > ompi_coll_tuned_sendrecv_actual () from [instdir]/lib/libmpi.so.0 > > > > ompi_coll_tuned_barrier_intra_bruck () from [instdir]/lib/libmpi.so.0 > > > > ompi_coll_tuned_barrier_intra_dec_fixed () from > > [instdir]/lib/libmpi.so.0 > > > > PMPI_Barrier () from [instdir]/lib/libmpi.so.0 > > > > main () > > > > > > > > If I use more nodes I can get it to hang with 1ppn, so that seems > > to rule out the sm btl (or interactions with it) as a culprit at least. > > > > > > > > I can't reproduce this with openmpi 1.5.3, interestingly. > > > > > > > > -Marcus > > > > > > > > > > > > On 05/10/2011 03:37 AM, Salvatore Podda wrote: > > > >> Dear all, > > > >> > > > >> we succeed in building several version of openmpi from 1.2.8 to > > 1.4.3 > > > >> with Intel composer XE 2011 (aka 12.0). > > > >> However we found a threshold in the number of cores (depending > > from the > > > >> application: IMB, xhpl or user applications > > > >> and form the number of required cores) above which the > > application hangs > > > >> (sort of deadlocks). > > > >> The building of openmpi with 'gcc' and 'pgi' does not show the > > same limits. > > > >> There are any known incompatibilities of openmpi with this > > version of > > > >> intel compiilers? > > > >> > > > >> The characteristics of our computational infrastructure are: > > > >> > > > >> Intel processors E7330, E5345, E5530 e E5620 > > > >> > > > >> CentOS 5.3, CentOS 5.5. > > > >> > > > >> Intel composer XE 2011 > > > >> gcc 4.1.2 > > > >> pgi 10.2-1 > > > >> > > > >> Regards > > > >> > > > >> Salvatore Podda > > > >> > > > >> ENEA UTICT-HPC > > > >> Department for Computer Science Development and ICT > > > >> Facilities Laboratory for Science and High Performace Computing > > > >> C.R. Frascati > > > >> Via E. Fermi, 45 > > > >> PoBox 65 > > > >> 00044 Frascati (Rome) > > > >> Italy > > > >> > > > >> Tel: +39 06 9400 5342 > > > >> Fax: +39 06 9400 5551 > > > >> Fax: +39 06 9400 5735 > > > >> E-mail: salvatore.po...@enea.it > > > >> Home Page: www.cresco.enea.it > > > >> _______________________________________________ > > > >> users mailing list > > > >> us...@open-mpi.org > > > >> http://www.open-mpi.org/mailman/listinfo.cgi/users > > > >> > > > > > > > > _______________________________________________ > > > > users mailing list > > > > us...@open-mpi.org > > > > http://www.open-mpi.org/mailman/listinfo.cgi/users > > > > > > > > > -- > > > Jeff Squyres > > > jsquy...@cisco.com > > > For corporate legal information go to: > > > http://www.cisco.com/web/about/doing_business/legal/cri/ > > > > > > > > > > > > > > > ------------------------------ > > > > > > Message: 6 > > > Date: Thu, 19 May 2011 22:01:00 -0400 > > > From: Jeff Squyres <jsquy...@cisco.com> > > > Subject: Re: [OMPI users] Openib with > 32 cores per node > > > To: Open MPI Users <us...@open-mpi.org> > > > Message-ID: <c18c4827-d305-484a-9dae-290902d40...@cisco.com> > > > Content-Type: text/plain; charset=us-ascii > > > > > > What Sam is alluding to is that the OpenFabrics driver code in OMPI > > is sucking up oodles of memory for each IB connection that you're > > using. The receive_queues param that he sent tells OMPI to use all > > shared receive queues (instead of defaulting to one per-peer receive > > queue and the rest shared receive queues -- the per-peer RQ sucks up > > all the memory when you multiple it by N peers). > > > > > > > > > On May 19, 2011, at 11:59 AM, Samuel K. Gutierrez wrote: > > > > > > > Hi, > > > > > > > > On May 19, 2011, at 9:37 AM, Robert Horton wrote > > > > > > > >> On Thu, 2011-05-19 at 08:27 -0600, Samuel K. Gutierrez wrote: > > > >>> Hi, > > > >>> > > > >>> Try the following QP parameters that only use shared receive queues. > > > >>> > > > >>> -mca btl_openib_receive_queues S,12288,128,64,32:S,65536,128,64,32 > > > >>> > > > >> > > > >> Thanks for that. If I run the job over 2 x 48 cores it now works > > and the > > > >> performance seems reasonable (I need to do some more tuning) but > > when I > > > >> go up to 4 x 48 cores I'm getting the same problem: > > > >> > > > >> > > [compute-1-7.local][[14383,1],86][../../../../../ompi/mca/btl/openib/connect/btl_openib_connect_oob.c:464:qp_create_one] > > > > error creating qp errno says Cannot allocate memory > > > >> [compute-1-7.local:18106] *** An error occurred in MPI_Isend > > > >> [compute-1-7.local:18106] *** on communicator MPI_COMM_WORLD > > > >> [compute-1-7.local:18106] *** MPI_ERR_OTHER: known error not in list > > > >> [compute-1-7.local:18106] *** MPI_ERRORS_ARE_FATAL (your MPI job > > will now abort) > > > >> > > > >> Any thoughts? > > > > > > > > How much memory does each node have? Does this happen at startup? > > > > > > > > Try adding: > > > > > > > > -mca btl_openib_cpc_include rdmacm > > > > > > > > I'm not sure if your version of OFED supports this feature, but > > maybe using XRC may help. I **think** other tweaks are needed to get > > this going, but I'm not familiar with the details. > > > > > > > > Hope that helps, > > > > > > > > Samuel K. Gutierrez > > > > Los Alamos National Laboratory > > > > > > > > > > > >> > > > >> Thanks, > > > >> Rob > > > >> -- > > > >> Robert Horton > > > >> System Administrator (Research Support) - School of Mathematical > > Sciences > > > >> Queen Mary, University of London > > > >> r.hor...@qmul.ac.uk - +44 (0) 20 7882 7345 > > > >> > > > >> _______________________________________________ > > > >> users mailing list > > > >> us...@open-mpi.org > > > >> http://www.open-mpi.org/mailman/listinfo.cgi/users > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > users mailing list > > > > us...@open-mpi.org > > > > http://www.open-mpi.org/mailman/listinfo.cgi/users > > > > > > > > > -- > > > Jeff Squyres > > > jsquy...@cisco.com > > > For corporate legal information go to: > > > http://www.cisco.com/web/about/doing_business/legal/cri/ > > > > > > > > > > > > > > > ------------------------------ > > > > > > Message: 7 > > > Date: Thu, 19 May 2011 22:04:46 -0400 > > > From: Jeff Squyres <jsquy...@cisco.com> > > > Subject: Re: [OMPI users] MPI_COMM_DUP freeze with OpenMPI 1.4.1 > > > To: Open MPI Users <us...@open-mpi.org> > > > Message-ID: <0dcf20b8-ca5c-4746-8187-a2dff39b1...@cisco.com> > > > Content-Type: text/plain; charset=us-ascii > > > > > > On May 13, 2011, at 8:31 AM, francoise.r...@obs.ujf-grenoble.fr wrote: > > > > > > > Here is the MUMPS portion of code (in zmumps_part1.F file) where > > the slaves call MPI_COMM_DUP , id%PAR and MASTER are initialized to 0 > > before : > > > > > > > > CALL MPI_COMM_SIZE(id%COMM, id%NPROCS, IERR ) > > > > > > I re-indented so that I could read it better: > > > > > > CALL MPI_COMM_SIZE(id%COMM, id%NPROCS, IERR ) > > > IF ( id%PAR .eq. 0 ) THEN > > > IF ( id%MYID .eq. MASTER ) THEN > > > color = MPI_UNDEFINED > > > ELSE > > > color = 0 > > > END IF > > > CALL MPI_COMM_SPLIT( id%COMM, color, 0, > > > & id%COMM_NODES, IERR ) > > > id%NSLAVES = id%NPROCS - 1 > > > ELSE > > > CALL MPI_COMM_DUP( id%COMM, id%COMM_NODES, IERR ) > > > id%NSLAVES = id%NPROCS > > > END IF > > > > > > IF (id%PAR .ne. 0 .or. id%MYID .NE. MASTER) THEN > > > CALL MPI_COMM_DUP( id%COMM_NODES, id%COMM_LOAD, IERR > > > ENDIF > > > > > > That doesn't look right -- both MPI_COMM_SPLIT and MPI_COMM_DUP are > > collective, meaning that all processes in the communicator must call > > them. In the first case, only some processes are calling > > MPI_COMM_SPLIT. Is there some other logic that forces the rest of the > > processes to call MPI_COMM_SPLIT, too? > > > > > > -- > > > Jeff Squyres > > > jsquy...@cisco.com > > > For corporate legal information go to: > > > http://www.cisco.com/web/about/doing_business/legal/cri/ > > > > > > > > > > > > > > > ------------------------------ > > > > > > Message: 8 > > > Date: Thu, 19 May 2011 22:30:03 -0400 > > > From: Jeff Squyres <jsquy...@cisco.com> > > > Subject: Re: [OMPI users] Trouble with MPI-IO > > > To: Open MPI Users <us...@open-mpi.org> > > > Message-ID: <eefb638f-72f1-4208-8ea2-4f25f610c...@cisco.com> > > > Content-Type: text/plain; charset=us-ascii > > > > > > Props for that testio script. I think you win the award for "most > > easy to reproduce test case." :-) > > > > > > I notice that some of the lines went over 72 columns, so I renamed > > the file x.f90 and changed all the comments from "c" to "!" and joined > > the two &-split lines. The error about implicit type for lenr went > > away, but then when I enabled better type checking by using "use mpi" > > instead of "include 'mpif.h'", I got the following: > > > > > > x.f90:99.77: > > > > > > call mpi_type_indexed(lenij,ijlena,ijdisp,mpi_real,ij_vector_type,ierr) > > > 1 > > > Error: There is no specific subroutine for the generic > > 'mpi_type_indexed' at (1) > > > > > > I looked at our mpi F90 module and see the following: > > > > > > interface MPI_Type_indexed > > > subroutine MPI_Type_indexed(count, array_of_blocklengths, > > array_of_displacements, oldtype, newtype, ierr) > > > integer, intent(in) :: count > > > integer, dimension(*), intent(in) :: array_of_blocklengths > > > integer, dimension(*), intent(in) :: array_of_displacements > > > integer, intent(in) :: oldtype > > > integer, intent(out) :: newtype > > > integer, intent(out) :: ierr > > > end subroutine MPI_Type_indexed > > > end interface > > > > > > I don't quite grok the syntax of the "allocatable" type ijdisp, so > > that might be the problem here...? > > > > > > Regardless, I'm not entirely sure if the problem is the >72 > > character lines, but then when that is gone, I'm not sure how the > > allocatable stuff fits in... (I'm not enough of a Fortran programmer > > to know) > > > > > > > > > > > > > > > On May 10, 2011, at 7:14 PM, Tom Rosmond wrote: > > > > > > > I would appreciate someone with experience with MPI-IO look at the > > > > simple fortran program gzipped and attached to this note. It is > > > > imbedded in a script so that all that is necessary to run it is do: > > > > 'testio' from the command line. The program generates a small 2-D > > input > > > > array, sets up an MPI-IO environment, and write a 2-D output array > > > > twice, with the only difference being the displacement arrays used to > > > > construct the indexed datatype. For the first write, simple > > > > monotonically increasing displacements are used, for the second the > > > > displacements are 'shuffled' in one dimension. They are printed during > > > > the run. > > > > > > > > For the first case the file is written properly, but for the > > second the > > > > program hangs on MPI_FILE_WRITE_AT_ALL and must be aborted manually. > > > > Although the program is compiled as an mpi program, I am running on a > > > > single processor, which makes the problem more puzzling. > > > > > > > > The program should be relatively self-explanatory, but if more > > > > information is needed, please ask. I am on an 8 core Xeon based Dell > > > > workstation running Scientific Linux 5.5, Intel fortran 12.0.3, and > > > > OpenMPI 1.5.3. I have also attached output from 'ompi_info'. > > > > > > > > T. Rosmond > > > > > > > > > > > > > > <testio.gz><info_ompi.gz>_______________________________________________ > > > > users mailing list > > > > us...@open-mpi.org > > > > http://www.open-mpi.org/mailman/listinfo.cgi/users > > > > > > > > > -- > > > Jeff Squyres > > > jsquy...@cisco.com > > > For corporate legal information go to: > > > http://www.cisco.com/web/about/doing_business/legal/cri/ > > > > > > > > > > > > > > > ------------------------------ > > > > > > Message: 9 > > > Date: Thu, 19 May 2011 20:24:25 -0700 > > > From: Tom Rosmond <rosm...@reachone.com> > > > Subject: Re: [OMPI users] Trouble with MPI-IO > > > To: Open MPI Users <us...@open-mpi.org> > > > Message-ID: <1305861865.4284.104.ca...@cedar.reachone.com> > > > Content-Type: text/plain > > > > > > Thanks for looking at my problem. Sounds like you did reproduce my > > > problem. I have added some comments below > > > > > > On Thu, 2011-05-19 at 22:30 -0400, Jeff Squyres wrote: > > > > Props for that testio script. I think you win the award for "most > > easy to reproduce test case." :-) > > > > > > > > I notice that some of the lines went over 72 columns, so I renamed > > the file x.f90 and changed all the comments from "c" to "!" and joined > > the two &-split lines. The error about implicit type for lenr went > > away, but then when I enabled better type checking by using "use mpi" > > instead of "include 'mpif.h'", I got the following: > > > > > > What fortran compiler did you use? > > > > > > In the original script my Intel compile used the -132 option, > > > allowing up to that many columns per line. I still think in > > > F77 fortran much of the time, and use 'c' for comments out > > > of habit. The change to '!' doesn't make any difference. > > > > > > > > > > x.f90:99.77: > > > > > > > > call > > mpi_type_indexed(lenij,ijlena,ijdisp,mpi_real,ij_vector_type,ierr) > > > > 1 > > > > Error: There is no specific subroutine for the generic > > 'mpi_type_indexed' at (1) > > > > > > Hmmm, very strange, since I am looking right at the MPI standard > > > documents with that routine documented. I too get this compile failure > > > when I switch to 'use mpi'. Could that be a problem with the Open MPI > > > fortran libraries??? > > > > > > > > I looked at our mpi F90 module and see the following: > > > > > > > > interface MPI_Type_indexed > > > > subroutine MPI_Type_indexed(count, array_of_blocklengths, > > array_of_displacements, oldtype, newtype, ierr) > > > > integer, intent(in) :: count > > > > integer, dimension(*), intent(in) :: array_of_blocklengths > > > > integer, dimension(*), intent(in) :: array_of_displacements > > > > integer, intent(in) :: oldtype > > > > integer, intent(out) :: newtype > > > > integer, intent(out) :: ierr > > > > end subroutine MPI_Type_indexed > > > > end interface > > > > > > > > I don't quite grok the syntax of the "allocatable" type ijdisp, so > > that might be the problem here...? > > > > > > Just a standard F90 'allocatable' statement. I've written thousands > > > just like it. > > > > > > > > Regardless, I'm not entirely sure if the problem is the >72 > > character lines, but then when that is gone, I'm not sure how the > > allocatable stuff fits in... (I'm not enough of a Fortran programmer > > to know) > > > > > > > Anyone else out that who can comment???? > > > > > > > > > T. Rosmond > > > > > > > > > > > > > > > > > On May 10, 2011, at 7:14 PM, Tom Rosmond wrote: > > > > > > > > > I would appreciate someone with experience with MPI-IO look at the > > > > > simple fortran program gzipped and attached to this note. It is > > > > > imbedded in a script so that all that is necessary to run it is do: > > > > > 'testio' from the command line. The program generates a small > > 2-D input > > > > > array, sets up an MPI-IO environment, and write a 2-D output array > > > > > twice, with the only difference being the displacement arrays > > used to > > > > > construct the indexed datatype. For the first write, simple > > > > > monotonically increasing displacements are used, for the second the > > > > > displacements are 'shuffled' in one dimension. They are printed > > during > > > > > the run. > > > > > > > > > > For the first case the file is written properly, but for the > > second the > > > > > program hangs on MPI_FILE_WRITE_AT_ALL and must be aborted manually. > > > > > Although the program is compiled as an mpi program, I am running > > on a > > > > > single processor, which makes the problem more puzzling. > > > > > > > > > > The program should be relatively self-explanatory, but if more > > > > > information is needed, please ask. I am on an 8 core Xeon based Dell > > > > > workstation running Scientific Linux 5.5, Intel fortran 12.0.3, and > > > > > OpenMPI 1.5.3. I have also attached output from 'ompi_info'. > > > > > > > > > > T. Rosmond > > > > > > > > > > > > > > > > > <testio.gz><info_ompi.gz>_______________________________________________ > > > > > users mailing list > > > > > us...@open-mpi.org > > > > > http://www.open-mpi.org/mailman/listinfo.cgi/users > > > > > > > > > > > > > > > > > > > > ------------------------------ > > > > > > Message: 10 > > > Date: Fri, 20 May 2011 09:25:14 +0200 > > > From: David B?ttner <david.buett...@in.tum.de> > > > Subject: Re: [OMPI users] Problem with MPI_Request, MPI_Isend/recv and > > > MPI_Wait/Test > > > To: Open MPI Users <us...@open-mpi.org> > > > Message-ID: <4dd6175a.1080...@in.tum.de> > > > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > > > > > Hello, > > > > > > thanks for the quick answer. I am sorry that I forgot to mention > > this: I > > > did compile OpenMPI with MPI_THREAD_MULTIPLE support and test if > > > required == provided after the MPI_Thread_init call. > > > > > > > I do not see any mechanism for protecting the accesses to the > > requests to a single thread? What is the thread model you're using? > > > > > > > Again I am sorry that this was not clear: In the pseudo code below I > > > wanted to indicate the access-protection I do by thread-id dependent > > > calls if(0 == thread-id) and by using the trylock(...) (using > > > pthread-mutexes). In the code all accesses concerning one MPI_Request > > > (which are pthread-global-pointers in my case) are protected and called > > > in sequential order, i.e. MPI_Isend/recv is returns before any > > thread is > > > allowed to call the corresponding MPI_Test and no-one can call MPI_Test > > > any more when a thread is allowed to call MPI_Wait. > > > I did this in the same manner before with other MPI implementations, > > but > > > also on the same machine with the same (untouched) OpenMPI > > > implementation, also using pthreads and MPI in combination, but I used > > > > > > MPI_Request req; > > > > > > instead of > > > > > > MPI_Request* req; > > > (and later) > > > req = (MPI_Request*)malloc(sizeof(MPI_Request)); > > > > > > > > > In my recent (problem) code, I also tried not using pointers, but got > > > the same problem. Also, as I described in the first mail, I tried > > > everything concerning the memory allocation of the MPI_Request objects. > > > I tried not calling malloc. This I guessed wouldn't work, but the > > > OpenMPI documentation says this: > > > > > > " Nonblocking calls allocate a communication request object and > > > associate it with the request handle the argument request). " > > > [http://www.open-mpi.org/doc/v1.4/man3/MPI_Isend.3.php] and > > > > > > " [...] if the communication object was created by a nonblocking > > send or > > > receive, then it is deallocated and the request handle is set to > > > MPI_REQUEST_NULL." > > > [http://www.open-mpi.org/doc/v1.4/man3/MPI_Test.3.php] and (in slightly > > > different words) [http://www.open-mpi.org/doc/v1.4/man3/MPI_Wait.3.php] > > > > > > So I thought that it might do some kind of optimized memory stuff > > > internally. > > > > > > I also tried allocating req (for each used MPI_Request) once before the > > > first use and deallocation after the last use (which I thought was the > > > way it was supposed to work), but that crashes also. > > > > > > I tried replacing the pointers through global variables > > > > > > MPI_Request req; > > > > > > which didn't do the job... > > > > > > The only thing that seems to work is what I mentioned below: Allocate > > > every time I am going to need it in the MPI_Isend/recv, use it in > > > MPI_Test/Wait and after that deallocate it by hand each time. > > > I don't think that this is supposed to be like this since I have to > > do a > > > call to malloc and free so often (for multiple MPI_Request objects in > > > each iteration) that it will most likely limit performance... > > > > > > Anyway I still have the same problem and am still unclear on what kind > > > of memory allocation I should be doing for the MPI_Requests. Is there > > > anything else (besides MPI_THREAD_MULTIPLE support, thread access > > > control, sequential order of MPI_Isend/recv, MPI_Test and MPI_Wait for > > > one MPI_Request object) I need to take care of? If not, what could I do > > > to find the source of my problem? > > > > > > Thanks again for any kind of help! > > > > > > Kind regards, > > > David > > > > > > > > > > > > > > From an implementation perspective, your code is correct only if > > you initialize the MPI library with MPI_THREAD_MULTIPLE and if the > > library accepts. Otherwise, there is an assumption that the > > application is single threaded, or that the MPI behavior is > > implementation dependent. Please read the MPI standard regarding to > > MPI_Init_thread for more details. > > > > > > > > Regards, > > > > george. > > > > > > > > On May 19, 2011, at 02:34 , David B?ttner wrote: > > > > > > > >> Hello, > > > >> > > > >> I am working on a hybrid MPI (OpenMPI 1.4.3) and Pthread code. I > > am using MPI_Isend and MPI_Irecv for communication and > > MPI_Test/MPI_Wait to check if it is done. I do this repeatedly in the > > outer loop of my code. The MPI_Test is used in the inner loop to check > > if some function can be called which depends on the received data. > > > >> The program regularly crashed (only when not using printf...) and > > after debugging it I figured out the following problem: > > > >> > > > >> In MPI_Isend I have an invalid read of memory. I fixed the > > problem with not re-using a > > > >> > > > >> MPI_Request req_s, req_r; > > > >> > > > >> but by using > > > >> > > > >> MPI_Request* req_s; > > > >> MPI_Request* req_r > > > >> > > > >> and re-allocating them before the MPI_Isend/recv. > > > >> > > > >> The documentation says, that in MPI_Wait and MPI_Test (if > > successful) the request-objects are deallocated and set to > > MPI_REQUEST_NULL. > > > >> It also says, that in MPI_Isend and MPI_Irecv, it allocates the > > Objects and associates it with the request object. > > > >> > > > >> As I understand this, this either means I can use a pointer to > > MPI_Request which I don't have to initialize for this (it doesn't work > > but crashes), or that I can use a MPI_Request pointer which I have > > initialized with malloc(sizeof(MPI_REQUEST)) (or passing the address > > of a MPI_Request req), which is set and unset in the functions. But > > this version crashes, too. > > > >> What works is using a pointer, which I allocate before the > > MPI_Isend/recv and which I free after MPI_Wait in every iteration. In > > other words: It only uses if I don't reuse any kind of MPI_Request. > > Only if I recreate one every time. > > > >> > > > >> Is this, what is should be like? I believe that a reuse of the > > memory would be a lot more efficient (less calls to malloc...). Am I > > missing something here? Or am I doing something wrong? > > > >> > > > >> > > > >> Let me provide some more detailed information about my problem: > > > >> > > > >> I am running the program on a 30 node infiniband cluster. Each > > node has 4 single core Opteron CPUs. I am running 1 MPI Rank per node > > and 4 threads per rank (-> one thread per core). > > > >> I am compiling with mpicc of OpenMPI using gcc below. > > > >> Some pseudo-code of the program can be found at the end of this > > e-mail. > > > >> > > > >> I was able to reproduce the problem using different amount of > > nodes and even using one node only. The problem does not arise when I > > put printf-debugging information into the code. This pointed me into > > the direction that I have some memory problem, where some write > > accesses some memory it is not supposed to. > > > >> I ran the tests using valgrind with --leak-check=full and > > --show-reachable=yes, which pointed me either to MPI_Isend or MPI_Wait > > depending on whether I had the threads spin in a loop for MPI_Test to > > return success or used MPI_Wait respectively. > > > >> > > > >> I would appreciate your help with this. Am I missing something > > important here? Is there a way to re-use the request in the different > > iterations other than I thought it should work? > > > >> Or is there a way to re-initialize the allocated memory before > > the MPI_Isend/recv so that I at least don't have to call free and > > malloc each time? > > > >> > > > >> Thank you very much for your help! > > > >> Kind regards, > > > >> David B?ttner > > > >> > > > >> _____________________ > > > >> Pseudo-Code of program: > > > >> > > > >> MPI_Request* req_s; > > > >> MPI_Request* req_w; > > > >> OUTER-LOOP > > > >> if(0 == threadid) > > > >> { > > > >> req_s = malloc(sizeof(MPI_Request)); > > > >> req_r = malloc(sizeof(MPI_Request)); > > > >> MPI_Isend(..., req_s) > > > >> MPI_Irecv(..., req_r) > > > >> } > > > >> pthread_barrier > > > >> INNER-LOOP (while NOT_DONE or RET) > > > >> if(TRYLOCK&& NOT_DONE) > > > >> { > > > >> if(MPI_TEST(req_r)) > > > >> { > > > >> Call_Function_A; > > > >> NOT_DONE = 0; > > > >> } > > > >> > > > >> } > > > >> RET = Call_Function_B; > > > >> } > > > >> pthread_barrier_wait > > > >> if(0 == threadid) > > > >> { > > > >> MPI_WAIT(req_s) > > > >> MPI_WAIT(req_r) > > > >> free(req_s); > > > >> free(req_r); > > > >> } > > > >> _____________ > > > >> > > > >> > > > >> -- > > > >> David B?ttner, Informatik, Technische Universit?t M?nchen > > > >> TUM I-10 - FMI 01.06.059 - Tel. 089 / 289-17676 > > > >> > > > >> _______________________________________________ > > > >> users mailing list > > > >> us...@open-mpi.org > > > >> http://www.open-mpi.org/mailman/listinfo.cgi/users > > > > "To preserve the freedom of the human mind then and freedom of the > > press, every spirit should be ready to devote itself to martyrdom; for > > as long as we may think as we will, and speak as we think, the > > condition of man will proceed in improvement." > > > > -- Thomas Jefferson, 1799 > > > > > > > > > > > > _______________________________________________ > > > > users mailing list > > > > us...@open-mpi.org > > > > http://www.open-mpi.org/mailman/listinfo.cgi/users > > > > > > -- > > > David B?ttner, Informatik, Technische Universit?t M?nchen > > > TUM I-10 - FMI 01.06.059 - Tel. 089 / 289-17676 > > > > > > > > > > > > ------------------------------ > > > > > > Message: 11 > > > Date: Fri, 20 May 2011 06:23:21 -0400 > > > From: Jeff Squyres <jsquy...@cisco.com> > > > Subject: Re: [OMPI users] Trouble with MPI-IO > > > To: Open MPI Users <us...@open-mpi.org> > > > Message-ID: <a5b121e9-e664-49d0-ae54-2cfe52712...@cisco.com> > > > Content-Type: text/plain; charset=us-ascii > > > > > > On May 19, 2011, at 11:24 PM, Tom Rosmond wrote: > > > > > > > What fortran compiler did you use? > > > > > > gfortran. > > > > > > > In the original script my Intel compile used the -132 option, > > > > allowing up to that many columns per line. > > > > > > Gotcha. > > > > > > >> x.f90:99.77: > > > >> > > > >> call > > mpi_type_indexed(lenij,ijlena,ijdisp,mpi_real,ij_vector_type,ierr) > > > >> 1 > > > >> Error: There is no specific subroutine for the generic > > 'mpi_type_indexed' at (1) > > > > > > > > Hmmm, very strange, since I am looking right at the MPI standard > > > > documents with that routine documented. I too get this compile failure > > > > when I switch to 'use mpi'. Could that be a problem with the Open MPI > > > > fortran libraries??? > > > > > > I think that that error is telling us that there's a compile-time > > mismatch -- that the signature of what you've passed doesn't match the > > signature of OMPI's MPI_Type_indexed subroutine. > > > > > > >> I looked at our mpi F90 module and see the following: > > > >> > > > >> interface MPI_Type_indexed > > > >> subroutine MPI_Type_indexed(count, array_of_blocklengths, > > array_of_displacements, oldtype, newtype, ierr) > > > >> integer, intent(in) :: count > > > >> integer, dimension(*), intent(in) :: array_of_blocklengths > > > >> integer, dimension(*), intent(in) :: array_of_displacements > > > >> integer, intent(in) :: oldtype > > > >> integer, intent(out) :: newtype > > > >> integer, intent(out) :: ierr > > > >> end subroutine MPI_Type_indexed > > > >> end interface > > > > > > Shouldn't ijlena and ijdisp be 1D arrays, not 2D arrays? > > > > > > -- > > > Jeff Squyres > > > jsquy...@cisco.com > > > For corporate legal information go to: > > > http://www.cisco.com/web/about/doing_business/legal/cri/ > > > > > > > > > > > > > > > ------------------------------ > > > > > > Message: 12 > > > Date: Fri, 20 May 2011 07:26:19 -0400 > > > From: Jeff Squyres <jsquy...@cisco.com> > > > Subject: Re: [OMPI users] MPI_Alltoallv function crashes when np > 100 > > > To: Open MPI Users <us...@open-mpi.org> > > > Message-ID: <f9f71854-b9dd-459f-999d-8a8aef8d6...@cisco.com> > > > Content-Type: text/plain; charset=GB2312 > > > > > > I missed this email in my INBOX, sorry. > > > > > > Can you be more specific about what exact error is occurring? You > > just say that the application crashes...? Please send all the > > information listed here: > > > > > > http://www.open-mpi.org/community/help/ > > > > > > > > > On Apr 26, 2011, at 10:51 PM, ?????? wrote: > > > > > > > It seems that the const variable SOMAXCONN who used by listen() > > system call causes this problem. Can anybody help me resolve this > > question? > > > > > > > > 2011/4/25 ?????? <xjun.m...@gmail.com> > > > > Dear all, > > > > > > > > As I mentioned, when I mpiruned an application with the parameter > > "np = 150(or bigger)", the application who used the MPI_Alltoallv > > function would carsh. The problem would recur no matter how many nodes > > we used. > > > > > > > > The edition of OpenMPI: 1.4.1 or 1.4.3 > > > > The OS: linux redhat 2.6.32 > > > > > > > > BTW, my nodes had enough memory to run the application, and the > > MPI_Alltoall function worked well at my environment. > > > > Did anybody meet the same problem? Thanks. > > > > > > > > > > > > Best Regards > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > users mailing list > > > > us...@open-mpi.org > > > > http://www.open-mpi.org/mailman/listinfo.cgi/users > > > > > > > > > -- > > > Jeff Squyres > > > jsquy...@cisco.com > > > For corporate legal information go to: > > > http://www.cisco.com/web/about/doing_business/legal/cri/ > > > > > > > > > > > > > > > ------------------------------ > > > > > > Message: 13 > > > Date: Fri, 20 May 2011 07:28:28 -0400 > > > From: Jeff Squyres <jsquy...@cisco.com> > > > Subject: Re: [OMPI users] MPI_ERR_TRUNCATE with MPI_Allreduce() error, > > > but only sometimes... > > > To: Open MPI Users <us...@open-mpi.org> > > > Message-ID: <caef632e-757b-49ee-b545-5cccbc712...@cisco.com> > > > Content-Type: text/plain; charset=us-ascii > > > > > > Sorry for the super-late reply. :-\ > > > > > > Yes, ERR_TRUNCATE means that the receiver didn't have a large enough > > buffer. > > > > > > Have you tried upgrading to a newer version of Open MPI? 1.4.3 is > > the current stable release (I have a very dim and not guaranteed to be > > correct recollection that we fixed something in the internals of > > collectives somewhere with regards to ERR_TRUNCATE...?). > > > > > > > > > On Apr 25, 2011, at 4:44 PM, Wei Hao wrote: > > > > > > > Hi: > > > > > > > > I'm running openmpi 1.2.8. I'm working on a project where one part > > involves communicating an integer, representing the number of data > > points I'm keeping track of, to all the processors. The line is simple: > > > > > > > > MPI_Allreduce(&np,&geo_N,1,MPI_INT,MPI_MAX,MPI_COMM_WORLD); > > > > > > > > where np and geo_N are integers, np is the result of a local > > calculation, and geo_N has been declared on all the processors. geo_N > > is nondecreasing. This line works the first time I call it (geo_N goes > > from 0 to some other integer), but if I call it later in the program, > > I get the following error: > > > > > > > > > > > > [woodhen-039:26189] *** An error occurred in MPI_Allreduce > > > > [woodhen-039:26189] *** on communicator MPI_COMM_WORLD > > > > [woodhen-039:26189] *** MPI_ERR_TRUNCATE: message truncated > > > > [woodhen-039:26189] *** MPI_ERRORS_ARE_FATAL (goodbye) > > > > > > > > > > > > As I understand it, MPI_ERR_TRUNCATE means that the output buffer > > is too small, but I'm not sure where I've made a mistake. It's > > particularly frustrating because it seems to work fine the first time. > > Does anyone have any thoughts? > > > > > > > > Thanks > > > > Wei > > > > _______________________________________________ > > > > users mailing list > > > > us...@open-mpi.org > > > > http://www.open-mpi.org/mailman/listinfo.cgi/users > > > > > > > > > -- > > > Jeff Squyres > > > jsquy...@cisco.com > > > For corporate legal information go to: > > > http://www.cisco.com/web/about/doing_business/legal/cri/ > > > > > > > > > > > > > > > ------------------------------ > > > > > > Message: 14 > > > Date: Fri, 20 May 2011 08:14:07 -0400 > > > From: Jeff Squyres <jsquy...@cisco.com> > > > Subject: Re: [OMPI users] Trouble with MPI-IO > > > To: Open MPI Users <us...@open-mpi.org> > > > Message-ID: <42db03b3-9cf4-4acb-aa20-b857e5f76...@cisco.com> > > > Content-Type: text/plain; charset="us-ascii" > > > > > > On May 20, 2011, at 6:23 AM, Jeff Squyres wrote: > > > > > > > Shouldn't ijlena and ijdisp be 1D arrays, not 2D arrays? > > > > > > Ok, if I convert ijlena and ijdisp to 1D arrays, I don't get the > > compile error (even though they're allocatable -- so allocate was a > > red herring, sorry). That's all that "use mpi" is complaining about -- > > that the function signatures didn't match. > > > > > > use mpi is your friend -- even if you don't use F90 constructs much. > > Compile-time checking is Very Good Thing (you were effectively > > "getting lucky" by passing in the 2D arrays, I think). > > > > > > Attached is my final version. And with this version, I see the hang > > when running it with the "T" parameter. > > > > > > That being said, I'm not an expert on the MPI IO stuff -- your code > > *looks* right to me, but I could be missing something subtle in the > > interpretation of MPI_FILE_SET_VIEW. I tried running your code with > > MPICH 1.3.2p1 and it also hung. > > > > > > Rob (ROMIO guy) -- can you comment this code? Is it correct? > > > > > > -- > > > Jeff Squyres > > > jsquy...@cisco.com > > > For corporate legal information go to: > > > http://www.cisco.com/web/about/doing_business/legal/cri/ > > > -------------- next part -------------- > > > A non-text attachment was scrubbed... > > > Name: x.f90 > > > Type: application/octet-stream > > > Size: 3820 bytes > > > Desc: not available > > > URL: > > <http://www.open-mpi.org/MailArchives/users/attachments/20110520/53a5461b/attachment.obj> > > > > > > ------------------------------ > > > > > > _______________________________________________ > > > users mailing list > > > us...@open-mpi.org > > > http://www.open-mpi.org/mailman/listinfo.cgi/users > > > > > > End of users Digest, Vol 1911, Issue 1 > > > ************************************** > > > > > > _______________________________________________ > > users mailing list > > us...@open-mpi.org > > http://www.open-mpi.org/mailman/listinfo.cgi/users > -------------- next part -------------- > HTML attachment scrubbed and removed > > ------------------------------ > > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users > > End of users Digest, Vol 1911, Issue 4 > **************************************
