Dick is, of course, correct. This topic has come up several times on this list: Open MPI currently does not do this kind of check. It is therefore possible for a sender to exhaust memory on a receiver if, for example, it continually sends short/eager messages that the receiver consumes off the socket, but does not MPI match (i.e., they're unexpected messages).
However, the kernel crash you're describing is not likely associated with this issue. 1. If you have a busy sender that sends oodles of short messages across TCP sockets to a receiver that is not even going into the MPI layer to progress anything from that socket, the kernel/TCP wire protocol will block the sender when the receiver's kernel buffer fills up. Hence, Open MPI will stop sending and buffer up all the pending messages on the send side. Linux defaults to 64k kernel buffers (IIRC). 2. If you have a busy sender that sends oodles of short messages across TCP sockets to a receiver that *is* going into MPI to make progress, but is not matching any of these short messages (i.e., the receiver is reading messages off the socket but not successfully MPI matching them), then the socket will never block because the kernel socket buffering is not filling up. Open MPI will continually buffer the messages on the receiver until a matching MPI receive is issued. If the effort of buffering all of these incoming messages consumes all of the receiver's memory, malloc() will fail in Open MPI and the MPI process will likely die. It would be hard to imagine that this crashes your kernel. I simplified the discussion a bit (i.e., didn't describe *all* possible cases), but it's good enough for this discussion. On May 11, 2010, at 9:38 AM, Richard Treumann wrote: > The MPI standard requires that when there is a free running task posting > isends to a task that is not keeping up on receives, the sending task will > switch to synchronous isend BEFORE the receive side runs out of memory and > fails. > > There should be no need for the sender to us MPI_Issend because the MPI > library should do it for you (under the covers) > > > Dick Treumann - MPI Team > IBM Systems & Technology Group > Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 > Tele (845) 433-7846 Fax (845) 433-8363 > > > <graycol.gif>Gijsbert Wiesenekker ---05/11/2010 03:19:49 AM---An OpenMPI > program of mine that uses MPI_Isend and MPI_Irecv crashes after some > non-reproducible tim > > <ecblank.gif> > From: <ecblank.gif> > Gijsbert Wiesenekker <gijsbert.wiesenek...@gmail.com> > <ecblank.gif> > To: <ecblank.gif> > Open MPI Users <us...@open-mpi.org> > <ecblank.gif> > Date: <ecblank.gif> > 05/11/2010 03:19 AM > <ecblank.gif> > Subject: <ecblank.gif> > [OMPI users] Questions about MPI_Isend > <ecblank.gif> > Sent by: <ecblank.gif> > users-boun...@open-mpi.org > > > > An OpenMPI program of mine that uses MPI_Isend and MPI_Irecv crashes after > some non-reproducible time my Fedora Linux kernel (invalid opcode), which > makes it hard to debug (there is no trace, even with the debug kernel, and if > I run it under valgrind it does not crash). > My guess is that the kernel crash is caused by OpenMPI running out if memory > because too many MPI_Irecv messages have been sent but not been processed yet. > My questions are: > What does the OpenMPI specification say about the behaviour of MPI_Isend when > many messages have been sent but have not been processed yet? Will it fail? > Will it block until more memory becomes available (I hope not, because this > would cause my program to deadlock)? > Ideally I would like to check how many MPI_Isend messages have not been > processed yet, so that I can stop sending messages if there are 'too many' > waiting. Is there a way to do this? > > Regards, > Gijsbert > > > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users > > > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
