On May 24, 2010, at 20:27 , Eugene Loh wrote: > Gijsbert Wiesenekker wrote: > >> My MPI program consists of a number of processes that send 0 or more >> messages (using MPI_Isend) to 0 or more other processes. The processes check >> periodically if messages are available to be processed. It was running fine >> until I increased the message size, and I got deadlock problems. Googling >> learned I was running into a classic deadlock problem if (see for example >> http://www.cs.ucsb.edu/~hnielsen/cs140/mpi-deadlocks.html). The workarounds >> suggested like changing the order of MPI_Send and MPI_Recv do not work in my >> case, as it could be that one processor does not send any message at all to >> the other processes, so MPI_Recv would wait indefinitely. >> Any suggestions on how to avoid deadlock in this case? >> > The problems you describe would seem to arise with blocking functions like > MPI_Send and MPI_Recv. With the non-blocking variants MPI_Isend/MPI_Irecv, > there shouldn't be this problem. There should be no requirement of ordering > the functions in the way that web page describes... that workaround is > suggested for the blocking calls. It feels to me that something is missing > from your description. > > If you know the maximum size any message will be, you can post an MPI_Irecv > with wild card tags and source ranks. You can post MPI_Isend calls for > whatever messages you want to send. You can use MPI_Test to check if any > message has been received; if so, process the received message and re-post > the MPI_Irecv. You can use MPI_Test to check if any send messages have > completed; if so, you can reuse those send buffers. You need some signal to > indicate to processes that no further messages will be arriving. > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users
My program was running fine using the methods you describe (MPI_Isend/MPI_Test/MPI_Irecv), until I increased the message size. My program was not running very efficient because of the MPI overhead associated with sending/receiving a large number of small messages. So I decided to combine messages before sending them, and then I got the deadlock problems: the MPI_Test calls never returned true, so the MPI_Isend calls never completed. As described on the link given above, the reason was that I exhausted the MPI system buffer space, in combination with the unsafe ordering of the send/receive calls (but I cannot see how I can change that order given the nature of my program). See for example also http://publib.boulder.ibm.com/infocenter/clresctr/vxrx/index.jsp?topic=/com.ibm.cluster.pe.doc/pe_422/am10600481.html: 'Destination buffer space unavailability cannot cause a safe MPI program to fail, but could cause hangs in unsafe MPI programs. An unsafe program is one that assumes MPI can guarantee system buffering of sent data until the receive is posted.' Gijsbert
