I'm running OpenMPI 1.2.6 under Red Hat Enterprise Linux Server
release 5.2 on an x86_64 cluster.
When I send a message with MPI_Isend I think it should eventually be
delivered (if I have a receive waiting), without my having to make any
other MPI calls. This appears to be guaranteed by the spec. In MPI
version 1.1, section 3.7.4, Semantics of Nonblocking Communications,
it says
Progress A call to MPI_WAIT that completes a receive will eventually
terminate and return if a matching send has been started, unless the
send is satisfied by another receive. In particular, if the matching
send is nonblocking, then the receive should complete even if no call
is executed by the sender to complete the send.
This appears never to work when my two processes are on different
nodes. I enclose a test case below. In this simple case, I can work
around the problem by waiting for the send to complete, but in general
after a bunch of communication I don't know any way that I can make
sure that all my sent messages have actually been sent, without
blocking.
In the following code, rank 0 sends a message to rank 7, sleeps for 5
seconds, and then calls MPI_Finalize. The output below shows that
rank 7 doesn't receive the message until finalize is called.
(Ranks 1-6 exist only to get the scheduler here to dispatch 0 and 7
to different nodes.)
Ken Olum
----------------------------------------------------------------------
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <time.h>
#include "mpi.h"
char *timestamp()
{ time_t now;
struct tm *data;
char *result;
time(&now);
data = localtime(&now);
result = malloc(9);
sprintf(result, "%2d:%02d:%02d", data->tm_hour, data->tm_min, data->tm_sec);
return result;
}
main( argc, argv )
int argc;
char **argv;
{
char message[20];
int myrank, mysize, flag, i;
MPI_Status status;
MPI_Request request;
MPI_Init(&argc, &argv);
MPI_Comm_size(MPI_COMM_WORLD, &mysize);
MPI_Comm_rank(MPI_COMM_WORLD, &myrank);
printf("Proc %d, %s: initialized\n", myrank, timestamp());
if (myrank == 0) /* code for process zero */
{
strcpy(message,"TEST");
printf("Proc %d, %s: sending '%s'\n", myrank, timestamp(), message);
MPI_Isend(message, strlen(message)+1, MPI_CHAR, mysize-1, 99,
MPI_COMM_WORLD, &request);
printf("Proc %d, %s: sent\n", myrank, timestamp());
}
else if (myrank == mysize-1) /* code for last process */
{
printf("Proc %d, %s: receiving\n", myrank, timestamp());
MPI_Irecv(message, 20, MPI_CHAR, 0, 99, MPI_COMM_WORLD, &request); /*
Start it */
MPI_Wait(&request, &status); /* Wait for it */
printf("Proc %d, %s: received '%s'\n", myrank, timestamp(), message);
}
printf("Proc %d, %s: sleeping\n", myrank, timestamp());
sleep(5);
printf("Proc %d, %s: finalizing\n", myrank, timestamp());
MPI_Finalize();
}
----------------------------------------------------------------------
Proc 2, 16:16:19: initialized
Proc 2, 16:16:19: sleeping
Proc 3, 16:16:19: initialized
Proc 3, 16:16:19: sleeping
Proc 0, 16:16:19: initialized
Proc 0, 16:16:19: sending 'TEST'
Proc 0, 16:16:19: sent
Proc 0, 16:16:19: sleeping
Proc 5, 16:16:19: initialized
Proc 5, 16:16:19: sleeping
Proc 6, 16:16:19: initialized
Proc 6, 16:16:19: sleeping
Proc 7, 16:16:19: initialized
Proc 7, 16:16:19: receiving
Proc 1, 16:16:19: initialized
Proc 1, 16:16:19: sleeping
Proc 4, 16:16:19: initialized
Proc 4, 16:16:19: sleeping
Proc 3, 16:16:24: finalizing
Proc 5, 16:16:24: finalizing
Proc 0, 16:16:24: finalizing
Proc 2, 16:16:24: finalizing
Proc 6, 16:16:24: finalizing
Proc 1, 16:16:24: finalizing
Proc 4, 16:16:24: finalizing
Proc 7, 16:16:24: received 'TEST'
Proc 7, 16:16:24: sleeping
Proc 7, 16:16:29: finalizing
