[openib-general] Fork issues with simple MPI program

[Arlin Davis]

We are seeing some fork issues with a simple MPI program (attached) running on 
a 2.6.16+ kernels and
OFED 1.1. We have tried both Intel MPI and mvapich2 with the same results:

t_fork> mpiexec -n 2 t_system_fork                                              
              
parent process
[0] started child process with pid=31552
send desc error
parent process
[0] Abort: [] Got completion with error 1, vendor code=69, dest rank=1
 at line 540 in file ibv_channel_manager.c
[1] I am child process with pid=25437
[1] started child process with pid=25437
[0] I am child process with pid=31552
child process
[1] finished pid=25437
child process
[0] finished pid=31552

rank 0 in job 2  svlmpicl400_32925   caused collective abort of all ranks
  exit status of rank 0: return code 252

If you run mvapich2 for uDAPL, it hangs before second MPI_Barrier() just like 
Intel MPI. If you use
the I_MPI_RDMA_USE_EVD_FALLBACK=1 option with Intel MPI you get the following 
error similar to
mvapich2:

parent process
parent process
[0] I am child process with pid=9596
[0] started child process with pid=9596
[1] I am child process with pid=11477
[1] started child process with pid=11477
[0][rdma_iba.c:1007] Intel MPI fatal error: DTO operation completed with error. 
status=0x2.
cookie=0x1
[1][rdma_iba.c:1007] Intel MPI fatal error: DTO operation completed with error. 
status=0x2.
cookie=0x1
child process
[1] finished pid=11477
child process
[0] finished pid=9596
rank 0 in job 8  cst-19_54707   caused collective abort of all ranks
  exit status of rank 0: return code 255

Any insight would be greatly appreciated. It was our assumption that the parent 
process can continue
to use IB resources after the fixes went into 2.6.16 and OFED 1.1. Is this 
true? 

Thanks,

-arlin

#include "mpi.h"
#include <stdio.h>
#include <stdlib.h>


int main(int argc,char *argv[])
{
    int  myid, numprocs;
    pid_t pid;

    MPI_Init(&argc,&argv);
    MPI_Comm_size(MPI_COMM_WORLD,&numprocs);
    MPI_Comm_rank(MPI_COMM_WORLD,&myid);

    MPI_Barrier(MPI_COMM_WORLD);

    system("echo parent process");
    
    pid = fork();
    
    if( pid == 0)
    {
        pid = getpid();
        printf("[%d] I am child process with pid=%d\n", myid, pid);    
        system("echo child process");
        
    } else
    {
        printf("[%d] started child process with pid=%d\n", myid, pid);
        MPI_Barrier(MPI_COMM_WORLD);
        MPI_Finalize();
        pid = getpid();
    }
    
    printf("[%d] finished pid=%d\n", myid, pid);
    return 0;
}

_______________________________________________
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general