It might call disconnect more than once if it creates multiple communicators. Here’s another test case for that behavior:
intercomm_create.c
Description: Binary data
> On Jun 4, 2018, at 7:08 AM, Bennet Fauber <ben...@umich.edu> wrote: > > Just out of curiosity, but would using Rmpi and/or doMPI help in any way? > > -- bennet > > > On Mon, Jun 4, 2018 at 10:00 AM, marcin.krotkiewski > <marcin.krotkiew...@gmail.com> wrote: >> Thanks, Ralph! >> >> Your code finishes normally, I guess then the reason might be lying in R. >> Running the R code with -mca pmix_base_verbose 1 i see that each rank calls >> ext2x:client disconnect twice (each PID prints the line twice) >> >> [...] >> 3 slaves are spawned successfully. 0 failed. >> [localhost.localdomain:11659] ext2x:client disconnect >> [localhost.localdomain:11661] ext2x:client disconnect >> [localhost.localdomain:11658] ext2x:client disconnect >> [localhost.localdomain:11646] ext2x:client disconnect >> [localhost.localdomain:11658] ext2x:client disconnect >> [localhost.localdomain:11659] ext2x:client disconnect >> [localhost.localdomain:11661] ext2x:client disconnect >> [localhost.localdomain:11646] ext2x:client disconnect >> >> In your example it's only called once per process. >> >> Do you have any suspicion where the second call comes from? Might this be >> the reason for the hang? >> >> Thanks! >> >> Marcin >> >> >> On 06/04/2018 03:16 PM, r...@open-mpi.org wrote: >> >> Try running the attached example dynamic code - if that works, then it >> likely is something to do with how R operates. >> >> >> >> >> >> On Jun 4, 2018, at 3:43 AM, marcin.krotkiewski >> <marcin.krotkiew...@gmail.com> wrote: >> >> Hi, >> >> I have some problems running R + Rmpi with OpenMPI 3.1.0 + PMIx 2.1.1. A >> simple R script, which starts a few tasks, hangs at the end on diconnect. >> Here is the script: >> >> library(parallel) >> numWorkers <- as.numeric(Sys.getenv("SLURM_NTASKS")) - 1 >> myCluster <- makeCluster(numWorkers, type = "MPI") >> stopCluster(myCluster) >> >> And here is how I run it: >> >> SLURM_NTASKS=5 mpirun -np 1 -mca pml ^yalla -mca mtl ^mxm -mca coll ^hcoll R >> --slave < mk.R >> >> Notice -np 1 - this is apparently how you start Rmpi jobs: ranks are spawned >> by R dynamically inside the script. So I ran into a number of issues here: >> >> 1. with HPCX it seems that dynamic starting of ranks is not supported, hence >> I had to turn off all of yalla/mxm/hcoll >> >> -------------------------------------------------------------------------- >> Your application has invoked an MPI function that is not supported in >> this environment. >> >> MPI function: MPI_Comm_spawn >> Reason: the Yalla (MXM) PML does not support MPI dynamic process >> functionality >> -------------------------------------------------------------------------- >> >> 2. when I do that, the program does create a 'cluster' and starts the ranks, >> but hangs in PMIx at MPI Disconnect. Here is the top of the trace from gdb: >> >> #0 0x00007f66b1e1e995 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007f669eaeba5b in PMIx_Disconnect (procs=procs@entry=0x2e25d20, >> nprocs=nprocs@entry=10, info=info@entry=0x0, ninfo=ninfo@entry=0) at >> client/pmix_client_connect.c:232 >> #2 0x00007f669ed6239c in ext2x_disconnect (procs=0x7ffd58322440) at >> ext2x_client.c:1432 >> #3 0x00007f66a13bc286 in ompi_dpm_disconnect (comm=0x2cc0810) at >> dpm/dpm.c:596 >> #4 0x00007f66a13e8668 in PMPI_Comm_disconnect (comm=0x2cbe058) at >> pcomm_disconnect.c:67 >> #5 0x00007f66a16799e9 in mpi_comm_disconnect () from >> /cluster/software/R-packages/3.5/Rmpi/libs/Rmpi.so >> #6 0x00007f66b2563de5 in do_dotcall () from >> /cluster/software/R/3.5.0/lib64/R/lib/libR.so >> #7 0x00007f66b25a207b in bcEval () from >> /cluster/software/R/3.5.0/lib64/R/lib/libR.so >> #8 0x00007f66b25b0fd0 in Rf_eval.localalias.34 () from >> /cluster/software/R/3.5.0/lib64/R/lib/libR.so >> #9 0x00007f66b25b2c62 in R_execClosure () from >> /cluster/software/R/3.5.0/lib64/R/lib/libR.so >> >> Might this also be related to the dynamic rank creation in R? >> >> Thanks! >> >> Marcin >> >> _______________________________________________ >> users mailing list >> users@lists.open-mpi.org >> https://lists.open-mpi.org/mailman/listinfo/users >> >> >> >> _______________________________________________ >> users mailing list >> users@lists.open-mpi.org >> https://lists.open-mpi.org/mailman/listinfo/users >> >> >> >> _______________________________________________ >> users mailing list >> users@lists.open-mpi.org >> https://lists.open-mpi.org/mailman/listinfo/users > _______________________________________________ > users mailing list > users@lists.open-mpi.org > https://lists.open-mpi.org/mailman/listinfo/users
_______________________________________________ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users