Howard - if there is a problem in PMIx that is causing this problem, then we really could use a report on it ASAP as we are getting ready to release v3.1.6 and I doubt we have addressed anything relevant to what is being discussed here.
On Aug 11, 2020, at 4:35 PM, Martín Morales via users <users@lists.open-mpi.org <mailto:users@lists.open-mpi.org> > wrote: Hi Howard. Great!, that works for the crashing problem with OMPI 4.0.4. However It stills hanging if I remove “master” (host which launches spawning processes) from my hostfile. I need spawn only in “worker”. Is there a way or workaround for doing this without mpirun? Thanks a lot for your assistance. Martín From: Howard Pritchard <mailto:hpprit...@gmail.com> Sent: lunes, 10 de agosto de 2020 19:13 To: Martín Morales <mailto:martineduardomora...@hotmail.com> Cc: Open MPI Users <mailto:users@lists.open-mpi.org> Subject: Re: [OMPI users] OMPI 4.0.4 crashes (or hangs) with dynamically processes allocation. OMPI 4.0.1 don't. Hi Martin, I was able to reproduce this with 4.0.x branch. I'll open an issue. If you really want to use 4.0.4, then what you'll need to do is build an external PMIx 3.1.2 (the PMIx that was embedded in Open MPI 4.0.1), and then build Open MPI using the --with-pmix=where your pmix is installed You will also need to build both Open MPI and PMIx against the same libevent. There's a configure option with both packages to use an external libevent installation. Howard Am Mo., 10. Aug. 2020 um 13:52 Uhr schrieb Martín Morales <martineduardomora...@hotmail.com <mailto:martineduardomora...@hotmail.com> >: Hi Howard. Unfortunately the issue persists in OMPI 4.0.5rc1. Do I have to post this on the bug section? Thanks and regards. Martín From: Howard Pritchard <mailto:hpprit...@gmail.com> Sent: lunes, 10 de agosto de 2020 14:44 To: Open MPI Users <mailto:users@lists.open-mpi.org> Cc: Martín Morales <mailto:martineduardomora...@hotmail.com> Subject: Re: [OMPI users] OMPI 4.0.4 crashes (or hangs) with dynamically processes allocation. OMPI 4.0.1 don't. Hello Martin, Between Open MPI 4.0.1 and Open MPI 4.0.4 we upgraded the internal PMIx version that introduced a problem with spawn for the 4.0.2-4.0.4 versions. This is supposed to be fixed in the 4.0.5 release. Could you try the 4.0.5rc1 tarball and see if that addresses the problem you're seeing? https://www.open-mpi.org/software/ompi/v4.0/ Howard Am Do., 6. Aug. 2020 um 09:50 Uhr schrieb Martín Morales via users <users@lists.open-mpi.org <mailto:users@lists.open-mpi.org> >: Hello people! I'm using OMPI 4.0.4 in a very simple scenario. Just 2 machines, one "master", one "worker" on a Ethernet LAN. Both with Ubuntu 18.04.I builded OMPI just like this: ./configure --prefix=/usr/local/openmpi-4.0.4/bin/ My hostfile is this: master slots=2 worker slots=2 I'm trying to dynamically allocate the processes with MPI_Comm_Spawn(). If I launch the processes only on the "master" machine It's ok. But if I use the hostfile crashes with this: -------------------------------------------------------------------------- At least one pair of MPI processes are unable to reach each other for MPI communications. This means that no Open MPI device has indicated that it can be used to communicate between these processes. This is an error; Open MPI requires that all MPI processes be able to reach each other. This error can sometimes be the result of forgetting to specify the "self" BTL. Process 1 ([[35155,2],1]) is on host: nos-GF7050VT-M Process 2 ([[35155,1],0]) is on host: unknown! BTLs attempted: tcp self Your MPI job is now going to abort; sorry. -------------------------------------------------------------------------- [nos-GF7050VT-M:22526] [[35155,2],1] ORTE_ERROR_LOG: Unreachable in file dpm/dpm.c at line 493 -------------------------------------------------------------------------- It looks like MPI_INIT failed for some reason; your parallel process is likely to abort. There are many reasons that a parallel process can fail during MPI_INIT; some of which are due to configuration or environment problems. This failure appears to be an internal failure; here's some additional information (which may only be relevant to an Open MPI developer): ompi_dpm_dyn_init() failed --> Returned "Unreachable" (-12) instead of "Success" (0) -------------------------------------------------------------------------- [nos-GF7050VT-M:22526] *** An error occurred in MPI_Init [nos-GF7050VT-M:22526] *** reported by process [2303918082,1] [nos-GF7050VT-M:22526] *** on a NULL communicator [nos-GF7050VT-M:22526] *** Unknown error [nos-GF7050VT-M:22526] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, [nos-GF7050VT-M:22526] *** and potentially your MPI job) Note: host "nos-GF7050VT-M" is "worker" But If I run without "master" in hostfile, the processes are launched but It hangs: MPI_Init() doesn't returns. I launched the script (pasted below) in this 2 ways with the same result: $ ./simple_spawn 2 $ mpirun -np 1 ./simple_spawn 2 The "simple_spawn" script: #include "mpi.h" #include <stdio.h> #include <stdlib.h> int main(int argc, char ** argv){ int processesToRun; MPI_Comm parentcomm, intercomm; MPI_Info info; int rank, size, hostName_len; char hostName[200]; MPI_Init( &argc, &argv ); MPI_Comm_get_parent( &parentcomm ); MPI_Comm_rank(MPI_COMM_WORLD, &rank); MPI_Comm_size(MPI_COMM_WORLD, &size); MPI_Get_processor_name(hostName, &hostName_len); if (parentcomm == MPI_COMM_NULL) { if(argc < 2 ){ printf("Processes number needed!"); return 0; } processesToRun = atoi(argv[1]); MPI_Info_create( &info ); MPI_Info_set( info, "hostfile", "./hostfile" ); MPI_Info_set( info, "map_by", "node" ); MPI_Comm_spawn( argv[0], MPI_ARGV_NULL, processesToRun, info, 0, MPI_COMM_WORLD, &intercomm, MPI_ERRCODES_IGNORE); printf("I'm the parent.\n"); } else { printf("I'm the spawned h: %s r/s: %i/%i.\n", hostName, rank, size ); } fflush(stdout); MPI_Finalize(); return 0; } I came from OMPI 4.0.1. In this version It's working... with some inconsistencies I'm afraid. That's why I decided to upgrade to OMPI 4.0.4. I tried several versions with no luck. Is there maybe an intrinsic problem with the OMPI dynamic allocation functionality? Any help will be very appreciated. Best regards. Martín
