Dear all,

My MPI application spawns a large number of MPI processes using MPI_Comm_spawn 
over its total lifetime. Unfortunately, I have experienced that this results in 
problems for all currently supported OpenMPI versions (2.1, 3.0, 3.1 and 4.0). 
I have written a short, self-contained program in C (included below) that 
spawns child processes using MPI_Comm_spawn in an infinite loop, where each 
child process exits after writing a message to stdout. This short program leads 
to the following issues:
In versions 2.1.2 (Ubuntu package) and 2.1.5 (compiled from source), the 
program leads to a pipe leak where pipes keep accumulating over time until my 
MPI application crashes because the maximum number of pipes has been reached.
In versions 3.0.3 and 3.1.3 (both compiled from source), there appears to be no 
pipe leak, but the program crashes with the following error message:
PMIX_ERROR: UNREACHABLE in file ptl_tcp_component.c at line 1257

In version 4.0.0 (compiled from source), I have not been able to test this 
issue very thoroughly because mpiexec ignores the --oversubscribe command-line 
flag (as detailed in this GitHub issue This prohibits the 
oversubscription of processor cores, which means that spawning additional 
processes immediately results in an error because "not enough slots" are 
available. A fix for this was proposed recently 
(, but since the v4.0.x developer 
branch is being actively developed right now, I decided not go into it.
I have found one e-mail thread on this mailing list about a similar problem 
( In this 
thread, Ralph Castain states that this is a known issue and suggests that it is 
fixed in the then upcoming v1.3.x release. However, version 1.3 is no longer 
supported and the issue has reappeared, hence this did not solve the issue.
I have created a GitHub gist that contains the output from "ompi_info --all" of 
all the OpenMPI installations mentioned here, as well as the config.log files 
for the OpenMPI installations that I compiled from source:
I have also attached the code for the short program that demonstrates these 
issues. For good measure, I have included it directly here as well:
#include <stdio.h>
#include <mpi.h>

int main(int argc, char *argv[]) {
// Initialize MPI

// Get parent
MPI_Comm parent;

// If the process was not spawned
if (parent == MPI_COMM_NULL) {

puts("I was not spawned!");
// Spawn child process in loop
char *cmd = argv[0];
char **cmd_argv = MPI_ARGV_NULL;
int maxprocs = 1;
MPI_Info info = MPI_INFO_NULL;
int root = 0;
MPI_Comm comm = MPI_COMM_SELF;
MPI_Comm intercomm;
int *array_of_errcodes = MPI_ERRCODES_IGNORE;

for (;;) {
MPI_Comm_spawn(cmd, cmd_argv, maxprocs, info, root, comm,
&intercomm, array_of_errcodes);


// If process was spawned
} else {

puts("I was spawned!");

// Finalize


Thanks in advance and best wishes,
Thomas Pak

Attachment: mpi-spawn.c
Description: Binary data

users mailing list

Reply via email to