We were thinking of doing the same (putting mpirun into its own process group). When doing that, we have to make sure to propagate _all_ relevant signals our top-level wrapper receives to mpirun. That’s also a viable option I guess. Now we have some choices on the table and have to decide which way to go.
Thanks for helping us to understand the behavior of Open MPI and for your suggestions! Moritz -- Moritz Kreutzer Siemens Digital Industries Software Simulation and Test Solutions, Product Development, High Performance Computing Nordostpark 3 90411 Nuremberg, Germany Tel.: +49 (911) 38379 8085 moritz.kreut...@siemens.com<mailto:moritz.kreut...@siemens.com> www.sw.siemens.com<http://www.sw.siemens.com/> From: Ralph Castain <r...@open-mpi.org> Sent: Montag, 6. April 2020 17:06 To: Open MPI Users <users@lists.open-mpi.org> Cc: Kreutzer, Moritz (DI SW STS PRD PLD HPC) <moritz.kreut...@siemens.com> Subject: Re: [OMPI users] Clean termination after receiving multiple SIGINT I don't know that it is officially documented anywhere - it does get printed out when the first CTRL-C arrives. On the plus side, it has been 5 seconds (as opposed to some other time) since the beginning of OMPI, so it is pretty safe to rely on it. I wonder if you could get around this problem another way - what if you pass mpirun the "--set-sid" option? This would put mpirun into its own process group and should (I believe) ensure that it doesn't see the user's CTRL-C so you could always just forward the signal. Would that work for you? On Apr 6, 2020, at 7:57 AM, Kreutzer, Moritz via users <users@lists.open-mpi.org<mailto:users@lists.open-mpi.org>> wrote: Thanks for the explanation, Ralph! I guess the reason we need to pass the signal down is to achieve correct behavior when a signal does not come via CTRL+C, but in case someone kills our top-level script (which eventually calls mpirun) using “kill $PID” or similar, in which case we would have to forward the signal to mpirun. I think it’s a convenience feature not relying on the user having to kill the entire process group, but also achieving the desired behavior when they just kill the top-level script. Maybe we manage to find a way to distinguish signals which are sent only to the wrapper script (which we would want to forward) from signals which are sent to the entire process group (which we would not want to forward). Maybe waiting for 5 seconds would also be a viable workaround. Is this time span documented somewhere? Thanks, Moritz -- Moritz Kreutzer Siemens Digital Industries Software Simulation and Test Solutions, Product Development, High Performance Computing Nordostpark 3 90411 Nuremberg, Germany Tel.: +49 (911) 38379 8085 moritz.kreut...@siemens.com<mailto:moritz.kreut...@siemens.com> www.sw.siemens.com<http://www.sw.siemens.com/> From: users <users-boun...@lists.open-mpi.org<mailto:users-boun...@lists.open-mpi.org>> On Behalf Of Ralph Castain via users Sent: Montag, 6. April 2020 16:32 To: Open MPI Users <users@lists.open-mpi.org<mailto:users@lists.open-mpi.org>> Cc: Ralph Castain <r...@open-mpi.org<mailto:r...@open-mpi.org>> Subject: Re: [OMPI users] Clean termination after receiving multiple SIGINT Currently, mpirun takes that second SIGINT to mean "you seem to be stuck trying to cleanly abort - just die", which means mpirun exits immediately without doing any cleanup. The individual procs all commit suicide when they see their daemons go away, which is why you don't get zombies left behind...but it does mean that the vader files are left. The second SIGINT has to come within a 5 second window of the first one to trigger that immediate exit, so one solution would be for you to delay your passing of the SIGINT to mpirun for more than 5 seconds. Alternatively, you could just not pass the signal at all since mpirun already received it - is there some reason why you need to pass the signal down? Are you trying to do your cleanup _before_ mpirun does its? Guess I'm wondering: why not just trap the signal, do your cleanup, and then wait for mpirun to terminate? On Apr 6, 2020, at 6:39 AM, Kreutzer, Moritz via users <users@lists.open-mpi.org<mailto:users@lists.open-mpi.org>> wrote: Hi, We are invoking mpirun from within a script which installs some signal handlers. Now, if we abort an Open MPI run with CTRL+C, the system sends SIGINT to the entire process group. Hence, the mpirun process receives a SIGINT from the system with si_code=SI_KERNEL. Additionally, our own signal handler intercepts SIGINT, does some clean up, and sends the SIGINT further to the mpirun process with si_code=SI_USER. Consequently, mpirun receives 2x SIGINT. This leads to unclean termination with Open MPI 4.0.3. While it does not leave behind any zombie processes, killing it in the described way leads to leftover vader shared memory segment files in /dev/shm (a known issue with Open MPI 3, but supposedly resolved in Open MPI 4). Also, strace shows that the mpirun process does not receive any SIGCHILD. If we remove our own signal handler (which is not our preferred option), mpirun receives only a single SIGINT and n times SIGCHILD (n is the number of processes). Also, this leads to correct clean up of vader shared memory segment files. Is it expected that the cleanup fails when mpirun receivs multiple signals at the same time? If yes, is the only way to guarantee proper clean up to always make sure that only a single signal gets propagated to mpirun? Thanks, Moritz -- Moritz Kreutzer Siemens Digital Industries Software Simulation and Test Solutions, Product Development, High Performance Computing Nordostpark 3 90411 Nuremberg, Germany Tel.: +49 (911) 38379 8085 moritz.kreut...@siemens.com<mailto:moritz.kreut...@siemens.com> www.sw.siemens.com<http://www.sw.siemens.com/> ----------------- Siemens Industry Software GmbH; Anschrift: Franz-Geuer-Str. 10, 50823 Köln; Gesellschaft mit beschränkter Haftung; Geschäftsführer: Dr. Erich Bürgel, Alexander Walter; Sitz der Gesellschaft: Köln; Registergericht: Amtsgericht Köln, HRB 84564; Vorsitzender des Aufsichtsrats: Jürgen Köhler ----------------- Siemens Industry Software GmbH; Anschrift: Franz-Geuer-Str. 10, 50823 Köln; Gesellschaft mit beschränkter Haftung; Geschäftsführer: Dr. Erich Bürgel, Alexander Walter; Sitz der Gesellschaft: Köln; Registergericht: Amtsgericht Köln, HRB 84564; Vorsitzender des Aufsichtsrats: Jürgen Köhler ----------------- Siemens Industry Software GmbH; Anschrift: Franz-Geuer-Str. 10, 50823 Köln; Gesellschaft mit beschränkter Haftung; Geschäftsführer: Dr. Erich Bürgel, Alexander Walter; Sitz der Gesellschaft: Köln; Registergericht: Amtsgericht Köln, HRB 84564; Vorsitzender des Aufsichtsrats: Jürgen Köhler
