Re: Multiprocessing, join(), and crashed processes

2020-02-05 Thread Cameron Simpson

On 05Feb2020 15:48, Israel Brewster  wrote:
In a number of places I have constructs where I launch several 
processes using the multiprocessing library, then loop through said 
processes calling join() on each one to wait until they are all 
complete. In general, this works well, with the *apparent* exception of 
if something causes one of the child processes to crash (not throw an 
exception, actually crash). In that event, it appears that the call to 
join() hangs indefinitely. How can I best handle this? Should I put a 
timeout on the join, and put it in a loop, such that every 5 seconds or 
so it breaks, checks to see if the process is still actually running, 
and if so goes back and calls join again? Or is there a better option 
to say “wait until this process is done, however long that may be, 
unless it crashes”?


What's your platform/OS? And what does "crash" mean, precisely?

If a subprocess exits, join() should terminate.

If the subprocess _hangs_, then join will not see it exit, because it 
hasn't. And join will hang.


You'll need to define what happens when your subprocesses crash.

Cheers,
Cameron Simpson 
--
https://mail.python.org/mailman/listinfo/python-list


Multiprocessing, join(), and crashed processes

2020-02-05 Thread Israel Brewster
In a number of places I have constructs where I launch several processes using 
the multiprocessing library, then loop through said processes calling join() on 
each one to wait until they are all complete. In general, this works well, with 
the *apparent* exception of if something causes one of the child processes to 
crash (not throw an exception, actually crash). In that event, it appears that 
the call to join() hangs indefinitely. How can I best handle this? Should I put 
a timeout on the join, and put it in a loop, such that every 5 seconds or so it 
breaks, checks to see if the process is still actually running, and if so goes 
back and calls join again? Or is there a better option to say “wait until this 
process is done, however long that may be, unless it crashes”?
---
Israel Brewster
Software Engineer
Alaska Volcano Observatory 
Geophysical Institute - UAF 
2156 Koyukuk Drive 
Fairbanks AK 99775-7320
Work: 907-474-5172
cell:  907-328-9145

-- 
https://mail.python.org/mailman/listinfo/python-list