[issue31308] forkserver process isn't re-launched if it died

[Antoine Pitrou]

Change by Antoine Pitrou :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31308] forkserver process isn't re-launched if it died

[Antoine Pitrou]

Antoine Pitrou  added the comment:


New changeset 019c99f325287741d1e0eefeef2b75c8e00b884f by Antoine Pitrou in 
branch '3.6':
[3.6] bpo-31308: If multiprocessing's forkserver dies, launch it again when 
necessary (GH-3246) (#4252)
https://github.com/python/cpython/commit/019c99f325287741d1e0eefeef2b75c8e00b884f


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31308] forkserver process isn't re-launched if it died

[Antoine Pitrou]

Change by Antoine Pitrou :


--
keywords: +patch
pull_requests: +4214

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31308] forkserver process isn't re-launched if it died

[Antoine Pitrou]

Antoine Pitrou  added the comment:


New changeset fc6b348b12ad401cab0261b7b71a65c60a08c0a8 by Antoine Pitrou in 
branch 'master':
bpo-31308: If multiprocessing's forkserver dies, launch it again when necessary 
(#3246)
https://github.com/python/cpython/commit/fc6b348b12ad401cab0261b7b71a65c60a08c0a8


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31308] forkserver process isn't re-launched if it died

[Antoine Pitrou]

Antoine Pitrou  added the comment:

Davin, I think I'm going to merge the PR for this.  If you object it, it can 
still be reverted later.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31308] forkserver process isn't re-launched if it died

[Matthew Rocklin]

Matthew Rocklin  added the comment:

>From a user's perspective I would definitely appreciate forkserver being 
>protected from SIGINT.  This bug affects me in practice.  If the forkserver 
>goes down I personally have no objection to it restarting automatically, 
>though I appreciate that I have a narrow view of this topic.

Davin, at your last comment it seemed like you had reservations about this 
going in.  Did Antoine's recent comments resolve these concerns or no?  Do you 
have any suggestions on what might be done to protect users from SIGINT 
crashing the forkserver?

--
nosy: +Matthew Rocklin

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31308] forkserver process isn't re-launched if it died

[Antoine Pitrou]

Antoine Pitrou  added the comment:

Looks like this will not make into 3.6.3.

I'd like to sum up the discussion a bit:
- I think at the very least least we must protect the forkserver against 
SIGINT, like the semaphore tracker already is
- I think it's beneficial to also restart it on demand if necessary
- I can't think of a use case where the user asks "no, please don't restart the 
forkserver on demand" as there is no way to start it manually anyway, and not 
restarting it breaks functionality

Also, bpo-31310 is a similar issue for the semaphore tracker.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31308] forkserver process isn't re-launched if it died

[Antoine Pitrou]

Antoine Pitrou added the comment:

> 1) The implicit restart of the forkserver process seems in conflict with the 
> zen of making things explicit.

Well, the forkserver process really is an implementation detail.  It already is 
started on-demand, so restarting it if needed sounded like a natural evolution.

Another example: in a multiprocessing Pool, if a worker process crashes, 
another will be restarted transparently (same in 
concurrent.futures.ProcessPoolExecutor, incidentally).

> 2) This would seem to make forkserver's behavior inconsistent with the 
> behavior of things like the Manager which similarly creates its own process 
> for managing resources but does not automatically restart that process if it 
> should die or become unreachable.

That's possible.  I've never seen the Manager used in the wild.  Though if some 
fundamental difference justifies the inconsistency, the inconsistency is not 
really a problem.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31308] forkserver process isn't re-launched if it died

[Davin Potts]

Davin Potts added the comment:

I have two concerns with this:
1) The implicit restart of the forkserver process seems in conflict with the 
zen of making things explicit.
2) This would seem to make forkserver's behavior inconsistent with the behavior 
of things like the Manager which similarly creates its own process for managing 
resources but does not automatically restart that process if it should die or 
become unreachable.  In the case of the Manager, I don't think we'd want it to 
automagically restart anything in these situations so it's not a simple matter 
of enhancing the Manager to adopt similar behavior.

I do appreciate the use cases that would be addressed by having a convenient 
way to detect that a forkserver has died and then restart it.  If the 
forkserver dies, I doubt we really want it to try to restart a potentially 
infinite number of times.

Maybe a better path would be if we had a way to explicitly request that the 
Process trigger a restart of the forkserver, if necessary, but this 
setting/request defaults to False?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31308] forkserver process isn't re-launched if it died

[Antoine Pitrou]

Changes by Antoine Pitrou :


--
stage: needs patch -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31308] forkserver process isn't re-launched if it died

[Antoine Pitrou]

Changes by Antoine Pitrou :


--
pull_requests: +3291

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31308] forkserver process isn't re-launched if it died

[Antoine Pitrou]

Changes by Antoine Pitrou :


--
nosy: +davin

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31308] forkserver process isn't re-launched if it died

[Antoine Pitrou]

New submission from Antoine Pitrou:

It may happen that the forkserver process dies (for example if SIGINT is 
received because the user pressed Ctrl-C) while the parent process is still 
alive.  In that case, if the parent tries to create a new Process instance, an 
exception is received.

The exception looks like this:

  File "/xxx/lib/python3.5/multiprocessing/popen_forkserver.py", line 52, in 
_launch
self.sentinel, w = forkserver.connect_to_new_process(self._fds)
  File "/xxx/lib/python3.5/multiprocessing/forkserver.py", line 66, in 
connect_to_new_process
client.connect(self._forkserver_address)
ConnectionRefusedError: [Errno 111] Connection refused

--
components: Library (Lib)
messages: 301027
nosy: pitrou
priority: normal
severity: normal
stage: needs patch
status: open
title: forkserver process isn't re-launched if it died
type: behavior
versions: Python 3.6, Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com