[issue18999] Robustness issues in multiprocessing.{get, set}_start_method
Roundup Robot added the comment: New changeset 72a5ac909c7a by Richard Oudkerk in branch 'default': Issue #18999: Make multiprocessing use context objects. http://hg.python.org/cpython/rev/72a5ac909c7a -- nosy: +python-dev ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18999 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18999] Robustness issues in multiprocessing.{get, set}_start_method
Lars Buitinck added the comment: BTW, the context objects are singletons. I haven't read all of your patch yet, but does this mean a forkserver will be started regardless of whether it is later used? That would be a good thing, since starting the fork server after reading in large data sets would mean the fork server would hold on to large swaths of memory even when the data set is deallocated in the master process. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18999 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18999] Robustness issues in multiprocessing.{get, set}_start_method
Richard Oudkerk added the comment: I haven't read all of your patch yet, but does this mean a forkserver will be started regardless of whether it is later used? No, it is started on demand. But since it is started using _posixsbuprocess.fork_exec(), nothing is inherited from the main process. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18999 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18999] Robustness issues in multiprocessing.{get, set}_start_method
Lars Buitinck added the comment: Ok, great. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18999 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18999] Robustness issues in multiprocessing.{get, set}_start_method
Richard Oudkerk added the comment: BTW, the context objects are singletons. I could not see a sensible way to make ctx.Process be a picklable class (rather than a method) if there can be multiple instances of a context type. This means that the helper processes survive until the program closes down. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18999 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18999] Robustness issues in multiprocessing.{get, set}_start_method
Richard Oudkerk added the comment:
Attached is a patch which allows the use of separate contexts. For example
try:
ctx = multiprocessing.get_context('forkserver')
except ValueError:
ctx = multiprocessing.get_context('spawn')
q = ctx.Queue()
p = ctx.Process(target=foo, args=(q,))
p.start()
...
Also, get_start_method(allow_none=True) will return None if the start method
has not yet been fixed.
--
Added file: http://bugs.python.org/file32034/context.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18999
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18999] Robustness issues in multiprocessing.{get, set}_start_method
Lars Buitinck added the comment: Ok. Do you (or jnoller?) have time to review my proposed patch, at least before 3.4 is released? I didn't see it in the release schedule, so it's probably not planned soon, but I wouldn't want the API to change *again* in 3.5. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18999 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18999] Robustness issues in multiprocessing.{get, set}_start_method
Richard Oudkerk added the comment: I'll review the patch. (According to http://www.python.org/dev/peps/pep-0429/ feature freeze is expected in late November, so there is not too much of rush.) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18999 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18999] Robustness issues in multiprocessing.{get, set}_start_method
Changes by Lars Buitinck larsm...@gmail.com: -- nosy: +jnoller ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18999 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18999] Robustness issues in multiprocessing.{get, set}_start_method
Lars Buitinck added the comment: I don't really see the benefit of a context manager over an argument. It's a power user feature anyway, and context managers (at least to me) signal cleanup actions, rather than construction options. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18999 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18999] Robustness issues in multiprocessing.{get, set}_start_method
Richard Oudkerk added the comment: By context I did not really mean a context manager. I just meant an object (possibly a singleton or module) which implements the same interface as multiprocessing. (However, it may be a good idea to also make it a context manager whose __enter__() method starts the helper processes, and whose __exit__() method shuts them down.) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18999 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18999] Robustness issues in multiprocessing.{get, set}_start_method
Olivier Grisel added the comment: The process pool executor [1] from the concurrent futures API would be suitable to explicitly start and stop the helper process for the `forkserver` mode. [1] http://docs.python.org/3.4/library/concurrent.futures.html#concurrent.futures.ProcessPoolExecutor The point would be to have as few state as possible encoded in the multiprocessing module (and its singletons) and move that state information to be directly managed by multiprocessing Process and Pool class instances so that libraries could customize the behavior (start_method, executable, ForkingPIckler reducers registry and so on) without mutating the state of the multiprocessing module singletons. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18999 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18999] Robustness issues in multiprocessing.{get, set}_start_method
Olivier Grisel added the comment: Richard Oudkerk: thanks for the clarification, that makes sense. I don't have the time either in the coming month, maybe later. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18999 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18999] Robustness issues in multiprocessing.{get, set}_start_method
Richard Oudkerk added the comment: There are lots of things that behave differently depending on the currently set start method: Lock(), Semaphore(), Queue(), Value(), ... It is not just when creating a Process or Pool that you need to know the start method. Passing a context or start_method argument to all of these constructors would be very awkward, which is why I think it is better to treat the context as an object with methods Process(), Pool(), Lock(), Semaphore(), etc. Unfortunately, I do not have time to work on this just now... -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18999 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18999] Robustness issues in multiprocessing.{get, set}_start_method
Olivier Grisel added the comment: Maybe it would be better to have separate contexts for each start method. That way joblib could use the forkserver context without interfering with the rest of the user's program. Yes in general it would be great if libraries could customize the multiprocessing default options without impacting any of the module singletons. That also include the ForkingPickler registry for custom: now it's a class attribute. It would be great to be able to scope custom reducers registration to a given multiprocessing.Pool or multiprocessing.Process instance. Now how to implement that kind of isolation: it could either be done by adding new constructor parameters or new public methods to the Process and Pool classes to be able to customize their behavior while sticking to the OOP paradigm if possible or by using a context manager as you suggested. I am not sure which option is best. Prototyping both is probably the best way to feel the tradeoffs. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18999 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18999] Robustness issues in multiprocessing.{get, set}_start_method
Changes by Lars Buitinck larsm...@gmail.com:
--
title: Allow multiple calls to multiprocessing.set_start_method - Robustness
issues in multiprocessing.{get,set}_start_method
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18999
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18999] Robustness issues in multiprocessing.{get, set}_start_method
Olivier Grisel added the comment: Related question: is there any good reason that would prevent to pass a custom `start_method` kwarg to the `Pool` constructor to make it use an alternative `Popen` instance (that is an instance different from the `multiprocessing._Popen` singleton)? This would allow libraries such as joblib to keep minimal side effect by try to impact the default multiprocessing runtime as low as possible. -- nosy: +Olivier.Grisel ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18999 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18999] Robustness issues in multiprocessing.{get, set}_start_method
Richard Oudkerk added the comment: With your patch, I think if you call get_start_method() without later calling set_start_method() then the helper process(es) will never be started. With the current code, popen.Popen() automatically starts the helper processes if they have not already been started. Also, set_start_method() can have the side-effect of starting helper process(es). I do not really approve of new processes being started as a side-effect of importing a library. But it is reasonable for a library to want a specific start method unless the user demands otherwise. I will have to think this over. BTW, the reason for discouraging using set_start_method() more than once is because some shared resources are created differently depending on what the current start method is. For instance using the fork method semaphores are created and then immediately unlinked. But with the other start methods we must not unlink the semaphore until we are finished with it (while being paranoid about cleanup). Maybe it would be better to have separate contexts for each start method. That way joblib could use the forkserver context without interfering with the rest of the user's program. from multiprocessing import forkserver_context as mp l = mp.Lock() p = mp.Process(...) with mp.Pool() as pool: ... -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18999 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18999] Robustness issues in multiprocessing.{get, set}_start_method
Changes by Lars Buitinck larsm...@gmail.com: Removed file: http://bugs.python.org/file31721/mp_getset_start_method.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18999 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18999] Robustness issues in multiprocessing.{get, set}_start_method
Lars Buitinck added the comment: Cleaned up the patch. -- Added file: http://bugs.python.org/file31722/mp_getset_start_method.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18999 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18999] Robustness issues in multiprocessing.{get, set}_start_method
Lars Buitinck added the comment: In my patched version, the private popen.get_start_method gets a kwarg set_if_needed=True. popen.Popen calls that as before, so its behavior should not change, while the public get_start_method sets the kwarg to False. I realise now that this has the side effect that get_start_method's output changes when multiprocessing has first been used, but then that reflects how the library works. Maybe this should be documented. As for the contexts, those would be great. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18999 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18999] Robustness issues in multiprocessing.{get, set}_start_method
Richard Oudkerk added the comment: In my patched version, the private popen.get_start_method gets a kwarg set_if_needed=True. popen.Popen calls that as before, so its behavior should not change, while the public get_start_method sets the kwarg to False. My mistake. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18999 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
