[issue16500] Add an 'afterfork' module
Changes by Graham Dumpleton graham.dumple...@gmail.com: -- nosy: +grahamd ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16500 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16500] Add an 'afterfork' module
Changes by Jesús Cea Avión j...@jcea.es: -- nosy: +jcea ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16500 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
Re: [issue16500] Add an 'afterfork' module
2012/11/20 Christian Heimes rep...@bugs.python.org IFF we are going to walk the hard and rocky road of exception handling, then we are going to need at least four hooks and a register function that takres four callables as arguments: register(prepare, error, parent, child). Each prepare() call pushes an error handling onto a stack. In case of an exception in a prepare handler, the error stack is popped until all error handlers are called. This approach allows a prepare handler to actually prevent a fork() call from succeeding. FWIW, PyPy already has a notion of fork hooks: https://bitbucket.org/pypy/pypy/src/b4e4017909bac6c102fbc883ac8d2e42fa41553b/pypy/module/posix/interp_posix.py?at=default#cl-682 Various subsystems (threads cleanup, import lock, threading.local...) register their hook functions. You may want to experiment from there :-) A new atfork module would be easy to implement. -- Amaury Forgeot d'Arc ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16500] Add an 'afterfork' module
Changes by Andrew Svetlov andrew.svet...@gmail.com: -- nosy: +asvetlov ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16500 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16500] Add an 'afterfork' module
Changes by Amaury Forgeot d'Arc amaur...@gmail.com: -- nosy: +amaury.forgeotdarc -Amaury.Forgeot.d'Arc ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16500 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16500] Add an 'afterfork' module
Richard Oudkerk added the comment: IFF we are going to walk the hard and rocky road of exception handling, then we are going to need at least four hooks and a register function that takres four callables as arguments: register(prepare, error, parent, child). Each prepare() call pushes an error handling onto a stack. In case of an exception in a prepare handler, the error stack is popped until all error handlers are called. This approach allows a prepare handler to actually prevent a fork() call from succeeding. I think there are two main options if a prepare callback fails: 1) The fork should not occur and the exception should be raised 2) The fork should occur and the exception should be only be printed I favour option 1 since, if they want, users can always wrap their prepare callbacks with try: ... except: sys.excepthook(*sys.exc_info()) With option 1 I don't see why error callbacks are necessary. Just unwind the stack of imaginary try...finally... clauses and let any exceptions propagate out using exception chaining if necessary. This is what pure-python-atfork.patch does. Note, however, that if the fork succeeds then any subsequent exception is only printed. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16500 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16500] Add an 'afterfork' module
Christian Heimes added the comment: Amaury: PyPy doesn't handle exceptions in hooks. Is there a reason why PyPy goes for the simplistic approach? Richard: An error callback has the benefit that the API can notice the hooks that some error has occurred. We may not need it, though. I can think of six exception scenarios that must be handled: (1) exception in a prepare hook - don't call the remaining prepare hooks, run all related parent hooks in FILO order, prevent fork() call (2) exception in parent hook during the handling of (1) - print exception, continue with next parent hook (3) exception in fork() call - run parent hooks in FILO order (4) exception in parent hook during the handling of (3) - print exception, continue with next parent hook (5) exception in parent hook when fork() has succeeded - print exception, continue with next parent hook (6) exception in child hook when fork() has succeeded - print exception, continue with next child hook Do you agree? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16500 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16500] Add an 'afterfork' module
Amaury Forgeot d'Arc added the comment: PyPy doesn't handle exceptions in hooks. Is there a reason why PyPy goes for the simplistic approach? Probably because nobody thought about it. At the moment, there is only one 'before', one 'parent' hook (so the FILO order is simple), and three 'child' hooks. And if the _PyImport_ReleaseLock call fails, you'd better not ignore the error... -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16500 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16500] Add an 'afterfork' module
Gregory P. Smith added the comment: I think you are solving a non-problem if you want to expose exceptions from such hooks. Nobody needs it. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16500 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16500] Add an 'afterfork' module
Antoine Pitrou added the comment: I think you are solving a non-problem if you want to expose exceptions from such hooks. Nobody needs it. Agreed. -- nosy: +pitrou ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16500 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16500] Add an 'afterfork' module
Christian Heimes added the comment: Your suggestion is that the hooks are called as: for hook in hooks: try: hook() except: try: sys.excepthook(*sys.exc_info()) except: pass That makes the implementation much easier. :) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16500 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16500] Add an 'afterfork' module
Christian Heimes added the comment: Thanks Richard! My first reaction was YAGNI but after I read the two tickets I now understand the need for three different hooks. I suggest that we implement our own hooks like the http://linux.die.net/man/3/pthread_atfork function, especially the order of function calls: The parent and child fork handlers shall be called in the order in which they were established by calls to pthread_atfork(). The prepare fork handlers shall be called in the opposite order. I like to focus on three hooks + the Python API and leave the usage of the hooks to other developers. Proposal: * Introduce a new module called atfork (Modules/atforkmodule.c) that is build into the core. * Move PyOS_AfterFork to Modules/atforkmodule.c. * Add PyOS_BeforeFork() (or PyOS_PrepareFork() ?) and PyOS_AfterForkParent() * call the two new methods around the calls to fork() in the stdlib. I'm not yet sure how to implement the Python API. I could either implement six methods: atfork.register_before_fork(callable, *args, **kwargs) atfork.register_after_fork_child(callable, *args, **kwargs) atfork.register_after_fork_parent(callable, *args, **kwargs) atfork.unregister_before_fork(callable) atfork.unregister_after_fork_child(callable) atfork.unregister_after_fork_parent(callable) or two: atfork.register(prepare=None, parent=None, child=None, *args, **kwargs) atfork.unregister(prepare=None, parent=None, child=None) -- nosy: +gregory.p.smith, twouters ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16500 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16500] Add an 'afterfork' module
Richard Oudkerk added the comment: Note that Gregory P. Smith has written http://code.google.com/p/python-atfork/ I also started a pure python patch but did not get round it posting it. (It also implements the fork lock idea.) I'll attach it here. How do you intend to handle the propagation of exceptions? I decided that after atfork.atfork(prepare1, parent1, child1) atfork.atfork(prepare2, parent2, child2) ... atfork.atfork(prepareN, parentN, childN) calling pid = os.fork() should be equivalent to pid = None prepareN() try: ... prepare2() try: prepare1() try: pid = posix.fork() finally: parent1() if pid != 0 else child1() finally: parent2() if pid != 0 else child2() ... finally: parentN() if pid != 0 else childN() -- keywords: +patch Added file: http://bugs.python.org/file28044/pure-python-atfork.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16500 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16500] Add an 'afterfork' module
Gregory P. Smith added the comment: I would not allow exceptions to propagate. No caller is expecting them. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16500 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16500] Add an 'afterfork' module
Gregory P. Smith added the comment: pthread_atfork() cannot be used to implement this. Another non-python thread started by a C extension module or the C application that is embedding Python within it is always free to call fork() on its own with zero knowledge that Python even exists at all. It's guaranteed that fork will be called while the Python GIL is held in this situation which would cause any pre-fork thing registered by Python to deadlock. At best, this can be implemented manually as we do with some of the before and after fork stuff today but it must come with the caveat warning that it cannot guarantee that these things are actually called before and after fork() other than direct os.fork() calls from Python code or extremely Python aware C extension modules that may call fork() (very rare, most C C++ libraries an extension module may be using assume that they've got the run of the house). ie: this problem is unsolvable unless you control 100% of the code being used by your entire user application. On Mon, Nov 19, 2012 at 3:59 PM, Gregory P. Smith rep...@bugs.python.orgwrote: Gregory P. Smith added the comment: I would not allow exceptions to propagate. No caller is expecting them. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16500 ___ -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16500 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16500] Add an 'afterfork' module
Christian Heimes added the comment: Meh! Exception handling takes all the fun of the API and is going to make it MUCH more complicated. pthread_atfork() ignores error handling for a good reason. It's going to be hard to get it right. :/ IFF we are going to walk the hard and rocky road of exception handling, then we are going to need at least four hooks and a register function that takres four callables as arguments: register(prepare, error, parent, child). Each prepare() call pushes an error handling onto a stack. In case of an exception in a prepare handler, the error stack is popped until all error handlers are called. This approach allows a prepare handler to actually prevent a fork() call from succeeding. The parent and child hooks are always called no matter what. Exception are recorded and a warning is emitted when at least one hook fails. We might raise an exception but it has to be a special exception that ships information if fork() has succeeded, if the code runs in child or parent and about the child's PID. I fear it's going to be *really* hard to get everything right. Gregory made a good point, too. We can rely on pthread_atfork() as we are unable to predict how third party code is using fork(): Take cover, dead locks ahead! :) A cooperative design of the C API with three function is my preferred way, too. PyOS_AfterForkParent() should take an argument to signal a failed fork() call. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16500 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16500] Add an 'afterfork' module
New submission from Christian Heimes: I propose the addition of an 'afterfork' module. The module shall fulfill a similar task as the 'atexit' module except that it handles process forks instead of process shutdown. The 'afterfork' module shall allow libraries to register callbacks that are executed on fork() inside the child process and as soon as possible. Python already has a function that must be called by C code: PyOS_AfterFork(). The 'afterfork' callbacks are called as the last step in PyOS_AfterFork(). Use case example: The tempfile module has a specialized RNG that re-initialized the RNG after fork() by comparing os.getpid() to an instance variable every time the RNG is accessed. The check can be replaced with an afterfork callback. Open questions: How should the afterfork() module handle exceptions that are raised by callbacks? Implementation: I'm going to use as much code from atexitmodule.c as possible. I'm going to copy common code to a template file and include the template from atexitmodule.c and afterforkmodule.c with some preprocessor tricks. -- assignee: christian.heimes components: Extension Modules, Interpreter Core keywords: needs review messages: 175878 nosy: christian.heimes priority: normal severity: normal status: open title: Add an 'afterfork' module type: enhancement versions: Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16500 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16500] Add an 'afterfork' module
Changes by Antoine Pitrou pit...@free.fr: -- nosy: +sbt ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16500 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16500] Add an 'afterfork' module
Richard Oudkerk added the comment: pthread_atfork() allows the registering of three types of callbacks: 1) prepare callbacks which are called before the fork, 2) parent callbacks which are called in the parent after the fork 3) child callbacks which are called in the child after the fork. I think all three should be supported. I also think that a recursive fork lock should be introduced which is held during the fork. This can be acquired around critical sections during which forks must not occur. This is more or less a duplicate of #6923. See also #6721. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16500 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com