[issue16500] Add an 'afterfork' module

2012-11-26 Thread Graham Dumpleton

Changes by Graham Dumpleton graham.dumple...@gmail.com:


--
nosy: +grahamd

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16500
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue16500] Add an 'afterfork' module

2012-11-23 Thread Jesús Cea Avión

Changes by Jesús Cea Avión j...@jcea.es:


--
nosy: +jcea

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16500
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



Re: [issue16500] Add an 'afterfork' module

2012-11-20 Thread Amaury Forgeot d'Arc
2012/11/20 Christian Heimes rep...@bugs.python.org

 IFF we are going to walk the hard and rocky road of exception handling,
 then we are going to need at least four hooks and a register function that
 takres four callables as arguments: register(prepare, error, parent,
 child). Each prepare() call pushes an error handling onto a stack. In case
 of an exception in a prepare handler, the error stack is popped until all
 error handlers are called. This approach allows a prepare handler to
 actually prevent a fork() call from succeeding.


FWIW, PyPy already has a notion of fork hooks:
https://bitbucket.org/pypy/pypy/src/b4e4017909bac6c102fbc883ac8d2e42fa41553b/pypy/module/posix/interp_posix.py?at=default#cl-682

Various subsystems (threads cleanup, import lock, threading.local...)
register their hook functions.

You may want to experiment from there :-)
A new atfork module would be easy to implement.

-- 
Amaury Forgeot d'Arc
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue16500] Add an 'afterfork' module

2012-11-20 Thread Andrew Svetlov

Changes by Andrew Svetlov andrew.svet...@gmail.com:


--
nosy: +asvetlov

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16500
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue16500] Add an 'afterfork' module

2012-11-20 Thread Amaury Forgeot d'Arc

Changes by Amaury Forgeot d'Arc amaur...@gmail.com:


--
nosy: +amaury.forgeotdarc -Amaury.Forgeot.d'Arc

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16500
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue16500] Add an 'afterfork' module

2012-11-20 Thread Richard Oudkerk

Richard Oudkerk added the comment:

 IFF we are going to walk the hard and rocky road of exception handling,
 then we are going to need at least four hooks and a register function that
 takres four callables as arguments: register(prepare, error, parent,
 child). Each prepare() call pushes an error handling onto a stack. In case
 of an exception in a prepare handler, the error stack is popped until all
 error handlers are called. This approach allows a prepare handler to
 actually prevent a fork() call from succeeding.

I think there are two main options if a prepare callback fails:

1) The fork should not occur and the exception should be raised
2) The fork should occur and the exception should be only be printed

I favour option 1 since, if they want, users can always wrap their prepare 
callbacks with

  try:
...
  except:
sys.excepthook(*sys.exc_info())

With option 1 I don't see why error callbacks are necessary.  Just unwind the 
stack of imaginary try...finally... clauses and let any exceptions propagate 
out using exception chaining if necessary.  This is what 
pure-python-atfork.patch does.  Note, however, that if the fork succeeds then 
any subsequent exception is only printed.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16500
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue16500] Add an 'afterfork' module

2012-11-20 Thread Christian Heimes

Christian Heimes added the comment:

Amaury:
PyPy doesn't handle exceptions in hooks. Is there a reason why PyPy goes for 
the simplistic approach?

Richard:
An error callback has the benefit that the API can notice the hooks that some 
error has occurred. We may not need it, though.

I can think of six exception scenarios that must be handled:

(1) exception in a prepare hook - don't call the remaining prepare hooks, run 
all related parent hooks in FILO order, prevent fork() call
(2) exception in parent hook during the handling of (1) - print exception, 
continue with next parent hook
(3) exception in fork() call - run parent hooks in FILO order
(4) exception in parent hook during the handling of (3) - print exception, 
continue with next parent hook
(5) exception in parent hook when fork() has succeeded - print exception, 
continue with next parent hook
(6) exception in child hook when fork() has succeeded  - print exception, 
continue with next child hook

Do you agree?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16500
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue16500] Add an 'afterfork' module

2012-11-20 Thread Amaury Forgeot d'Arc

Amaury Forgeot d'Arc added the comment:

 PyPy doesn't handle exceptions in hooks.
 Is there a reason why PyPy goes for the simplistic approach?

Probably because nobody thought about it.
At the moment, there is only one 'before', one 'parent' hook (so the FILO order 
is simple), and three 'child' hooks.
And if the _PyImport_ReleaseLock call fails, you'd better not ignore the 
error...

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16500
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue16500] Add an 'afterfork' module

2012-11-20 Thread Gregory P. Smith

Gregory P. Smith added the comment:

I think you are solving a non-problem if you want to expose exceptions from
such hooks. Nobody needs it.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16500
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue16500] Add an 'afterfork' module

2012-11-20 Thread Antoine Pitrou

Antoine Pitrou added the comment:

 I think you are solving a non-problem if you want to expose exceptions from
 such hooks. Nobody needs it.

Agreed.

--
nosy: +pitrou

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16500
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com




[issue16500] Add an 'afterfork' module

2012-11-20 Thread Christian Heimes

Christian Heimes added the comment:

Your suggestion is that the hooks are called as:

for hook in hooks:
try:
hook()
except:
try:
sys.excepthook(*sys.exc_info())
except:
pass

That makes the implementation much easier. :)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16500
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue16500] Add an 'afterfork' module

2012-11-19 Thread Christian Heimes

Christian Heimes added the comment:

Thanks Richard!

My first reaction was YAGNI but after I read the two tickets I now understand 
the need for three different hooks. I suggest that we implement our own hooks 
like the http://linux.die.net/man/3/pthread_atfork function, especially the 
order of function calls:

The parent and child fork handlers shall be called in the order in which they 
were established by calls to pthread_atfork().  The prepare fork handlers shall 
be called in the opposite order.

I like to focus on three hooks + the Python API and leave the usage of the 
hooks to other developers.

Proposal:
* Introduce a new module called atfork (Modules/atforkmodule.c) that is build 
into the core.
* Move PyOS_AfterFork to Modules/atforkmodule.c.
* Add PyOS_BeforeFork() (or PyOS_PrepareFork() ?) and PyOS_AfterForkParent() 
* call the two new methods around the calls to fork() in the stdlib.

I'm not yet sure how to implement the Python API. I could either implement six 
methods:

  atfork.register_before_fork(callable, *args, **kwargs)
  atfork.register_after_fork_child(callable, *args, **kwargs)
  atfork.register_after_fork_parent(callable, *args, **kwargs)
  atfork.unregister_before_fork(callable)
  atfork.unregister_after_fork_child(callable)
  atfork.unregister_after_fork_parent(callable)

or two:

  atfork.register(prepare=None, parent=None, child=None, *args, **kwargs)
  atfork.unregister(prepare=None, parent=None, child=None)

--
nosy: +gregory.p.smith, twouters

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16500
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue16500] Add an 'afterfork' module

2012-11-19 Thread Richard Oudkerk

Richard Oudkerk added the comment:

Note that Gregory P. Smith has written

http://code.google.com/p/python-atfork/

I also started a pure python patch but did not get round it posting it.  (It 
also implements the fork lock idea.)  I'll attach it here.

How do you intend to handle the propagation of exceptions?  I decided that after

atfork.atfork(prepare1, parent1, child1)
atfork.atfork(prepare2, parent2, child2)
...
atfork.atfork(prepareN, parentN, childN)

calling pid = os.fork() should be equivalent to

pid = None
prepareN()
try:
...
prepare2()
try:
prepare1()
try:
pid = posix.fork()
finally:
parent1() if pid != 0 else child1()
finally:
parent2() if pid != 0 else child2()
...
finally:
parentN() if pid != 0 else childN()

--
keywords: +patch
Added file: http://bugs.python.org/file28044/pure-python-atfork.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16500
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue16500] Add an 'afterfork' module

2012-11-19 Thread Gregory P. Smith

Gregory P. Smith added the comment:

I would not allow exceptions to propagate. No caller is expecting them.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16500
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue16500] Add an 'afterfork' module

2012-11-19 Thread Gregory P. Smith

Gregory P. Smith added the comment:

pthread_atfork() cannot be used to implement this. Another non-python
thread started by a C extension module or the C application that is
embedding Python within it is always free to call fork() on its own with
zero knowledge that Python even exists at all. It's guaranteed that fork
will be called while the Python GIL is held in this situation which would
cause any pre-fork thing registered by Python to deadlock.

At best, this can be implemented manually as we do with some of the before
and after fork stuff today but it must come with the caveat warning that it
cannot guarantee that these things are actually called before and after
fork() other than direct os.fork() calls from Python code or extremely
Python aware C extension modules that may call fork() (very rare, most C 
C++ libraries an extension module may be using assume that they've got the
run of the house).  ie: this problem is unsolvable unless you control 100%
of the code being used by your entire user application.

On Mon, Nov 19, 2012 at 3:59 PM, Gregory P. Smith rep...@bugs.python.orgwrote:


 Gregory P. Smith added the comment:

 I would not allow exceptions to propagate. No caller is expecting them.

 --

 ___
 Python tracker rep...@bugs.python.org
 http://bugs.python.org/issue16500
 ___


--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16500
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue16500] Add an 'afterfork' module

2012-11-19 Thread Christian Heimes

Christian Heimes added the comment:

Meh! Exception handling takes all the fun of the API and is going to make it 
MUCH more complicated. pthread_atfork() ignores error handling for a good 
reason. It's going to be hard to get it right. :/

IFF we are going to walk the hard and rocky road of exception handling, then we 
are going to need at least four hooks and a register function that takres four 
callables as arguments: register(prepare, error, parent, child). Each prepare() 
call pushes an error handling onto a stack. In case of an exception in a 
prepare handler, the error stack is popped until all error handlers are called. 
This approach allows a prepare handler to actually prevent a fork() call from 
succeeding.

The parent and child hooks are always called no matter what. Exception are 
recorded and a warning is emitted when at least one hook fails. We might raise 
an exception but it has to be a special exception that ships information if 
fork() has succeeded, if the code runs in child or parent and about the child's 
PID.

I fear it's going to be *really* hard to get everything right.

Gregory made a good point, too. We can rely on pthread_atfork() as we are 
unable to predict how third party code is using fork(): Take cover, dead locks 
ahead! :) A cooperative design of the C API with three function is my 
preferred way, too. PyOS_AfterForkParent() should take an argument to signal a 
failed fork() call.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16500
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue16500] Add an 'afterfork' module

2012-11-18 Thread Christian Heimes

New submission from Christian Heimes:

I propose the addition of an 'afterfork' module. The module shall fulfill a 
similar task as the 'atexit' module except that it handles process forks 
instead of process shutdown.

The 'afterfork' module shall allow libraries to register callbacks that are 
executed on fork() inside the child process and as soon as possible. Python 
already has a function that must be called by C code: PyOS_AfterFork(). The 
'afterfork' callbacks are called as the last step in PyOS_AfterFork().

Use case example:
The tempfile module has a specialized RNG that re-initialized the RNG after 
fork() by comparing os.getpid() to an instance variable every time the RNG is 
accessed. The check can be replaced with an afterfork callback.

Open questions:
How should the afterfork() module handle exceptions that are raised by 
callbacks?

Implementation:
I'm going to use as much code from atexitmodule.c as possible. I'm going to 
copy common code to a template file and include the template from 
atexitmodule.c and afterforkmodule.c with some preprocessor tricks.

--
assignee: christian.heimes
components: Extension Modules, Interpreter Core
keywords: needs review
messages: 175878
nosy: christian.heimes
priority: normal
severity: normal
status: open
title: Add an 'afterfork' module
type: enhancement
versions: Python 3.4

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16500
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue16500] Add an 'afterfork' module

2012-11-18 Thread Antoine Pitrou

Changes by Antoine Pitrou pit...@free.fr:


--
nosy: +sbt

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16500
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue16500] Add an 'afterfork' module

2012-11-18 Thread Richard Oudkerk

Richard Oudkerk added the comment:

pthread_atfork() allows the registering of three types of callbacks:

1) prepare callbacks which are called before the fork,
2) parent callbacks which are called in the parent after the fork
3) child callbacks which are called in the child after the fork.

I think all three should be supported.

I also think that a recursive fork lock should be introduced which is held 
during the fork.  This can be acquired around critical sections during which 
forks must not occur.

This is more or less a duplicate of #6923.  See also #6721.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16500
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com