On 03/17/2011 09:27 AM, Stefan Behnel wrote:
Dag Sverre Seljebotn, 17.03.2011 08:38:
On 03/17/2011 12:24 AM, Greg Ewing wrote:
Stefan Behnel wrote:
I'm not sure if this is a good idea. "nogil" blocks don't have a
way to
handle exceptions, so simply jumping out of them because an inner
'with
gil' block raised an exception can have unexpected side effects.
Seems to me that the __Pyx_WriteUnraisable should be done at
the end of the 'with gil' block, and execution should then
continue from there.
In other words, the effect on exception handling should be
the same as if the 'with gil' block had been factored out into
a separate function having no exception return value.
-1.
I consider the fact that exceptions don't propagate from some
functions a
"currently unfixable bug". We should plan for it being fixed some day.
It can't be fixed in general, because there are cases where exceptions
simply cannot be propagated. Think of C callbacks, for example. C
doesn't have a "normal" way of dealing with exceptions, so if an
exception that originated from a callback simply leads to returning
from the function, it may mean that the outer C code will simply
continue to execute normally. Nothing's won in that case.
Yes, that's a good point. (This is what I used setjmp/longjmp to work
around BTW, to longjmp across the calling Fortran code. I knew it wasn't
doing any mallocs/frees, let alone any file handling etc., so this was
safe.)
I'll admit that I'm mostly focused on code like
def f():
with nogil:
for ...:
A
if something_exceptional:
with gil:
raise Exception(...)
B
C
where I'd say it's up to me to make sure that B and C can safely be
skipped. It would be a major pain to have my raised exception here be
"trapped" -- in fact, it would make the "with gil" statement unusable
for my purposes.
In code:
cdef void c_callback(...) nogil:
... do some C stuff ...
with gil:
... do some Python stuff ...
... do some more C stuff ...
So far, there are two proposed ways of doing this.
1) acquire the GIL on entry and exit, handling unraisable exceptions
right before exiting.
2) keep all GIL requiring code inside of the "with gil" block,
including unraisable exceptions.
I find (2) a *lot* more intuitive, as well as much safer. We can't
know what effects the surrounding "do C stuff" code has. It may
contain thread-safe C level cleanup code for the "with gil" block, for
example, or preparation code that enables returning into the calling C
code. Simply jumping out of the GIL block without executing the
trailing code may simply not work at all.
I think you find (2) more intuitive because you have a very detailed
knowledge of Cython and CPython, but that somebody new to Cython would
expect a "with" statement to have the same control flow logic as the
Python with statement. Of course, I don't have any data for that.
How about this compromise: We balk on the code you wrote with:
Error line 345: Exceptions propagating from "with gil" block cannot be
propagated out of function, please insert try/except and handle exception
So that we require this:
with gil:
try:
...
except:
warnings.warning(...) # or even cython.unraisable(e)
This keeps me happy about not abusing the with statement for strange
control flow, and makes the "with gil" useful for raising exceptions
inside regular def functions with nogil blocks.
We could perhaps fix exception propagation from nogil functions by using
some conventions + setjmp/longjmp. Mono does this when calling into
native
code, and I recently did it manually in Cython to propagate exceptions
through the Fortran wrappers in SciPy.
Regardless of the topic of this thread, it would be nice to have
longjmp support in Cython. Lupa, my Cython wrapper for LuaJIT,
currently has to work around several quirks in that area.
Not sure what you mean here, I used longjmp (in a function without any
Python objects) and it seems to work just fine. Did I miss anything?
Also, the GIL may not be around
forever even in CPython? (All arguments I've seen for keeping it has
been
along the lines of "it slows down serial code", not that it is
considered a
good thing.)
If it ever gets removed, there will surely have to be an emulation
layer for C modules. Many of them simply use it as thread-lock, and
that's totally reasonable IMHO.
Good point. But there may be an option to disable said emulation layer
that we want to make use of in Cython...
(This is relevant today for Cython-on-.NET, for instance.)
Designing a language around the GIL feels like a dead-end to me.
We keep having diverging opinions about the GIL. I like it, and I keep
repeating myself by saying that "threading should be explicit". Having
a way to lock the whole interpreter and to keep parallel execution and
reentry points to well defined places in your code is a great feature.
I think all of this just comes from using Cython for totally different
things, which gives us different perspectives on the GIL. I guess we
should just sit down in Munich and look at each other's codes and see if
we can understand one another that way. A big discussion on whether "the
GIL is good or not" is not very constructive.
Dag Sverre
_______________________________________________
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
