Just for another data point w.r.t. C callbacks and the GIL, I actually
use this technique in my project HDF5 for Python
(h5py.googlecode.com). The HDF5 library allows you to set an
arbitrary callback function, so I set it to (abridged):
void err_callback(...) with gil:
raise SomeException("Whoops!" )
The library functions I care about are all imported with an exception
clause as e.g:
void HDF5_read_data(...) except -1
so that when I call into HDF5, if an error occurs, a Python exception
is constructed automatically. In particular, the code for reading
data is wrapped in a nogil block:
try:
with nogil:
HDF5_read_data(...)
finally:
... some cleanup ...
In practice this seems to work just fine, although I haven't
extensively tested it with threads. However, all Cython functions in
this module are protected by (the same) reentrant lock as HDF5 isn't
thread safe, so maybe this has been helping me avoid trouble.
Importantly, if the function HDF5_read_data is declared "except *",
Cython immediately crashes if an exception is raised; apparently it's
not valid to check for an exception without holding the GIL.
Declaring with "except <n>" seems to work just fine.
The code in question is at:
http://code.google.com/p/h5py/source/browse/trunk/h5py/h5.pyx (line 651)
and
http://code.google.com/p/h5py/source/browse/trunk/h5py/h5d.pyx (line 161)
Andrew Collette
On Tue, Apr 14, 2009 at 11:19 AM, Dag Sverre Seljebotn
<[email protected]> wrote:
> Thanks for the feedback. I'm quite keen to get this resolved myself so a
> patch is sure to follow once I can get the semantics part sorted.
>
> Stefan Behnel wrote:
>> Well, exceptions happen to be the error reporting mechanism in Python. In
>> any case, the above must exit the nogil block immediately and should not
>> crash (as long as we can prevent it in a reasonable way). To me, this
>> sounds like an exception is the right thing.
>>
>> However, it's quite possible that this kind of control flow makes error
>> handling and C-level cleanup measures a nightmare. How would such a nogil
>> block interact with try-except or try-finally, for example? We can't allow
>> try-except /inside/ the block, so you'd have to anticipate all errors and
>> wrap separate sections of the nogil block in their own try-except or
>> try-finally statements to make sure the cleanup works as required.
>
> I'm not sure I get what you mean here; but anyway:
>
> Looking at generated C source, it seems that "with nogil" sets up a
> plain "try/finally" block, with an acquisition of the GIL in the finally
> part. So even as it is it is perfectly possible to raise an exception
> the normal way; labels are set correctly and the exception-raising-goto
> path is resumed after acquisition of the GIL.
>
> However I don't think any code makes use of this (?), and if it is, it
> raises an issue with the (lack of) thread-safety of the
> __pyx_filename/lineno/clineno variables (which in other places are
> protected by the GIL).
>
> How about this:
>
> Change things so that instead of a GIL block setting up a "try/finally"
> to reacquire the GIL, any exception-raising code within a GIL block is
> responsible for reacquiring the GIL. (Basically, code.funcstate.in_nogil
> is maintained and if nodes want to raise an exception they reacquire the
> GIL as part of doing that.)
>
>>> Option 2) Only allow buffer use in nogil if @cython.boundscheck(False).
>>> This has some usability problems; it would be a pain to turn on/off
>>> nogil for debugging.
>>
>> Sounds like a viable alternative, though. Should be followed if exception
>> raising is not considered accurate from inside nogil blocks.
>
> I might end up doing this for 0.11 and await input for the scheme above
> for 0.12 (if I decide to bother with 0.11).
>
> --
> Dag Sverre
> _______________________________________________
> Cython-dev mailing list
> [email protected]
> http://codespeak.net/mailman/listinfo/cython-dev
>
>
_______________________________________________
Cython-dev mailing list
[email protected]
http://codespeak.net/mailman/listinfo/cython-dev
