On Thu, Aug 18, 2011 at 11:35 PM, Dag Sverre Seljebotn <d.s.seljeb...@astro.uio.no> wrote: > On 08/18/2011 09:27 PM, Robert Bradshaw wrote: >> >> On Wed, Aug 17, 2011 at 11:39 PM, Dag Sverre Seljebotn >> <d.s.seljeb...@astro.uio.no> wrote: >>> >>> On 08/17/2011 09:21 PM, Robert Bradshaw wrote: >>>> >>>> On Wed, Aug 17, 2011 at 11:46 AM, Dag Sverre Seljebotn >>>> <d.s.seljeb...@astro.uio.no> wrote: >>>>> >>>>> On 08/17/2011 08:19 PM, Robert Bradshaw wrote: >>>>>> >>>>>> That's a nice idea. I have to admit that all these special gil >>>>>> declarations are a bit messy. I'd also rather introduce clear >>>>>> decorators, e.g. >>>>>> >>>>>> @cython.requires_gil # expects gil >>>>>> cdef a(): ... >>>>>> >>>>>> @cython.requires.gil(False) # nogil >>>>>> cdef b(): ... >>>>>> >>>>>> @cython.aquires_gil # with gil >>>>>> cdef c(): ... >>>>>> >>>>>> (Actually, now that we have the "with gil" statement, it could be >>>>>> worth considering simply noticing the pattern of the entire function >>>>>> body in a with gil block/as the first statement and acquiring the GIL >>>>>> before argument parsing.) >>>>>> >>>>>> Note that we need to declare functions as requiring the GIL to allow >>>>>> for declaring cpython.pxd if extern functions are implicitly nogil. >>>>> >>>>> I agree, it's messy in the current situation, simplifying would be >>>>> good. >>>>> >>>>> Assuming we can't acquire the GIL in every single function just to be >>>>> sure, >>>>> I have a hunch that the "acquires_gil" aspect of a function is just >>>>> declared >>>>> in the wrong place. I mean, the same function might be passed as a >>>>> callback >>>>> to C both while holding the GIL and while not holding the GIL -- it >>>>> would >>>>> be >>>>> nice to automatically wrap it in a GIL-acquiring wrapper only when >>>>> needed. >>>>> >>>>> So to me it makes more sense to have acquires_gil be part of function >>>>> pointer types, or of the C call where the pointer is passed, or >>>>> similar. >>>>> Sort of like an FFI. Can't think of a practical scheme that's more >>>>> user-friendly than the current way though... >>>> >>>> I was thinking the opposite, "aquires_gil" should be completely >>>> transparent to the caller (assuming it's cheap enough to >>>> check-or-acquire, but that's an optimization not API issue). On the >>>> other hand requires/does not require does need to be visible to the >>>> caller though, which argues for it being part of the signature. >>> >>> Are you saying that every single function cdef function, ever, that are >>> not >>> "nogil" should have acqusition semantics? >> >> No, I was saying that the distinction between with gil and nogil >> should be transparent (semantically) to the caller. >> >>> Those semantics would be great, but I worry a bit about performance -- we >>> just agreed to make function GIL-acquisition even a bit more expensive >>> with >>> that check... >> >> According to your timings, the performance argument is a good one. >> >>> Hmm. How about this: >>> >>> i) Every cdef function that needs the GIL to be held generates a >>> GIL-acquiring wrapper function as well. >>> >>> ii) Whenever taking the address of such a function, you get the address >>> of >>> the GIL-requiring wrapper, which can safely be used from any code, >>> whether >>> holding the GIL or not. >> >> You mean GIL-acquiring wrapper, right? > > Yes, sorry. > >> >>> iii) However, Cython code that already holds the GIL can call the inner >>> function directly, as an optimization. Should be possible to do this >>> across >>> pxd's as well. >>> >>> iv) We may introduce a cython.nogil_function_address or similar just to >>> provide an option to get around ii). >>> >>> v) The "with gil" on functions simply ceases to take effect, and is >>> deprecated from the language. >>> >>>> Regarding inference, are you thinking the semantics being that we >>>> (re)-acquire the GIL for every individual operation that needs it >>>> (including python operations or entire with gil blocks), with the >>>> obvious optimization that we may choose to not release it between >>>> (nearly) consecutive blocks of code that all need the GIL? That would >>>> be truer to Python semantics, but would require risky guesswork at >>>> compile time or some kind of periodic runtime checks. (It would also >>>> be a strongly backwards incompatible move, so as mentioned you'd need >>>> some kind of a explicit marker and deprecation period.) >>>> >>> >>> No, I wasn't talking about this at all (this time on the list -- I did >>> mention that I wished for this behaviour in Munich, but that's really >>> long >>> term). >>> >>> All I wished for now was for simple code like this: >>> >>> cdef double f(double x): return x**2 + x >>> >>> to be callable without holding the GIL. In particular since this is the >>> major reason why prange users need to learn the "nogil" keyword. >>> >>> The semantics are simple: For all cdef functions, if "nogil" could have >>> been >>> applied without a syntax error, then it gets automatically applied. >> >> Give that the implicit nogil marker is a function of the function's >> body, how would this information be propagated across pxd files? >> Perhaps I'm confused, we have >> >> cdef double f(double x): # implicitly nogil >> return x**2 + x >> >> cdef double f(double x): # implicitly "with gil" or implicitly >> "requires gil"? >> print x >> return x**2 + x >> >> If "with gil" then constant, implicit gil acquisition could be a >> non-obvious performance hit; if "requires gil" then I don't see how to >> propagate accross .pxd files. (I do like the direction this is >> heading, just pointing out some cons.) > > True, I didn't think about the pxd issue. > > Brainstorming: > > I) Downsize my proposal to within-module calls, require nogil for > cross-module. > > II) Change Cython so that any function can be called from within "nogil" > sections (and then their GIL-acquiring wrapper, if any, will be called). Of > course this is a little fussy regarding exceptions etc. and need some > thinking through (which I haven't done). > > Together with the proposal above to deprecate the "with gil" modifier and > always generate GIL-acquiring wrappers, this works across modules: The > module function pointer table gets two slots per cdef function; one to call > if the GIL is held, and one to call if the GIL is not held. Functions that > are nogil (explicitly or implicitly) then simply use the same pointer in > both slots.
Sounds reasonable. Is there any scenario where one would explicitly not want the GIL-acquiring wrapper to be created/potentially used? >>> The only time this isn't completely safe is when the GIL is intentionally >>> being used as a lock. >> >> Or unintentionally being used as a lock... (Yes, that'd be user >> error.) The idea that adding a print statement to a function's body >> can produce such a large change is somewhat worrisome, but perhaps >> worth it. > > My hunch is to postpone the question of inferring the nogil modifier. > Instead, first focus on making the "with gil" modifier optional > (GIL-acquiring wrapper on all cdef functions), and see where that takes us. > > I guess I can raise the question again with a formal CEP when I have time to > do something about it in terms of code... Makes sense. - Robert _______________________________________________ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
