Robert Bradshaw wrote:
> Thanks! I think this will help immensely for watching for memory
> leaks in Sage.
Yep, this is great. Given that Dag seems to have been slowed down by
real world things I am curious what needs to be done to get this into a
more usable state? Linux only is fine for me for now, but if this
depeneds on 0.11-devel I would probably be not be able to use this.
Cheers,
Michael
> On Jan 7, 2009, at 6:12 AM, Dag Sverre Seljebotn wrote:
>
>> I've had little time for Cython lately and so refcount nanny
>> development
>> has been slow. But when development is going at this pace I figured it
>> was better to upload what I had so it doesn't start bit-rotting on
>> my disk.
>>
>> It is NOT FINISHED, but the ideas and framework is mostly in place, it
>> is just a matter of working out all the places it fails. I'll continue
>> it myself when I get time, but feel free...
>>
>> By default it is disabled and Cython works as before. To enable it:
>> cd Cython/Runtime
>> ./build.sh # yes, ugh! Let's deal with that when the rest is done
>> cd ../..
>> python runtests.py --refnanny ...
>>
>> One can then get reference count errors for two reasons:
>> - Most likely: The required calls to __Pyx_GOTREF and __Pyx_GIVEREF is
>> missing in a given situation
>> - Otherwise: There's a genuine reference counting problem.
>>
>> Here's the details:
>> - Cython generates code that can always be run with or without the
>> refnanny -- to run it with, compile the C code with CYTHON_REFNANNY
>> defined.
>> - Some boilerplate code is added when entering and exiting all
>> functions
>> (which should compile into nothing when not using the nanny)
>> - Many calls to Py_INCREF etc. are replaced with __Pyx_INCREF etc.
>> (but
>> not in utility code which is in general trusted and doesn't use the
>> nanny)
>> - New calls are added: __Pyx_GOTREF, __Pyx_GIVEREF, __Pyx_XGIVEREF
>> which
>> are called after getting a reference from another function and before
>> giving away a reference to a function/array/struct field. I.e:
>>
>> __Pyx_GIVEREF(obj)
>> PyList_SET_ITEM(..., obj, ...)
>>
>> and
>>
>> obj = Py_Get...
>> if (!obj) { ... }
>> __Pyx_GOTREF(obj)
>>
>> (Also, XGIVEREF is called on __pyx_r prior to returning.)
>>
>> - There's a hook module in Cython/Runtime/refnanny.pyx. For now
>> this is
>> manually compiled with a Linux-only shell script, this must of
>> course be
>> done something about. See [1] for more plans for this module.
>> - runtests.py has the --refnanny flag to turn on the refnanny (when it
>> is done, I expect this to be the default). What this does is
>> loading the
>> hook module and compiling modules with CYTHON_REFNANNY defined.
>>
>> End result:
>> - Extra decrefs are intercepted and only logged.
>> - Functions continue as normal when this happen, but when the function
>> returns for any reason a normal exception is raised if there's a
>> leak or
>> there was an extra decref somewhere.
>>
>> [1] The module can be written to contain as advanced diagnostics as we
>> like, and can be inspected by runtests.py to read logs etc.. It
>> would be
>> entirely possible to extend this to allow, say, putting assertions on
>> the refcount operations Cython code performs in the doctests that
>> runtests.py runs:
>>
>> def f():
>> """
>> Test that this functions does the reference counting we would like.
>>
>>>>> import refnanny
>>>>> a = 2
>>>>> refnanny.monitor_object(a)
>>>>> f(a)
>>>>> print refnanny.operations_on(a)
>> INCREFS=10, DECREFS=10
>> """
>> # ...
>>
>> That's an argument for keeping it as a seperate module, but that's all
>> I'm going to do about it for now.
>>
>> Dag Sverre
>> _______________________________________________
>> Cython-dev mailing list
>> [email protected]
>> http://codespeak.net/mailman/listinfo/cython-dev
>
> _______________________________________________
> Cython-dev mailing list
> [email protected]
> http://codespeak.net/mailman/listinfo/cython-dev
>
_______________________________________________
Cython-dev mailing list
[email protected]
http://codespeak.net/mailman/listinfo/cython-dev
