On Nov 24, 2008, at 3:56 PM, Michael Abshoff wrote: > Stefan Behnel wrote: >> Hi, > > Hi, > >> when re-reading an older thread about the struct syntax, I had a >> funny idea >> I wanted to share. >> >> Robert Bradshaw wrote: >>> I'm with Stefan that it is very >>> dangerous to hide the malloc and expect the user to explicitly >>> provide the free. If the user wants to manage their own memory, they >>> can do so with malloc and friends (understanding the associated >>> dangers), and any special memory-management stuff we add should >>> be as >>> implicit and easy to use as Python's garbage collection (and >>> probably >>> piggyback off of it). >> >> The "piggybacking" here makes me wonder if it would work to provide a >> dedicated "Memory" object, that would be a Python object but could >> be used >> as in >> >> def do_stuff_with_dynamic_memory(Py_ssize_t size): >> cdef Memory mem >> cdef void* ptr >> mem = Memory(10*size) # equiv to malloc() >> ptr = mem # automagic coercion to a pointer to the >> memory buffer >> # do stuff with *ptr, e.g. hand it around in C code >> mem = None # => Py_DECREF(mem), equiv to free() >> > > In general I consider this a great idea since people will be less > likely > to shoot themselves in the foot and as you mentioned for small allocs > this should really be a win. Having memory automatically freed on > exceptions would also be great since that is tricky to write. The less > the average user has to deal with memory allocs and deallocs in Cython > code the better Cython will be.
I think this is a great idea too. Memory could have two C attributes like len and ptr (to avoid automagic coercion), and a realloc method (which may of course move ptr). Could probably be implemented on top of the str/bytes object with a new entry in the builtins table and perhaps some tree transformations to make creating one very fast. > But I am concerned to some extend about the success of hunting memory > leaks if this is the default allocator in some future version of > Cython > and there is no way to use malloc() [currenly Sage provide > sagemalloc(), > so that for example could be made to use > do_stuff_with_dynamic_memory() > since we only need to change one define]. Looking for heap objects in > Python that are leaked or not properly dereferenced is a giant pain > right now and valgrind is next to useless here. muppy is starting to > solve that problem, but there are at least in Sage issues with ipython > allocating tens of thousands of heap objects at load time impacting > performance of muppy badly, but those issues will be hopefully fixed. > Anyway, no need to rant about python memory heap debugging here :) > > Since in Sage I have to deal with loads of small, but potentially > large > cumulative leaks I would consider it very nice if the allocator could > have some magic debug fallback to malloc(). That should certainly > be not > the default behavior, just like --without-pymalloc is not the default > for Python. +1 to this idea. - Robert _______________________________________________ Cython-dev mailing list [email protected] http://codespeak.net/mailman/listinfo/cython-dev
