Hi,
when re-reading an older thread about the struct syntax, I had a funny idea
I wanted to share.
Robert Bradshaw wrote:
> I'm with Stefan that it is very
> dangerous to hide the malloc and expect the user to explicitly
> provide the free. If the user wants to manage their own memory, they
> can do so with malloc and friends (understanding the associated
> dangers), and any special memory-management stuff we add should be as
> implicit and easy to use as Python's garbage collection (and probably
> piggyback off of it).
The "piggybacking" here makes me wonder if it would work to provide a
dedicated "Memory" object, that would be a Python object but could be used
as in
def do_stuff_with_dynamic_memory(Py_ssize_t size):
cdef Memory mem
cdef void* ptr
mem = Memory(10*size) # equiv to malloc()
ptr = mem # automagic coercion to a pointer to the memory buffer
# do stuff with *ptr, e.g. hand it around in C code
mem = None # => Py_DECREF(mem), equiv to free()
Note that the pointer coercion would not return a pointer to the object,
but to the memory buffer. The way this works would be exactly as with the
str() object, which allocates a variable sized memory block in one step
(i.e. a PyVarObject), and accesses the buffer as the last field in the
object struct.
This obviously gives us the overhead of a Python object in addition to the
allocated memory, but allocating tons of tiny amounts of memory using
malloc() is a bad thing to do anyway, so the average overhead for a medium
to large slice of memory should be acceptable given the advantage of easy
and safe memory handling - especially since AFAIR PyMem_Alloc() is (often)
a lot faster on Windows than malloc().
I first had my doubts if auto-coercion to a pointer is a good idea, as in a
plain
cdef void* ptr = mem
or as an argument in a function call, so that you could pass "mem" directly
into a C function. What I would like to avoid, is that people write
cdef void* ptr = Memory(1000)
and thus drop the Python reference immediately. But Cython should actually
be able to see that, just as it prevents the char* coercion of temporary
Python strings.
Obviously, Memory must implement the buffer protocol so that you can write
cdef Memory[float, ndim=2] mem = Memory(100*100*sizeof(float))
We could also allow fancy stuff like
mem = Memory(itemsize=sizeof(int), count=1000)
or maybe
cdef Memory[float, ndim=2] mem = \
Memory(itemsize=sizeof(float), dimensions=(100,100))
Supporting realloc is also trivial as in
mem.realloc(2 * len(mem))
and respectively
mem.realloc(itemsize=sizeof(float), dimensions=(100,200))
The nice thing about this is that for the case that realloc() fails, the
method could raise a PyErr_NoMemory() immediately and return NULL to use
the normal exception propagation mechanism. Even one thing less to care
about for users.
BTW, it would be best to actually implement this in Cython, not C - except
that the custom object allocation itself isn't currently supported in Cython.
Stefan
_______________________________________________
Cython-dev mailing list
[email protected]
http://codespeak.net/mailman/listinfo/cython-dev
