First of all, this is from the view of someone who has never created/implemented a programming language, so some of my ideas might be a bit n00bish.
1. Why not give Memory a keyword option type that could be used both when allocating *and* when resizing? I'm thinking something like Memory(type=float, length=i). realloc() or equivalent would instead just take a size that is automatically multiplied by sizeof(...). 2. For resizing arrays, the D programming language uses array.length = i. For Cython it could be: mem.length = i # equivalent to realloc(ptr, sizeof(float) * i) mem.length *= i # equivalent to realloc(ptr, sizeof(float) * i * len(mem) length = mem.length # Get the number of bytes divided by sizeof size = mem.size # Get the number of bytes mem.size = sizeof(a_type) * i # Bypass the automatic sizeof calculations when reallocating or, for multidimensional arrays mem.length = i, j # First dimension = i, second dimension = j mem.length *= i, j # Multiply length by i for the first dimension, j for the second dimension mem.length *= i # Multiply length of both dimensions by i length = mem.length # Get a tuple of the dimensions size = mem.size # The number of bytes *total* I don't really know enough about the Cython compiler to know how those would be implemented. Maybe properties? - Aaron DeVore On Mon, Nov 24, 2008 at 2:11 AM, Stefan Behnel <[EMAIL PROTECTED]> wrote: > Hi, > > when re-reading an older thread about the struct syntax, I had a funny idea > I wanted to share. > > Robert Bradshaw wrote: >> I'm with Stefan that it is very >> dangerous to hide the malloc and expect the user to explicitly >> provide the free. If the user wants to manage their own memory, they >> can do so with malloc and friends (understanding the associated >> dangers), and any special memory-management stuff we add should be as >> implicit and easy to use as Python's garbage collection (and probably >> piggyback off of it). > > The "piggybacking" here makes me wonder if it would work to provide a > dedicated "Memory" object, that would be a Python object but could be used > as in > > def do_stuff_with_dynamic_memory(Py_ssize_t size): > cdef Memory mem > cdef void* ptr > mem = Memory(10*size) # equiv to malloc() > ptr = mem # automagic coercion to a pointer to the memory buffer > # do stuff with *ptr, e.g. hand it around in C code > mem = None # => Py_DECREF(mem), equiv to free() > > Note that the pointer coercion would not return a pointer to the object, > but to the memory buffer. The way this works would be exactly as with the > str() object, which allocates a variable sized memory block in one step > (i.e. a PyVarObject), and accesses the buffer as the last field in the > object struct. > > This obviously gives us the overhead of a Python object in addition to the > allocated memory, but allocating tons of tiny amounts of memory using > malloc() is a bad thing to do anyway, so the average overhead for a medium > to large slice of memory should be acceptable given the advantage of easy > and safe memory handling - especially since AFAIR PyMem_Alloc() is (often) > a lot faster on Windows than malloc(). > > I first had my doubts if auto-coercion to a pointer is a good idea, as in a > plain > > cdef void* ptr = mem > > or as an argument in a function call, so that you could pass "mem" directly > into a C function. What I would like to avoid, is that people write > > cdef void* ptr = Memory(1000) > > and thus drop the Python reference immediately. But Cython should actually > be able to see that, just as it prevents the char* coercion of temporary > Python strings. > > Obviously, Memory must implement the buffer protocol so that you can write > > cdef Memory[float, ndim=2] mem = Memory(100*100*sizeof(float)) > > We could also allow fancy stuff like > > mem = Memory(itemsize=sizeof(int), count=1000) > > or maybe > > cdef Memory[float, ndim=2] mem = \ > Memory(itemsize=sizeof(float), dimensions=(100,100)) > > Supporting realloc is also trivial as in > > mem.realloc(2 * len(mem)) > > and respectively > > mem.realloc(itemsize=sizeof(float), dimensions=(100,200)) > > The nice thing about this is that for the case that realloc() fails, the > method could raise a PyErr_NoMemory() immediately and return NULL to use > the normal exception propagation mechanism. Even one thing less to care > about for users. > > BTW, it would be best to actually implement this in Cython, not C - except > that the custom object allocation itself isn't currently supported in Cython. > > Stefan > _______________________________________________ > Cython-dev mailing list > [email protected] > http://codespeak.net/mailman/listinfo/cython-dev > _______________________________________________ Cython-dev mailing list [email protected] http://codespeak.net/mailman/listinfo/cython-dev
