Dag Sverre Seljebotn skrev: > he usecase here was wrapping a C library, and wrapping data returned by > the C lib in an ndarray. Setting base is necesarry to free the memory... > >
Here is an example that I and Gaël Varoquaux wrote some time ago. It is an ndarray that uses named shared memory as buffer, and is pickled by name, i.e. the buffer is not copied by pickle, only its name. Thus it can be quickly recreated in a different process, allowing processes to communicate shared-memory ndarrays using multiprocessing.Queue. The way we set base is actually by calling np.asarray on an object that has an __array_interface__ attribute, which is a Python dictionary. We don't write to the base attribute directly. http://folk.uio.no/sturlamo/python/sharedmem-feb13-2009.zip The array passed to np.asarray is BufferWrapperArray, defined like this (mostly Robert Kern's code): def get_fortran_strides(shape, dtype): """ Create a strides entry for the given structural information for a Fortran-strided array. """ strides = tuple(dtype.itemsize * np.cumprod((1,) + shape[:-1])) return strides class BufferWrapperArray(object): """ An object that exists to expose a BufferWrapper with an __array_interface__. """ def __init__(self, wrapper, shape, dtype, order, strides, offset): self.wrapper = wrapper if strides is None: if order == 'C': strides = None else: strides = get_fortran_strides(shape, dtype) self.__array_interface__ = dict( data = (wrapper.get_address() + offset, False), descr = dtype.descr, shape = shape, strides = strides, typestr = dtype.str, version = 3, ) Without going into details, the shared memory segment (created with Windows API or System V IPC) is owned by a "Handle" class written in Cython. It commits suicide by calling __dealloc__ when no references to it are kept. That is what frees up the memory. The chain of references down to the ndarray is then this: Handle --> SharedMemoryBuffer --> BufferWrapper --> BufferWrapperArray --> ndarray The ndarray is constructed in plain Python, using an int to store the data address in the __array_interface__. The boolean that comes after the address in the 'data' tuple, says that the ndarray does not own the memory. The transition from Cython to Python is between SharedMemoryBuffer and BufferWrapper. Thus we never set base directly, np.asarray takes care of that. Sturla Molden _______________________________________________ Cython-dev mailing list [email protected] http://codespeak.net/mailman/listinfo/cython-dev
