I'm attempting to write a library for reading data via USB from a
device and processing the data to display graphs. I have already
implemented parts of this code as pure python, as a proof of concept
but I have now moved on to implementing the functions in a C
extension.
My original plan was to have the data processing and data acquisition
functions running in separate processes, with a multiprocessing.Queue
for passing the raw data packets. The raw data is read in as a char*,
with a non constant length, hence I have allocated memory using
PyMem_Malloc and I am returning from the acquisition function a
PyCObject containing a pointer to this char* buffer, along with a
destructor. The following code shows a simple test function I've
written (with some module/class boilerplate removed) to demonstrate
this.
static void p_destruct(void *p) {
PyMem_Free((void*)p);
}
static PyObject *malloc_buffer(MyClass *k1) {
PyObject *cobj;
char *foo = PyMem_Malloc(1024 * sizeof(char));
if (foo == NULL) {
return NULL;
}
foo = "foo";
cobj = PyCObject_FromVoidPtr(foo, p_destruct);
return cobj;
}
static PyObject *retrieve_buffer(MyClass *k1, PyObject *args) {
char *foo2;
PyObject cobj2;
char *kwlist[] = {"foo1", NULL};
if (!PyArg_ParseTuple(args, "O", &cobj2)) {
return NULL;
}
foo2 = PyCObject_AsVoidPtr(cobj2);
//Do something
PySys_WriteStdout(foo2);
Py_RETURN_NONE;
}
So if I call these functions in a loop, e.g. The following will
generate ~10GB of data
x = MyClass()
for i in xrange(0, 10 * 2**20):
c = x.malloc_buffer()
x.retrieve_buffer(c)
All my memory disapears, until python crashes with a MemoryError. By
placing a print in the destructor function I know it's being called,
however it's not actually freeing the memory. So in short, what am I
doing wrong?
This is the first time I've written a non-trivial python C extension,
and I'm still getting my head round the whole Py_INC/DECREF and the
correct way to manage memory, so I spent a while playing around with
incref/decref but I left these out of my above example to keep what
I'm trying to achieve clearer.
Also, I'm aware PyCObject is deprecated in >=2.7 but I'm targeting
Python 2.6 at the moment, and I will move on to using capsules once
I've made the big jump with some other libraries. So if there is
anything that could be hugely different using capsules could you point
this out.
I'm developing using:
Python - 2.6.5
Windows XP (although linux is a future target platform)
msvc compiler
Cheers, any help would be greatly appreciated.
--
http://mail.python.org/mailman/listinfo/python-list
