Re: [Numpy-discussion] Numpy array from ctypes pointer object?
Albert Strasheim wrote: > Hello all > > Various people wrote: > >>> Im curious though: the several projects recently using ctypes >>> and numpy to wrap libraries (Pygame SDL, OpenGL, svm) must have come >>> across the issue of using a creating a numpy array from a ctypes >>> pointer. Ill have to look further. >>> >>> >> It depends on whether or not the library creates memory or not. Not >> every library manages memory (some expect you to pass in memory already >> owned --- this is easy to do already with ctypes and numpy). >> > > I see two main models for sending back data from C land. > > One is to have the C function allocate memory and return the pointer to the > caller. This raises the question of who owns the memory. It also leads to > some *very* interesting crashes on Windows when you start freeing memory > allocated in code linked against one runtime in code linked against another > runtime. I think, if you have the option, avoid this kind of thing at all > costs. > > The other model has functions that take their normal arguments, and pointers > to buffers where they can store their results. Typically the caller would > also specify the size of the buffer. If the buffer is large enough for the > function to do its work, it uses it. If not, it returns an error code. This > way, one entity is always in charge of the memory. > > If you're writing C code that you plan to use with ctypes and NumPy, I think > the second model will lead to more robust code and less time spent debugging > crashes and memory leaks. > Yes, I agree, C libs written with your model #2 would make life much easier to create robust wrappers. Many of the ones that come to my mind, and deal with array like data and thus relevant, are model #1 types. I think the reason this is so goes something like this: Joe Foo developer & friends writes an entire library API in C. To make the API complete and easy to use Joe includes getting started 'make the data' C functions. Examples: SDL: SDL_surface* SDL_CreateRGBSurface(params) Opencv: IplImage* cvCreateImage(params), ... libdc1394: uint_t * capture buffer //created by the OS 1394 driver etc. If the Joe doesn't do this then the first thing Joe's 'users' must do to create the simplest application is call malloc, sizeof( figure out what element type...), blah, blah, i.e. do lots of memory management having zilch to do with the problem at hand. Further, Joe, being conscientious, includes boat loads of examples all using the 'make the data' calls. > libsvm (which I'm wrapping with ctypes) mostly follows the second model, > except when training new models, in which case it returns a pointer to a > newly allocated structure. To deal with this, I keep a pointer to this > allocated memory in a Python object that has the following function: > > def __del__(self): > libsvm.svm_destroy_model(self.model) > > Nice > By providing this destroy function, libsvm avoids the problem of mixing > allocation and deallocation across runtimes. > > Regards, > > Albert > > -Mark - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy array from ctypes pointer object?
Hello all Various people wrote: > > Im curious though: the several projects recently using ctypes > > and numpy to wrap libraries (Pygame SDL, OpenGL, svm) must have come > > across the issue of using a creating a numpy array from a ctypes > > pointer. Ill have to look further. > > > It depends on whether or not the library creates memory or not. Not > every library manages memory (some expect you to pass in memory already > owned --- this is easy to do already with ctypes and numpy). I see two main models for sending back data from C land. One is to have the C function allocate memory and return the pointer to the caller. This raises the question of who owns the memory. It also leads to some *very* interesting crashes on Windows when you start freeing memory allocated in code linked against one runtime in code linked against another runtime. I think, if you have the option, avoid this kind of thing at all costs. The other model has functions that take their normal arguments, and pointers to buffers where they can store their results. Typically the caller would also specify the size of the buffer. If the buffer is large enough for the function to do its work, it uses it. If not, it returns an error code. This way, one entity is always in charge of the memory. If you're writing C code that you plan to use with ctypes and NumPy, I think the second model will lead to more robust code and less time spent debugging crashes and memory leaks. libsvm (which I'm wrapping with ctypes) mostly follows the second model, except when training new models, in which case it returns a pointer to a newly allocated structure. To deal with this, I keep a pointer to this allocated memory in a Python object that has the following function: def __del__(self): libsvm.svm_destroy_model(self.model) By providing this destroy function, libsvm avoids the problem of mixing allocation and deallocation across runtimes. Regards, Albert - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy array from ctypes pointer object?
Mark Heslep wrote: > Travis Oliphant wrote: > >> The problem here is that from Python NumPy has no way to create an >> ndarray from a pointer. Doing this creates a situtation where it is >> unclear who owns the memory. It is probably best to wrap the pointer >> into some kind of object exposing the buffer protocol and then pass >> that to frombuffer (or ndarray.__new__). >> > Yep thats where I just ended up: > > from ctypes import * > import numpy as N > ... > func = pythonapi.PyBuffer_FromMemory > func.restype = py_object > buffer = func( im.imageData, size_of_the_data ) >return N.frombuffer( buffer, N.uint8 ) > > Works! Nice job! > Im curious though: the several projects recently using ctypes > and numpy to wrap libraries (Pygame SDL, OpenGL, svm) must have come > across the issue of using a creating a numpy array from a ctypes > pointer. Ill have to look further. > > It depends on whether or not the library creates memory or not. Not every library manages memory (some expect you to pass in memory already owned --- this is easy to do already with ctypes and numpy). >> When an ndarray is using memory that is not its own, it expects >> another object to be "in charge" of that memory and the ndarray will >> point its base attribute to it and increment its reference count. >> What should the object that is "in charge" of the memory be? >> Perhaps a suitable utility function could be created that can work >> with ctypes to create ndarrays from ctypes memory locations and either >> own or disown the data. >> >> > I suppose that is still the case w/ PyBuffer_From.. above. That is, the > underlying im.imageData pointer can not be released before buffer. > Yes, you are right. It is the memory that is most critical. Who owns the memory pointed to by im.imageData? When will it be freed? The ndarray object is holding a reference to the Python buffer object which is just *hoping* that the memory it was initiated with is not going to be freed before it gets deallocated (which won't happen until at least the ndarray object is deallocated). So, managing the memory can be a bit tricky. But, if you are sure that im.imageData memory will not be freed then you are O.K. -Travis - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy array from ctypes pointer object?
Travis Oliphant wrote: > The problem here is that from Python NumPy has no way to create an > ndarray from a pointer. Doing this creates a situtation where it is > unclear who owns the memory. It is probably best to wrap the pointer > into some kind of object exposing the buffer protocol and then pass > that to frombuffer (or ndarray.__new__). Yep thats where I just ended up: from ctypes import * import numpy as N ... func = pythonapi.PyBuffer_FromMemory func.restype = py_object buffer = func( im.imageData, size_of_the_data )When an ndarray is using memory that is not its own, it expects > another object to be "in charge" of that memory and the ndarray will > point its base attribute to it and increment its reference count. > What should the object that is "in charge" of the memory be? > Perhaps a suitable utility function could be created that can work > with ctypes to create ndarrays from ctypes memory locations and either > own or disown the data. > I suppose that is still the case w/ PyBuffer_From.. above. That is, the underlying im.imageData pointer can not be released before buffer. Mark > This needs to be thought through a bit, however. > > -Travis > > > >> The attributes in nparray.__array_interface_ are not writable, so no >> joy there. >> >> On the C side the PyArray_SimpleNewFromData( ..dimensions, ...data >> ptr) C API does the job nicely. Is there a ctypes paradigm for >> SimpleNew...? >> >> Mark >> >> >> >> - >> >> Using Tomcat but need to do more? Need to support web services, >> security? >> Get stuff done quickly with pre-integrated technology to make your >> job easier >> Download IBM WebSphere Application Server v.1.0.1 based on Apache >> Geronimo >> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 >> ___ >> Numpy-discussion mailing list >> Numpy-discussion@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/numpy-discussion >> > - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy array from ctypes pointer object?
Mark Heslep wrote: > Travis Oliphant wrote: > >> Mark Heslep wrote: >> >>> I don't see a clean way to create a numpy array from a ctypes pointer >>> object. Is the __array_interface_ in ctypes the thing thats missing >>> needed to make this happen? I've followed Albert's Scipy cookbook >>> on ctypes here >>> >>> On the C side the PyArray_SimpleNewFromData( ..dimensions, ...data >>> ptr) C API does the job nicely. Is there a ctypes paradigm for >>> SimpleNew...? >>> >>> >> Can you somehow call this function using ctypes? >> >> -Travis >> > That might work, though indirectly. As I think I understand from ctypes > docs: Ctypes uses functions exposed in a shared library, macros > existing only a header are not available. If its PyArray... is a macro > then I a) need to compile and make a little library directly from > arrayobject.h or b) need to use the root function upon which the macro > is based, PyArrayNew? > > This is more complicated because all the C-API functions are actually just pointers stored in _ARRAY_API of multiarray.So, something would have to be built to interpret the C-pointers in that C-Object. I'm not sure that is possible. -Travis - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy array from ctypes pointer object?
Mark Heslep wrote: > I don't see a clean way to create a numpy array from a ctypes pointer object. > Is the __array_interface_ in ctypes the thing thats missing needed to make > this happen? I've followed Albert's Scipy cookbook on ctypes here > > http://www.scipy.org/Cookbook/Ctypes <-- getting dated now for numpy SVN but > the idea is clear > > This shows clean ways to > 1) Create a ctype array object from a numpy array: > npptr= nparray.__array_interface__['data'][0] > ctarray = (c_double * nparray.size).from_address(npptr) > > 2) Create a numpy array from a ctypes array (not a pointer): > ctarray = (c_double * 2)() > nparray = N.array( ctarray ) > > But if I'm starting with say, a POINTER(c_int) and a separate size argument > that's returned from a ctypes foreign function or provided by _fields_ > element in a ctypes structure, then I'm unclear how to provide the pointer in > Python to numpy. The numpy methods or creating an array in Python as I see > it are basically two: > > N.array, N.asarray <-- require a sequence object > N.fromstring, N.frombuffer <-- not available unless given c_char_p and even > thats wrong if I don't want zero termination. > The problem here is that from Python NumPy has no way to create an ndarray from a pointer. Doing this creates a situtation where it is unclear who owns the memory. It is probably best to wrap the pointer into some kind of object exposing the buffer protocol and then pass that to frombuffer (or ndarray.__new__). When an ndarray is using memory that is not its own, it expects another object to be "in charge" of that memory and the ndarray will point its base attribute to it and increment its reference count. What should the object that is "in charge" of the memory be? Perhaps a suitable utility function could be created that can work with ctypes to create ndarrays from ctypes memory locations and either own or disown the data. This needs to be thought through a bit, however. -Travis > The attributes in nparray.__array_interface_ are not writable, so no joy > there. > > On the C side the PyArray_SimpleNewFromData( ..dimensions, ...data ptr) C API > does the job nicely. Is there a ctypes paradigm for SimpleNew...? > > Mark > > > > - > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > ___ > Numpy-discussion mailing list > Numpy-discussion@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy array from ctypes pointer object?
Mark Heslep schrieb: > Travis Oliphant wrote: >> Mark Heslep wrote: >>> I don't see a clean way to create a numpy array from a ctypes pointer >>> object. Is the __array_interface_ in ctypes the thing thats missing >>> needed to make this happen? I've followed Albert's Scipy cookbook >>> on ctypes here >>> >>> On the C side the PyArray_SimpleNewFromData( ..dimensions, ...data >>> ptr) C API does the job nicely. Is there a ctypes paradigm for >>> SimpleNew...? >>> >> Can you somehow call this function using ctypes? >> >> -Travis > That might work, though indirectly. As I think I understand from ctypes > docs: Ctypes uses functions exposed in a shared library, macros > existing only a header are not available. If its PyArray... is a macro > then I a) need to compile and make a little library directly from > arrayobject.h or b) need to use the root function upon which the macro > is based, PyArrayNew? Sure, macros will not work. > ctypes has built-in access to the functions in the core python library, > but Im not sure which library contains all the PyArray API calls, since > its all typically hidden by distutils and setup.py On Linux it doesn't matter, since you don't need to know which library exports a function (If I understand shared libs in Linux correctly). At least dlsym can be called with a NULL handle. On windows, functions can only be retrieved from exactly the library that exposes them. Plus, the function must explicitly marked exported either by compiler directives in the source code of by listing them in a .def file. Thomas - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy array from ctypes pointer object?
Travis Oliphant wrote: > Mark Heslep wrote: >> I don't see a clean way to create a numpy array from a ctypes pointer >> object. Is the __array_interface_ in ctypes the thing thats missing >> needed to make this happen? I've followed Albert's Scipy cookbook >> on ctypes here >> >> On the C side the PyArray_SimpleNewFromData( ..dimensions, ...data >> ptr) C API does the job nicely. Is there a ctypes paradigm for >> SimpleNew...? >> > Can you somehow call this function using ctypes? > > -Travis That might work, though indirectly. As I think I understand from ctypes docs: Ctypes uses functions exposed in a shared library, macros existing only a header are not available. If its PyArray... is a macro then I a) need to compile and make a little library directly from arrayobject.h or b) need to use the root function upon which the macro is based, PyArrayNew? ctypes has built-in access to the functions in the core python library, but Im not sure which library contains all the PyArray API calls, since its all typically hidden by distutils and setup.py Mark - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy array from ctypes pointer object?
Mark Heslep wrote: > I don't see a clean way to create a numpy array from a ctypes pointer object. > Is the __array_interface_ in ctypes the thing thats missing needed to make > this happen? I've followed Albert's Scipy cookbook on ctypes here > > On the C side the PyArray_SimpleNewFromData( ..dimensions, ...data ptr) C API > does the job nicely. Is there a ctypes paradigm for SimpleNew...? > Can you somehow call this function using ctypes? -Travis - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
[Numpy-discussion] Numpy array from ctypes pointer object?
I don't see a clean way to create a numpy array from a ctypes pointer object. Is the __array_interface_ in ctypes the thing thats missing needed to make this happen? I've followed Albert's Scipy cookbook on ctypes here http://www.scipy.org/Cookbook/Ctypes <-- getting dated now for numpy SVN but the idea is clear This shows clean ways to 1) Create a ctype array object from a numpy array: npptr= nparray.__array_interface__['data'][0] ctarray = (c_double * nparray.size).from_address(npptr) 2) Create a numpy array from a ctypes array (not a pointer): ctarray = (c_double * 2)() nparray = N.array( ctarray ) But if I'm starting with say, a POINTER(c_int) and a separate size argument that's returned from a ctypes foreign function or provided by _fields_ element in a ctypes structure, then I'm unclear how to provide the pointer in Python to numpy. The numpy methods or creating an array in Python as I see it are basically two: N.array, N.asarray <-- require a sequence object N.fromstring, N.frombuffer <-- not available unless given c_char_p and even thats wrong if I don't want zero termination. The attributes in nparray.__array_interface_ are not writable, so no joy there. On the C side the PyArray_SimpleNewFromData( ..dimensions, ...data ptr) C API does the job nicely. Is there a ctypes paradigm for SimpleNew...? Mark - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
