Re: [Numpy-discussion] Best way to expose std::vector to be used with numpy
Sorry about SWIG -- maybe a chance to move on ;-) I'd go with Cython -- this is pretty straightforward, and it handles the buffer protocol for you under the hood. And with XDress, you can get numpy wrapped std::vector out of the box, I think: https://s3.amazonaws.com/xdress/index.html if you REALLY want to stick with SWIG, take a look a the SWIG numpy interface files -- they are designed for old-fashioned C, but you could probably adapt them. -Chris On Tue, Oct 14, 2014 at 8:59 AM, Toby St Clere Smithe m...@tsmithe.net wrote: John Zwinck jzwi...@gmail.com writes: Some time ago I needed to do something similar. I fused the NumPy C API and Boost.Python with a small bit of code which I then open-sourced as part of a slightly larger library. The most relevant part for you is here: https://github.com/jzwinck/pccl/blob/master/NumPyArray.hpp There is also a 'Boost.NumPy', which is quite nice, though perhaps a bit heavy-duty: https://github.com/ndarray/Boost.NumPy/ Cheers, Toby ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Best way to expose std::vector to be used with numpy
On 15.10.2014 21:48, Chris Barker wrote: Sorry about SWIG -- maybe a chance to move on ;-) I'd go with Cython -- this is pretty straightforward, and it handles the buffer protocol for you under the hood. +1 All the standard containers are automatically wrapped and C++ exceptions can be caught and translated to Python ones. Armando ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Best way to expose std::vector to be used with numpy
On Mon, Oct 13, 2014 at 8:39 PM, Charles R Harris charlesr.har...@gmail.com wrote: On Mon, Oct 13, 2014 at 12:54 PM, Sebastian Berg sebast...@sipsolutions.net wrote: On Mo, 2014-10-13 at 13:35 +0200, Daniele Nicolodi wrote: Hello, I have a C++ application that collects float, int or complex data in a possibly quite large std::vector. The application has some SWIG generated python wrappers that expose this vector to python. However, the standard way in which SWIG exposes the data is to create a touple and pass this to python, where it is very often converted to a numpy array for processing. Of course this is not efficient. I would like therefore to generate a smarter python interface. In python 3 I would without doubt go for implementing the buffer protocol, which enables seamless interfacing with numpy. However, I need to support also python 2 and there the buffer protocol is not as nice. Isn't the new buffer protocol in python 2.6 or 2.7 already? There is at least a memoryview object in python 2, which maybe could be used to the same effect? No memoryview in python2.6, but the older buffer protocol it there. Is Boost Python an option? snip Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion Here's an idea on how to avoid the buffer protocol entirely. It won't be perfectly optimal, but it is a simple solution that may be good enough. It could be that the shape and type inference and the Python loops that are run when converting the large tuple to a NumPy array are the main sticking points. In that case, the primary bottleneck could be eliminated by looping and copying at the C level. You could use Cython and just copy the needed information manually into the array. Here's a rough outline of what the Cython file might look like: # distutils: language = c++ # distutils: libraries = (whatever shared library you need) # distutils: library_dirs = (folders where the libraries are) # distutils: include_dirs = (wherever the headers are) # cython: boundscheck = False # cython: wraparound = False import numpy as np from libcpp.vector cimport vector # Here do the call to your C++ library # This can be done by declaring the # functions or whatever else you need in # a cdef extern from header file block # and then calling the functions to get # the vector you need. cdef int[::1] arr_from_vector(vector[int] v): cdef: int i int[::1] a = np.empty(v.size(), int) for i in xrange(v.size()): a[i] = v[i] return np.asarray(a) def get_data(): # Run your C++ computation here. return arr_from_vector(v) ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Best way to expose std::vector to be used with numpy
On Oct 14, 2014 4:40 AM, Charles R Harris charlesr.har...@gmail.com wrote: On Mon, Oct 13, 2014 at 12:54 PM, Sebastian Berg sebast...@sipsolutions.net wrote: On Mo, 2014-10-13 at 13:35 +0200, Daniele Nicolodi wrote: Hello, I have a C++ application that collects float, int or complex data in a possibly quite large std::vector. The application has some SWIG generated python wrappers that expose this vector to python. However, the standard way in which SWIG exposes the data is to create a touple and pass this to python, where it is very often converted to a numpy array for processing. Of course this is not efficient. I would like therefore to generate a smarter python interface. In python 3 I would without doubt go for implementing the buffer protocol, which enables seamless interfacing with numpy. However, I need to support also python 2 and there the buffer protocol is not as nice. Isn't the new buffer protocol in python 2.6 or 2.7 already? There is at least a memoryview object in python 2, which maybe could be used to the same effect? No memoryview in python2.6, but the older buffer protocol it there. Is Boost Python an option? Boost.Numpy may be a useful tool to use here. Stefan snip Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Best way to expose std::vector to be used with numpy
On 14/10/14 04:39, Charles R Harris wrote: On Mon, Oct 13, 2014 at 12:54 PM, Sebastian Berg sebast...@sipsolutions.net mailto:sebast...@sipsolutions.net wrote: On Mo, 2014-10-13 at 13:35 +0200, Daniele Nicolodi wrote: Hello, I have a C++ application that collects float, int or complex data in a possibly quite large std::vector. The application has some SWIG generated python wrappers that expose this vector to python. However, the standard way in which SWIG exposes the data is to create a touple and pass this to python, where it is very often converted to a numpy array for processing. Of course this is not efficient. I would like therefore to generate a smarter python interface. In python 3 I would without doubt go for implementing the buffer protocol, which enables seamless interfacing with numpy. However, I need to support also python 2 and there the buffer protocol is not as nice. Isn't the new buffer protocol in python 2.6 or 2.7 already? There is at least a memoryview object in python 2, which maybe could be used to the same effect? No memoryview in python2.6, but the older buffer protocol it there. Is Boost Python an option? The old buffer protocol is an option, but it is much less nice than the new one, as it requires to use numpy.frombuffer() with an exlicit dtype instead of the siumpler numpy.asarray() sufficient in python 3. Boost Python may be an option as the codebase already depends on Boost, but probably not yet on Boost Python. Can you point me to the relevant documentation, and maybe to an example? One of the problems I have is that the current wrapping is done auto-magically with SWIG and I would like to deviate the less possible from that patter. Thank you! Cheers, Daniele ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Best way to expose std::vector to be used with numpy
If the goal is to have something that works kind of like the new buffer protocol but with a wider variety of python versions, then you might find the old array interface useful: http://docs.scipy.org/doc/numpy/reference/arrays.interface.html I always get confused by the history here but I believe that that's the numpy-only interface that later got cleaned up and generalized to become the new buffer interface. Numpy itself still supports it. -n On 14 Oct 2014 11:19, Daniele Nicolodi dani...@grinta.net wrote: On 14/10/14 04:39, Charles R Harris wrote: On Mon, Oct 13, 2014 at 12:54 PM, Sebastian Berg sebast...@sipsolutions.net mailto:sebast...@sipsolutions.net wrote: On Mo, 2014-10-13 at 13:35 +0200, Daniele Nicolodi wrote: Hello, I have a C++ application that collects float, int or complex data in a possibly quite large std::vector. The application has some SWIG generated python wrappers that expose this vector to python. However, the standard way in which SWIG exposes the data is to create a touple and pass this to python, where it is very often converted to a numpy array for processing. Of course this is not efficient. I would like therefore to generate a smarter python interface. In python 3 I would without doubt go for implementing the buffer protocol, which enables seamless interfacing with numpy. However, I need to support also python 2 and there the buffer protocol is not as nice. Isn't the new buffer protocol in python 2.6 or 2.7 already? There is at least a memoryview object in python 2, which maybe could be used to the same effect? No memoryview in python2.6, but the older buffer protocol it there. Is Boost Python an option? The old buffer protocol is an option, but it is much less nice than the new one, as it requires to use numpy.frombuffer() with an exlicit dtype instead of the siumpler numpy.asarray() sufficient in python 3. Boost Python may be an option as the codebase already depends on Boost, but probably not yet on Boost Python. Can you point me to the relevant documentation, and maybe to an example? One of the problems I have is that the current wrapping is done auto-magically with SWIG and I would like to deviate the less possible from that patter. Thank you! Cheers, Daniele ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Best way to expose std::vector to be used with numpy
On 14/10/14 13:39, Nathaniel Smith wrote: If the goal is to have something that works kind of like the new buffer protocol but with a wider variety of python versions, then you might find the old array interface useful: http://docs.scipy.org/doc/numpy/reference/arrays.interface.html I always get confused by the history here but I believe that that's the numpy-only interface that later got cleaned up and generalized to become the new buffer interface. Numpy itself still supports it. Hello Nathaniel, thanks for the pointer, that's what I need. However, reading the documentation you linked, it looks like the new buffer protocol is also available in Python 2.6, and I don't have the need to support older versions, and this is confirmed by the python documentation: https://docs.python.org/2.7/c-api/buffer.html So the issue is just on how to tell SWIG to implement this interface in the wrappers it generates. Does anyone have a pointer to some relevant documentation or examples? Thanks! Cheers, Daniele ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Best way to expose std::vector to be used with numpy
On Tue, Oct 14, 2014 at 6:19 PM, Daniele Nicolodi dani...@grinta.net wrote: On Mo, 2014-10-13 at 13:35 +0200, Daniele Nicolodi wrote: I have a C++ application that collects float, int or complex data in a possibly quite large std::vector. The application has some SWIG generated python wrappers that expose this vector to python. However, the standard way in which SWIG exposes the data is to create a touple and pass this to python, where it is very often converted to a numpy array for processing. Of course this is not efficient. Boost Python may be an option as the codebase already depends on Boost, but probably not yet on Boost Python. Can you point me to the relevant documentation, and maybe to an example? One of the problems I have is that the current wrapping is done auto-magically with SWIG and I would like to deviate the less possible from that patter. Some time ago I needed to do something similar. I fused the NumPy C API and Boost.Python with a small bit of code which I then open-sourced as part of a slightly larger library. The most relevant part for you is here: https://github.com/jzwinck/pccl/blob/master/NumPyArray.hpp In particular, it offers this function: // use already-allocated storage for the array // (it will not be initialized, and its format must match the given dtype) boost::python::object makeNumPyArrayWithData( boost::python::list const dtype, unsigned count, void* data); What is dtype? For that you can use another small widget in my library: https://github.com/jzwinck/pccl/blob/master/NumPyDataType.hpp So the outline for your use case: you have a C array or C++ vector of contiguous plain old data. You create a NumPyDataType(), call append() on it with your data type, then pass all of the above to makeNumPyArrayWithData(). The result is a boost::python::object which you can return from C++ to Python with zero copies made of your data. Even if you don't decide to use Boost.Python, maybe you will find the implementation instructive. The functions I described take only a few lines. John Zwinck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Best way to expose std::vector to be used with numpy
On 14/10/14 14:11, Daniele Nicolodi wrote: On 14/10/14 13:39, Nathaniel Smith wrote: If the goal is to have something that works kind of like the new buffer protocol but with a wider variety of python versions, then you might find the old array interface useful: http://docs.scipy.org/doc/numpy/reference/arrays.interface.html I always get confused by the history here but I believe that that's the numpy-only interface that later got cleaned up and generalized to become the new buffer interface. Numpy itself still supports it. Hello Nathaniel, thanks for the pointer, that's what I need. However, reading the documentation you linked, it looks like the new buffer protocol is also available in Python 2.6, and I don't have the need to support older versions, and this is confirmed by the python documentation: https://docs.python.org/2.7/c-api/buffer.html I found one more problem: one of the data types I would like to expose is a complex float array equivalent to numpy.complex64 dtype. However, there is not such type format string defined in the python struct module, Experimenting with numpy it seems like it extends the struct format string definition with a new modifier 'Z' for complex: a = np.ones(10, dtype=np.complex64) m = memoryview(a) m.format 'Zf' How standard is that? Should I do the same in my extension? Thank you. Cheers, Daniele ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Best way to expose std::vector to be used with numpy
John Zwinck jzwi...@gmail.com writes: Some time ago I needed to do something similar. I fused the NumPy C API and Boost.Python with a small bit of code which I then open-sourced as part of a slightly larger library. The most relevant part for you is here: https://github.com/jzwinck/pccl/blob/master/NumPyArray.hpp There is also a 'Boost.NumPy', which is quite nice, though perhaps a bit heavy-duty: https://github.com/ndarray/Boost.NumPy/ Cheers, Toby ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Best way to expose std::vector to be used with numpy
Hello, I have a C++ application that collects float, int or complex data in a possibly quite large std::vector. The application has some SWIG generated python wrappers that expose this vector to python. However, the standard way in which SWIG exposes the data is to create a touple and pass this to python, where it is very often converted to a numpy array for processing. Of course this is not efficient. I would like therefore to generate a smarter python interface. In python 3 I would without doubt go for implementing the buffer protocol, which enables seamless interfacing with numpy. However, I need to support also python 2 and there the buffer protocol is not as nice. What is the best thing to do to expose the data buffer to python 2, without introducing a dependency on numpy in the application? Bonus points if the proposed solution comes with a pointer on how to implement in in SWIG (which I don't know much and feel a bit too magic for my taste). Thank you! Cheers, Daniele ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Best way to expose std::vector to be used with numpy
On Mo, 2014-10-13 at 13:35 +0200, Daniele Nicolodi wrote: Hello, I have a C++ application that collects float, int or complex data in a possibly quite large std::vector. The application has some SWIG generated python wrappers that expose this vector to python. However, the standard way in which SWIG exposes the data is to create a touple and pass this to python, where it is very often converted to a numpy array for processing. Of course this is not efficient. I would like therefore to generate a smarter python interface. In python 3 I would without doubt go for implementing the buffer protocol, which enables seamless interfacing with numpy. However, I need to support also python 2 and there the buffer protocol is not as nice. Isn't the new buffer protocol in python 2.6 or 2.7 already? There is at least a memoryview object in python 2, which maybe could be used to the same effect? - Sebastian What is the best thing to do to expose the data buffer to python 2, without introducing a dependency on numpy in the application? Bonus points if the proposed solution comes with a pointer on how to implement in in SWIG (which I don't know much and feel a bit too magic for my taste). Thank you! Cheers, Daniele ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion signature.asc Description: This is a digitally signed message part ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Best way to expose std::vector to be used with numpy
On Mon, Oct 13, 2014 at 12:54 PM, Sebastian Berg sebast...@sipsolutions.net wrote: On Mo, 2014-10-13 at 13:35 +0200, Daniele Nicolodi wrote: Hello, I have a C++ application that collects float, int or complex data in a possibly quite large std::vector. The application has some SWIG generated python wrappers that expose this vector to python. However, the standard way in which SWIG exposes the data is to create a touple and pass this to python, where it is very often converted to a numpy array for processing. Of course this is not efficient. I would like therefore to generate a smarter python interface. In python 3 I would without doubt go for implementing the buffer protocol, which enables seamless interfacing with numpy. However, I need to support also python 2 and there the buffer protocol is not as nice. Isn't the new buffer protocol in python 2.6 or 2.7 already? There is at least a memoryview object in python 2, which maybe could be used to the same effect? No memoryview in python2.6, but the older buffer protocol it there. Is Boost Python an option? snip Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
