[C++-sig] interfacing headle binary file library, raw pointer and buffer
Dear all,
I am trying to interface a binary file library using py++.
There is a function in the library like
int read(void* buffer, size_t len)
which is hard to wrap. It read a piece (size len) of the file, and store
it in buffer, and then return the buffer length (most likely equal to len).
If I do nothing special, the py++ generated module would be like:
,
| >>> s = ""
| >>> l = 1
| >>> binary_file.read(s, l)
|
| did not match C++ signature:
| (python str type could not match void*)
`
Here are what I have tried (with no success):
1. I found the keyword "opaque pointer" in boost.python and py++. My
knowledge is so limited that I couldn't figure out if my case is
_actually_ an opaque pointer problem, cuz all the documentation[1] for
py++ and boost.python are talking about opaque return pointer, not a
passed pointer. I poked around py++ to add some class policies (from the
example of gmp wrapper) resulting in no change in the py++ generated
boost.python code.
2. Then I tried to change the function to
int read(char* buffer, size_t len)
with type casting done internally in c++. Then
,
| >>> s = ""
| >>> l = 1
| >>> binary_file.read(s, l)
| 1
| (the function executed successfully, with no c++ signature type mismatch)
| >>> s
| Segment Fault (python crash)
`
maybe it is the methodology of python that not touching "low level" mem
management. I guess that's what making this problem non-trivial.
I may recompile my python with debugging information an trace it with
gdb to find out why it is crashing. But not yet.
3. I tried ctypes:
,
| >>> s = ctypes.c_char_p
| >>> l = 1
| >>> binary_file.read(s, l)
| (c++ signature type mismatch)
`
I guess we just cannot mix ctypes with boost.python directly.
Now I need to learn and try something new to solve this issue. After
some web searching, I find some possiblities:
1. py++ ctypes integration[2]. It looks promising. I would like to
confirm on this list if it is suited to my case.
2. write a wrapper like
,
| struct void_p {
|void * ptr
| }
`
and then change all the original c++ interface from func(void * buf,
...) to func(void_p buf, ...) (there are around 6 such functions) and
take care of the internal convertion by hand. But in this case the
interface becomes thicker. I regard it as the last choice cuz I want to
keep new c++ code minimum (hopefully all auto generated by py++).
3. I came across struct module[3] in python. It seems to suit my
needs; I'm not sure. I couldn't figure out how to use it right now.
4. pyOgre might have been already solved this kind of raw memory
problem. I could learn pyOgre team's example and apply their method to
my case. There may be a sharp learning curve though.
... I am not sure which path I should take.
I am a beginner in c++ and python and c++/python integration. The
documentations and mailing list archives seems overwhelming to me. I
couldn't find a fresh point to start digging into.
Please give me some suggestion and advice. I am ready to learn and try
and practise.
Cheers!
Benda
footnotes:
1. http://www.boost.org/doc/libs/1_43_0/libs/python/doc/v2/opaque.html
http://language-binding.net/pyplusplus/documentation/functions/call_policies/call_policies.html
2.
http://language-binding.net/pyplusplus/documentation/ctypes/ctypes_integration.html
3. http://docs.python.org/library/struct.html
--
XU Benda
Research Center for Neutrino Science
Tohoku University
JAPAN
http://www.awa.tohoku.ac.jp/~benda
___
Cplusplus-sig mailing list
[email protected]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Re: [C++-sig] interfacing headle binary file library, raw pointer and buffer
Hello, I have just found this thread interesting http://thread.gmane.org/gmane.comp.python.c++/12912/focus=12913 the pointers are PyString_FromStringAndSize PyArg_ParseTuple* and http://docs.python.org/c-api/buffer.html How can those fit in py++ and boost.python? Cheers, -- XU Benda Research Center for Neutrino Science Tohoku University JAPAN http://www.awa.tohoku.ac.jp/~benda ___ Cplusplus-sig mailing list [email protected] http://mail.python.org/mailman/listinfo/cplusplus-sig
Re: [C++-sig] interfacing headle binary file library, raw pointer and buffer
Allo Nat,
Thank you very much for your reply. I didn't realized I could have this
flexibility with py++ generated code.
Now I could interface these functions.
Cheers!
Benda
Nat Goodspeed writes:
> You want to write a C++ wrapper function something like this:
>
> std::string wrapped_read(size_t len)
> {
> // manually allocate buffer of size 'len',
> // e.g. by using std::vector
> int bytes_read = read(buffer, len);
> return std::string(buffer, bytes_read);
--
XU Benda
Research Center for Neutrino Science
Tohoku University
JAPAN
http://www.awa.tohoku.ac.jp/~benda
___
Cplusplus-sig mailing list
[email protected]
http://mail.python.org/mailman/listinfo/cplusplus-sig
[C++-sig] numpy integration with boost.python
Dear all, This is a cross post of stackoverflow[0]. I am interfacing a C++ data intense library with Python by py++[1]/boost.python[2]. After profiling my program, I find 70% of run time is spent on code like this: ni = range(v2o.getHits()) tau = np.array([v2o.TofCorrectedTime[i] for i in ni]) q = np.array[v2o.getCharge()[i] for i in ni] v2o.TofCorrectedTime is typed __array_1_float_2368[3] from py++. v2o.getCharge() is typed _impl_details_range_iterator_[4] from py++, too. Size being about 2000, the convertion from these py++ array wrappers to numpy is slow: In [42]: timeit np.array(v2o.TofCorrectedTime) 100 loops, best of 3: 2.52 ms per loop In [43]: timeit np.array(v2o.getCharge()) 100 loops, best of 3: 4.94 ms per loop In [44]: timeit np.array([0]*2368) 1000 loops, best of 3: 310 µs per loop In [45]: timeit np.array(np.zeros(2368)) 10 loops, best of 3: 4.41 µs per loop I searched the web for a solution. The candidates are: 1. Cython[5] with memoryview[6] 2. pyublas[7] 3. Boost.NumPy[8] Questions: - Is cython/memoryview easy to be integrated with boost.python and py++? I want to keep the rest of the library wrapper. - Which one best suites my problem in terms of convertion overhead? Thanks, Benda [0]: http://stackoverflow.com/questions/20376021/exchange-numpy-array-with-boost-python-pyublas-or-boost-numpy [1]: http://svn.code.sf.net/p/pygccxml/svn/pyplusplus_dev [2]: http://www.boost.org/libs/python [3]: http://svn.code.sf.net/p/pygccxml/svn/pyplusplus_dev/pyplusplus/code_repository/array_1.py [4]: http://svn.code.sf.net/p/pygccxml/svn/pyplusplus_dev/pyplusplus/code_repository/return_range.py [5]: http://cython.org [6]: http://docs.cython.org/src/userguide/memoryviews.html [7]: http://mathema.tician.de/software/pyublas/ [8]: https://github.com/ndarray/Boost.NumPy ___ Cplusplus-sig mailing list [email protected] https://mail.python.org/mailman/listinfo/cplusplus-sig
Re: [C++-sig] numpy integration with boost.python
Dear all, [email protected] writes: > This is a cross post of stackoverflow[0]. > > [0]: > http://stackoverflow.com/questions/20376021/exchange-numpy-array-with-boost-python-pyublas-or-boost-numpy Feedback as a happy user. Jim has answered on stackoverflow. I decided to go with Boost.NumPy. And thanks Neal for sharing with your experience. The array data needs not to be copied anymore, giving a 2000-sized array exposing to Python 1000x faster than iteration- over-elements scheme. I had tens of such arrays to maintain. Too tedious to do it manually. So I made a patch against Py++ SVN trunk[1]. I think Roman is in this list. Could you please review it? Jim, thanks again! I hope boost.python v3 is going on well. Cheers, Benda 1. https://sourceforge.net/p/pygccxml/patches/2/ ___ Cplusplus-sig mailing list [email protected] https://mail.python.org/mailman/listinfo/cplusplus-sig
