[C++-sig] Efficient string passing
Dear all,
I recently sent an e-mail which (I think) didn't reach the list, about
trying to return large strings from C++ into Python without copying them.
By looking at the C/Python API I've subsequently figured out roughly how to
do this... something like
Class World
{
object return_object()
{
PyObject *result = PyString_FromStringAndSize(0, 7);
// initialise the underlying string using PyString_AsString
handle<> h(result);
return object();
}
...
main_namespace["World"] = class_("World")
.def("return_object", &World::return_object);
Unfortunately this doesn't work; when I call w.return_object(), I get back
None. Returning a raw PyObject * instead of a boost::python::object works,
but I can't tell if that involves a copy... and using
return_value_policy() with a PyObject * return crashes.
Any advice here would be very much appreciated!
Thank you and best wishes,
Mohan
___
Cplusplus-sig mailing list
[email protected]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Re: [C++-sig] Efficient string passing
On 11/19/2009 07:30 AM, Mohan Ganesalingam wrote:
Dear all,
I recently sent an e-mail which (I think) didn't reach the list, about
trying to return large strings from C++ into Python without copying them.
I believe this is impossible: Python treats strings as immutable, which
means you can't manipulate them in-place, and there is no C API function
that allows you to pass in a string by reference, as that would make
memory management impossibly complex.
By looking at the C/Python API I've subsequently figured out roughly
how to do this... something like
Class World
{
object return_object()
{
PyObject *result = PyString_FromStringAndSize(0, 7);
This creates a new string, copying whatever you pass as input.
// initialise the underlying string using PyString_AsString
handle<> h(result);
return object();
I'm not sure what your intent is, but returning a default-constructed
object is the C++ equivalent of None.
}
...
main_namespace["World"] = class_("World")
.def("return_object", &World::return_object);
Unfortunately this doesn't work; when I call w.return_object(), I get
back None.
...because above you "return object();", presumably.
Returning a raw PyObject * instead of a boost::python::object works,
but I can't tell if that involves a copy... and using
return_value_policy() with a PyObject * return
crashes.
Any advice here would be very much appreciated!
I don't think you can get what you want using strings directly. You may
create and expose a wrapper type that provides a string-like interface,
with appropriate conversions.
Stefan
--
...ich hab' noch einen Koffer in Berlin...
___
Cplusplus-sig mailing list
[email protected]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Re: [C++-sig] Efficient string passing
Thank you for the reply! As you spotted, I was making an embarrassingly idiotic typo (due to too much cut and pasting)... it should of course have been handle<> h(result); return object(h); Once that was fixed, everything worked fine. I believe this is impossible: Python treats strings as immutable, which means you can't manipulate them in-place, and there is no C API function There is apparently one exception to this, which I was utilising: char* PyString_AsString(PyObject *string) Returns a NUL-terminated representation of the contents of string. The pointer refers to the internal buffer of string, not a copy. The data must not be modified in any way, unless the string was just created using PyString_FromStringAndSize(NULL, size). I.e. if you create a string using PyString_FromStringAndSize(0, n), then you can immediately modify its contents using PyString_AsString. AFAICS this functionality is (reasonably enough) not wrapped by boost::python::str, so you have to use the underlying API. Thanks again for spotting that! Best wishes, Mohan ___ Cplusplus-sig mailing list [email protected] http://mail.python.org/mailman/listinfo/cplusplus-sig
Re: [C++-sig] Efficient string passing
On 11/19/2009 08:30 AM, Mohan Ganesalingam wrote: I believe this is impossible: Python treats strings as immutable, which means you can't manipulate them in-place, and there is no C API function There is apparently one exception to this, which I was utilising: char* PyString_AsString(PyObject *string) Returns a NUL-terminated representation of the contents of string. The pointer refers to the internal buffer of string, not a copy. The data must not be modified in any way, unless the string was just created using PyString_FromStringAndSize(NULL, size). I.e. if you create a string using PyString_FromStringAndSize(0, n), then you can immediately modify its contents using PyString_AsString. AFAICS this functionality is (reasonably enough) not wrapped by boost::python::str, so you have to use the underlying API. OK, perhaps I misunderstood your use-case. Indeed, you may allocate a string using the Python C API, and have a small window within which you can manipulate its content, before handing control back to the Python runtime. If that is what you need, great. For avoidance of doubt: I don't think it would be valid to modify that string after you have handed control back to Python. The Python runtime may well create hashes from it for efficient lookup, which build on the immutability of this type of object. Regards, Stefan -- ...ich hab' noch einen Koffer in Berlin... ___ Cplusplus-sig mailing list [email protected] http://mail.python.org/mailman/listinfo/cplusplus-sig
[C++-sig] Boost.Python + OpenGL segmentation faults
Hello everyone, I've recently posted a problem to SO (http://stackoverflow.com/questions/1751408/boost-python-opengl-segmentation-faults), where I haven't had much luck with answer. My problem is that if I import OpenGL python libraries before importing my B.P bindings, I get strange segmentation faults for no apparent reason. I isolated a small test case which I'm attaching below. The current boost library version I'm using is 1.37; you'll have to make a change to Makefile if you have any other version installed. I'd greatly appreciate if anyone can give some insight on what's happening. Dimitri. foo.cpp Description: Binary data glcrash.py Description: Binary data Makefile Description: Binary data ___ Cplusplus-sig mailing list [email protected] http://mail.python.org/mailman/listinfo/cplusplus-sig
Re: [C++-sig] Boost.Python + OpenGL segmentation faults
Dimitri Tcaciuc wrote: Hello everyone, I've recently posted a problem to SO (http://stackoverflow.com/questions/1751408/boost-python-opengl-segmentation-faults), where I haven't had much luck with answer. My problem is that if I import OpenGL python libraries before importing my B.P bindings, I get strange segmentation faults for no apparent reason. I isolated a small test case which I'm attaching below. The current boost library version I'm using is 1.37; you'll have to make a change to Makefile if you have any other version installed. I'd greatly appreciate if anyone can give some insight on what's happening. The only thing that strikes me is that I'd just use the vector_indexing_suite on the 'vector elements'. Beyond that, the next thing you'd want is a --with-pydebug --without-pymalloc build of python (with associated opengl and boost.python built against this library), get some stacktraces from running under gdb, and run things under valgrind with the valgrind-python.supp list from the Misc directory of the python distribution. It is a bit of work to set up but you get a lot more debuggability. -t ___ Cplusplus-sig mailing list [email protected] http://mail.python.org/mailman/listinfo/cplusplus-sig
Re: [C++-sig] Boost.Python + OpenGL segmentation faults
Just a remark: We are heavily using Boost.Python extensions together with OpenGL, and have so for years. It works for us without problems under Windows, Mac OS X, Linux. We are not using PyOpenGL, though, but have our own Boost.Python-based wrappers for the entire OpenGL library (http://cci.lbl.gov/gltbx/). Back in 2005/2006 all that worked even with PyOpenGL imported under Windows. I'd try running with valgrind to debug. - Original Message From: Dimitri Tcaciuc To: [email protected] Sent: Thu, November 19, 2009 1:54:51 PM Subject: [C++-sig] Boost.Python + OpenGL segmentation faults Hello everyone, I've recently posted a problem to SO (http://stackoverflow.com/questions/1751408/boost-python-opengl-segmentation-faults), where I haven't had much luck with answer. My problem is that if I import OpenGL python libraries before importing my B.P bindings, I get strange segmentation faults for no apparent reason. I isolated a small test case which I'm attaching below. The current boost library version I'm using is 1.37; you'll have to make a change to Makefile if you have any other version installed. I'd greatly appreciate if anyone can give some insight on what's happening. Dimitri. ___ Cplusplus-sig mailing list [email protected] http://mail.python.org/mailman/listinfo/cplusplus-sig
