[C++-sig] Problem with shared_ptr converter in boost 1.60

2016-01-12 Thread Carlos Roig 
Dear colleges,

I work in a project which makes use of the Boost.python module.

Las week we received feedback from some people that was having problems
after
updating boost from version 1.59 to 1.60.
Specifically the problem appears while trying to use a shared_ptr class.

Until now, we were registering our classes as follows:

class_, boost::noncopyable >("FooClass");

being Foo::Pointer defined as boost::shared_ptr

Which automatically created the wrapper for python. Since version 1.60, the
following error appears while trying to access that elements:

>>> for Foo in Mesh.GetFooPointers()
>>> print(Foo)

TypeError: No to_python (by-value) converter found for C++ type:
boost::shared_ptr

We have been "googling" for a while and we found a work-around which
consist in adding:

register_ptr_to_python();

This would be pretty annoying to do for all our code, as is quite big and
was
designed under the assumption of that boost::shared_ptr<> was automatically
handled by boost.python and there was no need for calling
"register_ptr_to_python", as specified here:
https://wiki.python.org/moin/boost.python/PointersAndSmartPointers

We haven't been able to identify any change that may be affecting this, so
my question is:
Is there any way in which we can restore the old behaviour?
Does this change is inteended?

If it helps, we have perfom tests using clang 3.7 and gcc 5.3, both with
--std=c++11 enabled under linux64 (Kubuntu 15.10, Ubuntu 15.04 and 15.10,
and Arch) using python 3.4.3 with identical results..

Best regards and thank you in advance,

Charlie.
___
Cplusplus-sig mailing list
[email protected]
https://mail.python.org/mailman/listinfo/cplusplus-sig

Re: [C++-sig] Passing memory allocated in C++ to Python

2016-01-12 Thread Tony Cappellini 
Stefan,


To: [email protected]
Subject: Re: [C++-sig] Passing memory allocated in C++ to Python
Message-ID: <[email protected]>
Content-Type: text/plain; charset=windows-1252

> Essentially- Python needs access to the return value from a malloc() call.

>>Really ? Why ? You say your C++ code needs to access the pointer(s).
>>Python has no notion of "pointer", but it will happily pass around
>>atever data you expose from C++.

Sorry about that Stefan, my post should have been clearer.

I know Python doesn't know about pointers.
I should have said "Python needs access to the memory pointed to by the
memory allocated with malloc()".


>>f you expose the above class together with the constructor and thei
>>'call_ioctl()' member function I think you have all you need.

In your something class, the data type returned from allocate_memory()
needs to be something that Python understands. Since that allocation
function (member) will be allocating 100s of MB of memory, how will this
memory map to a Python data type?

Is a bytearray better to use than a list (as far as performance is
concerned)? Is a list better to use than a string (as far as performance is
concerned). Are there other data types that should be considered?

The user will need to access that data as bytes, words, longs, and possibly
sequences.

I've tried using boost's extract, to convert the char * returned from
malloc() into a Python string,
and a Python list, but these result in compile errors.
___
Cplusplus-sig mailing list
[email protected]
https://mail.python.org/mailman/listinfo/cplusplus-sig

Re: [C++-sig] Passing memory allocated in C++ to Python

2016-01-12 Thread Stefan Seefeld 
On 12.01.2016 12:52, Tony Cappellini wrote:
>
> Stefan,
>
>
> To: [email protected] 
> Subject: Re: [C++-sig] Passing memory allocated in C++ to Python
> Message-ID: <[email protected]
> >
> Content-Type: text/plain; charset=windows-1252
>
> > Essentially- Python needs access to the return value from a malloc() call.
>
> >>Really ? Why ? You say your C++ code needs to access the pointer(s).
> >>Python has no notion of "pointer", but it will happily pass around
> >>atever data you expose from C++.
>
> Sorry about that Stefan, my post should have been clearer.
>
> I know Python doesn't know about pointers.
> I should have said "Python needs access to the memory pointed to by
> the memory allocated with malloc()".
>
>
> >>f you expose the above class together with the constructor and thei
> >>'call_ioctl()' member function I think you have all you need.
>
> In your something class, the data type returned from allocate_memory()
> needs to be something that Python understands. Since that allocation
> function (member) will be allocating 100s of MB of memory, how will
> this memory map to a Python data type?

Does it have to be a Python (native) data type ? Could you explain your
use-case a little more ?

>
> Is a bytearray better to use than a list (as far as performance is
> concerned)? Is a list better to use than a string (as far as
> performance is concerned). Are there other data types that should be
> considered?

It all depends on what you intend to do with the memory.

>
> The user will need to access that data as bytes, words, longs, and
> possibly sequences.
>
> I've tried using boost's extract, to convert the char * returned from
> malloc() into a Python string,
> and a Python list, but these result in compile errors.

That's because the extract<>() logic is meant to be used for the
inverse: to get access to C/C++ objects embedded into a Python object.
Once you reflect your C++ type "something" to Python, boost.python will
implicitly convert between the Python (wrapper) type and the C++
"something" type for all function calls.
However, sometimes you may still have a Python object, knowing that it
actually contains a "something". extract<> is meant to give access to
that (either by-value or by-reference, depending on your usage).

If you really need to expose the memory as a buffer to Python, you may
want to use one of the newer C APIs such as
|PyMemoryView_FromMemory
(https://docs.python.org/3/c-api/memoryview.html). Support for that
isn't built into boost.python yet, so you'll have to add some glue code
(e.g.
http://stackoverflow.com/questions/23064407/expose-c-buffer-as-python-3-bytes).
This may actually be worth including into the library. We just haven't
had anyone asking for such a feature - yet.
|
(I have used similar approaches in the past, where memory from a C++
library I wrote is bound to a NumPy array, so the data can be accessed
copy-free from other Python modules via the NumPy interface.)

Regards,
Stefan

-- 

  ...ich hab' noch einen Koffer in Berlin...

___
Cplusplus-sig mailing list
[email protected]
https://mail.python.org/mailman/listinfo/cplusplus-sig

Re: [C++-sig] Passing memory allocated in C++ to Python

2016-01-12 Thread Tony Cappellini 
> In your something class, the data type returned from allocate_memory()
> needs to be something that Python understands. Since that allocation
> function (member) will be allocating 100s of MB of memory, how will
> this memory map to a Python data type?

>>Does it have to be a Python (native) data type ? Could you explain your
>>use-case a little more ?

It needs to be a data type that can be allocated in C++ but accessible in
Python

>>Could you explain your use-case a little more ?
I have C++ code which is interfacing to a driver, from Python.
The user invokes a python function which calls my C++ code (via Boost). The
C++ code makes a C++ structure, allocates memory for the ioctl call,
puts a pointer to the allocated memory into the C++ structure (as well as
filling in other structure members), calls the ioctl and returns the
success/failure
results to Python. The user needs a way to get access to the allocated
memory, which was filled in by the ioctl call.

>>f you really need to expose the memory as a buffer to Python, you may want
to use one of the newer C APIs such as
I'm sure there are many ways to do this, but I've already chosen boost for
this project. Boost exists in order to
pass data back and forth between C++ & Python, there must be some way to do
this

The boost documentation is not very helpful, but I know many people have
shared data between Python & C++ for
a long time this

Thanks

On Tue, Jan 12, 2016 at 10:14 AM, Stefan Seefeld 
wrote:

> On 12.01.2016 12:52, Tony Cappellini wrote:
> >
> > Stefan,
> >
> >
> > To: [email protected] 
> > Subject: Re: [C++-sig] Passing memory allocated in C++ to Python
> > Message-ID: <[email protected]
> > >
> > Content-Type: text/plain; charset=windows-1252
> >
> > > Essentially- Python needs access to the return value from a malloc()
> call.
> >
> > >>Really ? Why ? You say your C++ code needs to access the pointer(s).
> > >>Python has no notion of "pointer", but it will happily pass around
> > >>atever data you expose from C++.
> >
> > Sorry about that Stefan, my post should have been clearer.
> >
> > I know Python doesn't know about pointers.
> > I should have said "Python needs access to the memory pointed to by
> > the memory allocated with malloc()".
> >
> >
> > >>f you expose the above class together with the constructor and thei
> > >>'call_ioctl()' member function I think you have all you need.
> >
> > In your something class, the data type returned from allocate_memory()
> > needs to be something that Python understands. Since that allocation
> > function (member) will be allocating 100s of MB of memory, how will
> > this memory map to a Python data type?
>
> Does it have to be a Python (native) data type ? Could you explain your
> use-case a little more ?
>
> >
> > Is a bytearray better to use than a list (as far as performance is
> > concerned)? Is a list better to use than a string (as far as
> > performance is concerned). Are there other data types that should be
> > considered?
>
> It all depends on what you intend to do with the memory.
>
> >
> > The user will need to access that data as bytes, words, longs, and
> > possibly sequences.
> >
> > I've tried using boost's extract, to convert the char * returned from
> > malloc() into a Python string,
> > and a Python list, but these result in compile errors.
>
> That's because the extract<>() logic is meant to be used for the
> inverse: to get access to C/C++ objects embedded into a Python object.
> Once you reflect your C++ type "something" to Python, boost.python will
> implicitly convert between the Python (wrapper) type and the C++
> "something" type for all function calls.
> However, sometimes you may still have a Python object, knowing that it
> actually contains a "something". extract<> is meant to give access to
> that (either by-value or by-reference, depending on your usage).
>
> If you really need to expose the memory as a buffer to Python, you may
> want to use one of the newer C APIs such as
> |PyMemoryView_FromMemory
> (https://docs.python.org/3/c-api/memoryview.html). Support for that
> isn't built into boost.python yet, so you'll have to add some glue code
> (e.g.
>
> http://stackoverflow.com/questions/23064407/expose-c-buffer-as-python-3-bytes
> ).
> This may actually be worth including into the library. We just haven't
> had anyone asking for such a feature - yet.
> |
> (I have used similar approaches in the past, where memory from a C++
> library I wrote is bound to a NumPy array, so the data can be accessed
> copy-free from other Python modules via the NumPy interface.)
>
> Regards,
> Stefan
>
> --
>
>   ...ich hab' noch einen Koffer in Berlin...
>
>
___
Cplusplus-sig mailing list
[email protected]
https://mail.python.org/mailman/listinfo/cplusplus-sig

Re: [C++-sig] Passing memory allocated in C++ to Python

2016-01-12 Thread Tony Cappellini 
>>(https://docs.python.org/3/c-api/memoryview.html).
I should have mentioned that I am using Python 2.7 (as part of a group
project- others are using Python 2.7 as well).

Python 3.x is out of the question at the moment, but the memory view looks
like an interesting idea.
___
Cplusplus-sig mailing list
[email protected]
https://mail.python.org/mailman/listinfo/cplusplus-sig

Re: [C++-sig] Passing memory allocated in C++ to Python

2016-01-12 Thread Stefan Seefeld 
On 12.01.2016 13:54, Tony Cappellini wrote:
>
> > In your something class, the data type returned from allocate_memory()
> > needs to be something that Python understands. Since that allocation
> > function (member) will be allocating 100s of MB of memory, how will
> > this memory map to a Python data type?
>
> >>Does it have to be a Python (native) data type ? Could you explain your
> >>use-case a little more ?
>
> It needs to be a data type that can be allocated in C++ but accessible
> in Python

"accessible" is too vague. The question is what interface you need,
whether an object-oriented one with methods to access and modify the
internal state, or one giving you unstructured access to the raw data.

>
> >>Could you explain your use-case a little more ?
> I have C++ code which is interfacing to a driver, from Python.
> The user invokes a python function which calls my C++ code (via
> Boost). The C++ code makes a C++ structure, allocates memory for the
> ioctl call,
> puts a pointer to the allocated memory into the C++ structure (as well
> as filling in other structure members), calls the ioctl and returns
> the success/failure
> results to Python. The user needs a way to get access to the allocated
> memory, which was filled in by the ioctl call.

Can you share your code, or at least a stripped-down version ? I think
at this point it would make more sense to go over some specifics.
Your description above doesn't sound very object-oriented. You are
calling a function that returns the success of the operation. What you
actually want is return an object that holds the state (including the
memory you are talking about), which you can then manipulate in
subsequent calls.

Even some pseudo-code would do, to discuss the general idea.

> I should have mentioned that I am using Python 2.7 (as part of a group
> project- others are using Python 2.7 as well).
>
> Python 3.x is out of the question at the moment, but the memory view
> looks like an interesting idea.

Fine, I just mentioned it to illustrate the idea. There are other APIs
to achieve the same.

Regards,
Stefan

-- 

  ...ich hab' noch einen Koffer in Berlin...

___
Cplusplus-sig mailing list
[email protected]
https://mail.python.org/mailman/listinfo/cplusplus-sig