Re: [C++-sig] [boost python] : how to pass a tuple of lists from python to C++
Finally able to reproduce the "real" problem with a "dummy" example : seems
that, at python side, when you use "np.append" C++ get screwed data ?!...
(note that with or without "np.append" all is OK at python side).
Can somebody help here ? Known problem ? Possible workaround or fix ?
Should I open a bug for that ? (if yes where)
Franck
>> make; python dummyTest.py ; python dummyTest2.py
g++ -I/usr/include/python2.7 -o dummy.so -shared -fPIC dummy.cpp
-lboost_python -lboost_numpy
[1 2 3]
[1.2 2.3 3.4]
ptrInt[0] = 1, 0x5643649d5640
ptrInt[1] = 2, 0x5643649d5644
ptrInt[2] = 3, 0x5643649d5648
ptrFloat[0] = 1.2, 0x5643649d5660
ptrFloat[1] = 2.3, 0x5643649d5664
ptrFloat[2] = 3.4, 0x5643649d5668
[1 2 3]
[1.2 2.3 3.4]
ptrInt[0] = 1, 0x559e0710e550
ptrInt[1] = 0, 0x559e0710e554
ptrInt[2] = 2, 0x559e0710e558
ptrFloat[0] = 4.17233e-08, 0x559e0710e570
ptrFloat[1] = 1.9, 0x559e0710e574
ptrFloat[2] = 2.72008e+23, 0x559e0710e578
Le dim. 9 févr. 2020 à 10:58, HOUSSEN Franck a écrit :
> Oh men ! Totally missed / forget that... Thanks Sefan : the dummy simple
> example works now :D
> Unfortunately, this dummy example was extracted from a much more complex
> one which seems to do things like dummy but still does not work... so I am
> still debugging the real case...
> Anyway, thanks for the help on the dummy case ! Hope this could help some
> other people
>
> Franck
>
> Le dim. 9 févr. 2020 à 05:04, Stefan Seefeld a
> écrit :
>
>>
>> On 2020-02-08 12:07 p.m., HOUSSEN Franck wrote:
>>
>> I tried to play with extract::check() without much success : line 13
>> crashes !?...
>> This is a very simple exemple : would like to get it to work. Could
>> somebody help ? Still googling / searching for a solution
>>
>> You need to call np::initialize() in your module, i.e. insert
>>
>> `np::initialize();`
>>
>> at the top of your `dummy` module definition.
>>
>> [image: Stefan]
>> --
>>
>> ...ich hab' noch einen Koffer in Berlin...
>>
>>
>
>
Makefile
Description: Binary data
#include
#include
#include
namespace bp = boost::python;
namespace np = boost::python::numpy;
void doStuffs(boost::python::tuple & t) {
boost::python::extract lsIntExt(t[0]);
if (lsIntExt.check()) {
np::ndarray lsInt = lsIntExt(); // Conversion to np::ndarray.
int nbInt = lsInt.shape(0);
int * ptrInt = reinterpret_cast(lsInt.get_data());
for(auto k = 0; ptrInt && k < nbInt; k++) {std::cout << "ptrInt[" << k << "] = " << ptrInt[k] << ", " << ptrInt+k << std::endl;};
}
boost::python::extract lsFloatExt(t[1]);
if (lsFloatExt.check()) {
np::ndarray lsFloat = lsFloatExt(); // Conversion to np::ndarray.
int nbFloat = lsFloat.shape(0);
float * ptrFloat = reinterpret_cast(lsFloat.get_data());
for(auto k = 0; k < nbFloat; k++) {std::cout << "ptrFloat[" << k << "] = " << ptrFloat[k] << ", " << ptrFloat+k << std::endl;};
}
}
BOOST_PYTHON_MODULE(dummy) {
np::initialize();
def("doStuffs", doStuffs);
}
#!/usr/bin/env python
import numpy as np
lsInt = np.array([], dtype='int32')
lsInt = np.append(lsInt, 1)
lsInt = np.append(lsInt, 2)
lsInt = np.append(lsInt, 3)
lsFloat = np.array([], dtype='float32')
lsFloat = np.append(lsFloat, 1.2)
lsFloat = np.append(lsFloat, 2.3)
lsFloat = np.append(lsFloat, 3.4)
t = (lsInt, lsFloat)
import dummy
print(lsInt)
print(lsFloat)
dummy.doStuffs(t)
#!/usr/bin/env python
import numpy as np
lsInt = np.array([1, 2, 3], dtype='int32')
lsFloat = np.array([1.2, 2.3, 3.4], dtype='float32')
t = (lsInt, lsFloat)
import dummy
print(lsInt)
print(lsFloat)
dummy.doStuffs(t)
___
Cplusplus-sig mailing list
[email protected]
https://mail.python.org/mailman/listinfo/cplusplus-sig
Re: [C++-sig] [boost python] : how to pass a tuple of lists from python to C++
> On Feb 11, 2020, at 13:24, HOUSSEN Franck wrote: > > Finally able to reproduce the "real" problem with a "dummy" example : seems > that, at python side, when you use "np.append" C++ get screwed data ?!... > (note that with or without "np.append" all is OK at python side). > Can somebody help here ? Known problem ? Possible workaround or fix ? Should > I open a bug for that ? (if yes where) This is a side-effect of how np.append() broadcasts its arguments to find a common type. You've asked it to append a Python float (which is secretly a double) to an array of np.float32. The only safe way to do that is to cast both to numpy.float64, which np.append() happily does for you. Before you reinterpret_cast the data pointer of an ndarray to a given type, you need to check whether the dtype is compatible (and also, strictly speaking, whether the flags are CARRAY or CARRAY_RO). There's a whole lot more background information in the Numpy C API docs: https://docs.scipy.org/doc/numpy/reference/c-api.html Cheers, Jakob ___ Cplusplus-sig mailing list [email protected] https://mail.python.org/mailman/listinfo/cplusplus-sig
Re: [C++-sig] [boost python] : how to pass a tuple of lists from python to C++
OK, I understand this is a type related problem. I found a workaround (dummyTest3.py - init numpy arrays with list seems to work). But, I unfortunately do not get how to change the code to get dummyTest2.py to work ?!... Should I read reinterpret_cast'ed data sizeof(double) by sizeof(double) : seems cryptic (?!), what's the natural way to do that ? Also a question raises : if the size of the data type you want to pass from python to C++ is smaller than sizeof(double), python will always allocate MORE memory than needed ? Why is that ? In case the size is large, the extra memory allocated may be large and trigger problems like slowdown (swapping) or out-of-memory, no ?... Why is this like so ? Can somebody help to fix the dummy code : this would be the best explanation !... :D Franck Le mar. 11 févr. 2020 à 14:57, Jakob van Santen a écrit : > > > On Feb 11, 2020, at 13:24, HOUSSEN Franck wrote: > > > > Finally able to reproduce the "real" problem with a "dummy" example : > seems that, at python side, when you use "np.append" C++ get screwed data > ?!... (note that with or without "np.append" all is OK at python side). > > Can somebody help here ? Known problem ? Possible workaround or fix ? > Should I open a bug for that ? (if yes where) > > This is a side-effect of how np.append() broadcasts its arguments to find > a common type. You've asked it to append a Python float (which is secretly > a double) to an array of np.float32. The only safe way to do that is to > cast both to numpy.float64, which np.append() happily does for you. Before > you reinterpret_cast the data pointer of an ndarray to a given type, you > need to check whether the dtype is compatible (and also, strictly speaking, > whether the flags are CARRAY or CARRAY_RO). > > There's a whole lot more background information in the Numpy C API docs: > https://docs.scipy.org/doc/numpy/reference/c-api.html > > Cheers, > Jakob > ___ > Cplusplus-sig mailing list > [email protected] > https://mail.python.org/mailman/listinfo/cplusplus-sig > #include #include #include namespace bp = boost::python; namespace np = boost::python::numpy; void doStuffs(boost::python::tuple & t) { boost::python::extract lsIntExt(t[0]); if (lsIntExt.check()) { np::ndarray lsInt = lsIntExt(); // Conversion to np::ndarray. int nbInt = lsInt.shape(0); int * ptrInt = reinterpret_cast(lsInt.get_data()); for(auto k = 0; ptrInt && k < nbInt; k++) {std::cout << "ptrInt[" << k << "] = " << ptrInt[k] << ", " << ptrInt+k << std::endl;}; } boost::python::extract lsFloatExt(t[1]); if (lsFloatExt.check()) { np::ndarray lsFloat = lsFloatExt(); // Conversion to np::ndarray. int nbFloat = lsFloat.shape(0); float * ptrFloat = reinterpret_cast(lsFloat.get_data()); for(auto k = 0; k < nbFloat; k++) {std::cout << "ptrFloat[" << k << "] = " << ptrFloat[k] << ", " << ptrFloat+k << std::endl;}; } boost::python::extract lsDoubleExt(t[2]); if (lsDoubleExt.check()) { np::ndarray lsDouble = lsDoubleExt(); // Conversion to np::ndarray. int nbDouble = lsDouble.shape(0); double * ptrDouble = reinterpret_cast(lsDouble.get_data()); for(auto k = 0; k < nbDouble; k++) {std::cout << "ptrDouble[" << k << "] = " << ptrDouble[k] << ", " << ptrDouble+k << std::endl;}; } } BOOST_PYTHON_MODULE(dummy) { np::initialize(); def("doStuffs", doStuffs); } #!/usr/bin/env python import numpy as np ls = [1, 2, 3] lsInt = np.array(ls, dtype='int32') ls = [1.2, 2.3, 3.4] lsFloat = np.array(ls, dtype='float32') ls = [4.5, 5.6, 6.7] lsDouble = np.array(ls, dtype='float64') t = (lsInt, lsFloat, lsDouble) import dummy print(lsInt) print(lsFloat) print(lsDouble) dummy.doStuffs(t) #!/usr/bin/env python import numpy as np lsInt = np.array([1, 2, 3], dtype='int32') lsFloat = np.array([1.2, 2.3, 3.4], dtype='float32') lsDouble = np.array([4.5, 5.6, 6.7], dtype='float64') t = (lsInt, lsFloat, lsDouble) import dummy print(lsInt) print(lsFloat) print(lsDouble) dummy.doStuffs(t) Makefile Description: Binary data #!/usr/bin/env python import numpy as np lsInt = np.array([], dtype='int32') lsInt = np.append(lsInt, 1) lsInt = np.append(lsInt, 2) lsInt = np.append(lsInt, 3) lsFloat = np.array([], dtype='float32') lsFloat = np.append(lsFloat, 1.2) lsFloat = np.append(lsFloat, 2.3) lsFloat = np.append(lsFloat, 3.4) lsDouble = np.array([], dtype='float64') lsDouble = np.append(lsDouble, 4.5) lsDouble = np.append(lsDouble, 5.6) lsDouble = np.append(lsDouble, 6.7) t = (lsInt, lsFloat, lsDouble) import dummy print(lsInt) print(lsFloat) print(lsDouble) dummy.doStuffs(t) ___ Cplusplus-sig mailing list [email protected] https://mail.python.org/mailman/listinfo/cplusplus-sig
Re: [C++-sig] [boost python] : how to pass a tuple of lists from python to C++
On 2020-02-11 2:05 p.m., HOUSSEN Franck wrote: OK, I understand this is a type related problem. I found a workaround (dummyTest3.py - init numpy arrays with list seems to work). But, I unfortunately do not get how to change the code to get dummyTest2.py to work ?!... Should I read reinterpret_cast'ed data sizeof(double) by sizeof(double) : seems cryptic (?!), what's the natural way to do that ? Can't you change your Python code so instead of np.append(lsFloat, 1.2) you would call np.append(lsFloat, np.float32(1.2)) to make sure the casting happens before the broadcasting. (Note: I haven't actually tried that; it just seems the natural fix to the issue ;-) ) Stefan -- ...ich hab' noch einen Koffer in Berlin... ___ Cplusplus-sig mailing list [email protected] https://mail.python.org/mailman/listinfo/cplusplus-sig
Re: [C++-sig] [boost python] : how to pass a tuple of lists from python to C++
Works great ! Thanks ! Code attached. Franck >> make; python dummyTest1.py; python dummyTest2.py; python dummyTest3.py g++ -I/usr/include/python2.7 -o dummy.so -shared -fPIC dummy.cpp -lboost_python -lboost_numpy [1 2 3] [1.2 2.3 3.4] [4.5 5.6 6.7] ptrInt[0] = 1, 0x555a42aede80 ptrInt[1] = 2, 0x555a42aede84 ptrInt[2] = 3, 0x555a42aede88 ptrFloat[0] = 1.2, 0x555a429c4f60 ptrFloat[1] = 2.3, 0x555a429c4f64 ptrFloat[2] = 3.4, 0x555a429c4f68 ptrDouble[0] = 4.5, 0x555a429c4f80 ptrDouble[1] = 5.6, 0x555a429c4f88 ptrDouble[2] = 6.7, 0x555a429c4f90 [1 2 3] [1.2 2.3 3.4] [4.5 5.6 6.7] ptrInt[0] = 1, 0x55dbaa536470 ptrInt[1] = 2, 0x55dbaa536474 ptrInt[2] = 3, 0x55dbaa536478 ptrFloat[0] = 1.2, 0x55dbaa4b9bb0 ptrFloat[1] = 2.3, 0x55dbaa4b9bb4 ptrFloat[2] = 3.4, 0x55dbaa4b9bb8 ptrDouble[0] = 4.5, 0x55dbaa4b9bd0 ptrDouble[1] = 5.6, 0x55dbaa4b9bd8 ptrDouble[2] = 6.7, 0x55dbaa4b9be0 [1 2 3] [1.2 2.3 3.4] [4.5 5.6 6.7] ptrInt[0] = 1, 0x563725f958f0 ptrInt[1] = 2, 0x563725f958f4 ptrInt[2] = 3, 0x563725f958f8 ptrFloat[0] = 1.2, 0x563725e87a10 ptrFloat[1] = 2.3, 0x563725e87a14 ptrFloat[2] = 3.4, 0x563725e87a18 ptrDouble[0] = 4.5, 0x563725fc0de0 ptrDouble[1] = 5.6, 0x563725fc0de8 ptrDouble[2] = 6.7, 0x563725fc0df0 Le mar. 11 févr. 2020 à 20:12, stefan a écrit : > > On 2020-02-11 2:05 p.m., HOUSSEN Franck wrote: > > OK, I understand this is a type related problem. I found a workaround > (dummyTest3.py - init numpy arrays with list seems to work). > > But, I unfortunately do not get how to change the code to get > dummyTest2.py to work ?!... Should I read reinterpret_cast'ed data > sizeof(double) by sizeof(double) : seems cryptic (?!), what's the natural > way to do that ? > > Can't you change your Python code so instead of > > np.append(lsFloat, 1.2) > > you would call > > np.append(lsFloat, np.float32(1.2)) > > to make sure the casting happens before the broadcasting. > > (Note: I haven't actually tried that; it just seems the natural fix to the > issue ;-) ) > [image: Stefan] > > -- > > ...ich hab' noch einen Koffer in Berlin... > > > ___ > Cplusplus-sig mailing list > [email protected] > https://mail.python.org/mailman/listinfo/cplusplus-sig > #!/usr/bin/env python import numpy as np ls = [1, 2, 3] lsInt = np.array(ls, dtype='int32') ls = [1.2, 2.3, 3.4] lsFloat = np.array(ls, dtype='float32') ls = [4.5, 5.6, 6.7] lsDouble = np.array(ls, dtype='float64') t = (lsInt, lsFloat, lsDouble) import dummy print(lsInt) print(lsFloat) print(lsDouble) dummy.doStuffs(t) #!/usr/bin/env python import numpy as np lsInt = np.array([1, 2, 3], dtype='int32') lsFloat = np.array([1.2, 2.3, 3.4], dtype='float32') lsDouble = np.array([4.5, 5.6, 6.7], dtype='float64') t = (lsInt, lsFloat, lsDouble) import dummy print(lsInt) print(lsFloat) print(lsDouble) dummy.doStuffs(t) Makefile Description: Binary data #!/usr/bin/env python import numpy as np lsInt = np.array([], dtype='int32') lsInt = np.append(lsInt, np.int32(1)) lsInt = np.append(lsInt, np.int32(2)) lsInt = np.append(lsInt, np.int32(3)) lsFloat = np.array([], dtype='float32') lsFloat = np.append(lsFloat, np.float32(1.2)) lsFloat = np.append(lsFloat, np.float32(2.3)) lsFloat = np.append(lsFloat, np.float32(3.4)) lsDouble = np.array([], dtype='float64') lsDouble = np.append(lsDouble, np.float64(4.5)) lsDouble = np.append(lsDouble, np.float64(5.6)) lsDouble = np.append(lsDouble, np.float64(6.7)) t = (lsInt, lsFloat, lsDouble) import dummy print(lsInt) print(lsFloat) print(lsDouble) dummy.doStuffs(t) #include #include #include namespace bp = boost::python; namespace np = boost::python::numpy; void doStuffs(boost::python::tuple & t) { boost::python::extract lsIntExt(t[0]); if (lsIntExt.check()) { np::ndarray lsInt = lsIntExt(); // Conversion to np::ndarray. int nbInt = lsInt.shape(0); int * ptrInt = reinterpret_cast(lsInt.get_data()); for(auto k = 0; ptrInt && k < nbInt; k++) {std::cout << "ptrInt[" << k << "] = " << ptrInt[k] << ", " << ptrInt+k << std::endl;}; } boost::python::extract lsFloatExt(t[1]); if (lsFloatExt.check()) { np::ndarray lsFloat = lsFloatExt(); // Conversion to np::ndarray. int nbFloat = lsFloat.shape(0); float * ptrFloat = reinterpret_cast(lsFloat.get_data()); for(auto k = 0; k < nbFloat; k++) {std::cout << "ptrFloat[" << k << "] = " << ptrFloat[k] << ", " << ptrFloat+k << std::endl;}; } boost::python::extract lsDoubleExt(t[2]); if (lsDoubleExt.check()) { np::ndarray lsDouble = lsDoubleExt(); // Conversion to np::ndarray. int nbDouble = lsDouble.shape(0); double * ptrDouble = reinterpret_cast(lsDouble.get_data()); for(auto k = 0; k < nbDouble; k++) {std::cout << "ptrDouble[" << k << "] = " << ptrDouble[k] << ", " << ptrDouble+k << std::endl;}; } } BOOST_PYTHON_MODULE(dummy) { np::initialize(); def("doStuffs", doStuffs); } ___ Cplusplus-sig mailing list [email protected]
