Re: [C++-sig] RAII for the GIL in Boost.Python?
On 16/12/2012 22:58, Adam Preble wrote: Is the particular focus of your code on getting the GIL released during longer method invocations? I don't have any sway with Boost or anything, so I'm just asking out of my own personal curiosities. When I first saw the message, without seeing the code, I was wondering if you might have coincidentally created a scope-lock kind of thing for acquiring the GIL in the first place. Yes, the main point is to provide a reasonably safe, RAII-style mechanism for releasing the GIL around long-running C++ functions. But in addition to a class for releasing (and later reacquiring, hence RAII), I made a class for acquiring the GIL (if it is not held) and releasing it at the end. This allows nesting: complex C++ computations can be done with the GIL released, yet some bits of C++ can call back into Python by using the second class to acquire the GIL when needed. You needn't have sway with Boost to help here--simply reviewing the code and letting us know if it would help you would be great. ___ Cplusplus-sig mailing list Cplusplus-sig@python.org http://mail.python.org/mailman/listinfo/cplusplus-sig
Re: [C++-sig] RAII for the GIL in Boost.Python?
On 17/12/2012 03:25, Niall Douglas wrote: I won't go into too much detail here (search this list's archives), but in short there isn't just GIL management but also interpreter management, and on top of that both of those have to work right as exception throws happen plus policies must be instituted for things like container iterators (e.g. do you release GIL each iteration, every 10 iterations etc) and policies for multiple interpreter interactions (e.g. do we timeslice interpreters, or wait for one to go to i/o sleep first?). The code I published is agnostic about iterators and iteration--the user must explicitly decide when the GIL is to be released and reacquired. I'm sure there could be fancier ways of doing it, e.g. with tags given to BPL's class_::def() for long-running methods. But I didn't worry about that level of complexity--I think the simple solution offers benefits today, and can be expanded upon later. Having multiple Python interpreters in one process seems fraught anyway, so I haven't really considered whether my code can (or should) support that. I hope you'll agree it is not the common case. ___ Cplusplus-sig mailing list Cplusplus-sig@python.org http://mail.python.org/mailman/listinfo/cplusplus-sig
Re: [C++-sig] RAII for the GIL in Boost.Python?
On 25 Dec 2012 at 12:14, John Zwinck wrote: > The code I published is agnostic about iterators and iteration--the user > must explicitly decide when the GIL is to be released and reacquired. > I'm sure there could be fancier ways of doing it, e.g. with tags given > to BPL's class_::def() for long-running methods. But I didn't worry > about that level of complexity--I think the simple solution offers > benefits today, and can be expanded upon later. > > Having multiple Python interpreters in one process seems fraught anyway, > so I haven't really considered whether my code can (or should) support > that. I hope you'll agree it is not the common case. Solving multiple interpreters is absolutely the same problem as solving GIL management correctly. You get multiple interpreters "for free". I also didn't mention the problem of multiple copies of BPL, which is a related issue. Right now, if python loads two BPL based modules, two separate copies of the BPL runtime are loaded. The problem is that there are now two BPL registries, so you start encountering "Unknown type" exceptions when the wrong registry gets looked up. The traditional solution is to hack the flags passed to dlopen() by python to use RTLD_GLOBAL. Now you get one single type registry, and things start to work again. But now you have the problem of symbol collision due to the ODR rule, so if extension A defines type Foo and extension B defines a different type Foo, they collide. Basically, you get a runtime warning when you try to load the second extension, and it segfaults or memory corrupts during usage. What BPL needs is to do execution stack aware type registry lookups, so you need to examine from which extension the lookup is coming from and search the type registry appropriately. Thankfully, metaprogramming makes this uber easy (just slip in a pointer to any local function, and have a O(log N) routine map it to its containing module, then cache that so lookup henceforth is instant). As I mentioned earlier, BPL basically needs to be refactored and rewritten pretty much from scratch. The wider solution I'm currently prototyping - if it gets greenlit - solves all these problems, and a ton load more I haven't mentioned here (e.g. what happens if the two BPLs are different versions, and are slightly binary incompatible?). Chances are though it will be rejected sadly. And even if greenlit, we're talking 2014 at the earliest. So, for now, you may be right that submitting a toy solution now in the hope its limitations spur people to get it fixed properly may be the better approach. Certainly our collective hesitation so far hasn't achieved much. So sure, press on. Niall -- Any opinions or advice expressed here do NOT reflect those of my employer Research In Motion Inc. Work Portfolio: http://careers.stackoverflow.com/nialldouglas/ SMime.p7s Description: S/MIME cryptographic signature ___ Cplusplus-sig mailing list Cplusplus-sig@python.org http://mail.python.org/mailman/listinfo/cplusplus-sig
Re: [C++-sig] How to converter std::string* in boost.python?
For a BPL wrapped object to be a python string, it simply needs to provide the same API as a Python string. Replicate the magic underscore methods of python's string object and your wrapped std::string becomes a python string. It used to be the case that inheriting your BPL object from python's string was problematic. I'd assume recent Pythons have fixed that. Niall On 25 Dec 2012 at 14:22, simon zhang wrote: > If Jim is right,string's memory allocated is different in python and > c++.However,before python > handle some strings from c++,it must convert string from c++'s memory to > python's memory. > > So smart pointers have not changed anything.It must keep a separate copy of > a string in c++ memory and python memory. > > My idea.In the future, I will be write a wrapped class. When the c++ > interface, it store the data by std :: string.When the python > interface,it store the data by the Python str object. And perhaps this is > a good way. > > > 2012/12/25 Niall Douglas > > > Traditionally, the proper solution to avoid deep copies is to wrap a > > std::shared_ptr instead of a std::string directly. > > > > If your strings are short of course, it may well be faster to leave > > it as is. std::shared_ptr is not lightweight and may use atomic > > instructions, the bandwidth for which in a system is always limited. > > > > Niall > > > > On 23 Dec 2012 at 16:16, simon zhang wrote: > > > > > ..so,It seems I can only convert std::string and make a deep copy.I can > > > only avoid it as much as possible.Thank you. > > > > > > > > > 2012/12/22 Jim Bosch > > > > > > > On 12/21/2012 03:52 AM, simon zhang wrote: > > > > > > > >> How to converter std::string* in boost.python?I have to handle some > > data > > > >> of c++ in python.The data may be big.So I return a pointer to python. > > > >> But there are some errors. > > > >> > > > >> > > > > If the data is big, and you really want to avoid a deep copy, the only > > way > > > > to use it is if you manually allocate the memory as a Python str object > > > > using the Python C API or return it as something else (like a NumPy > > array > > > > or buffer object, or a custom Boost.Python-wrapped class) that can hold > > > > C++-allocated memory. A Python str object always owns its own memory, > > and > > > > so does std::string, so you can't convert from one to the other > > without a > > > > deep copy. > > > > > > > > Jim > > > > > > > > __**_ > > > > Cplusplus-sig mailing list > > > > Cplusplus-sig@python.org > > > > http://mail.python.org/**mailman/listinfo/cplusplus-sig< > > http://mail.python.org/mailman/listinfo/cplusplus-sig> > > > > > > > > > > > > > -- > > Any opinions or advice expressed here do NOT reflect those > > of my employer Research In Motion Inc. > > Work Portfolio: http://careers.stackoverflow.com/nialldouglas/ > > > > > > > > > > ___ > > Cplusplus-sig mailing list > > Cplusplus-sig@python.org > > http://mail.python.org/mailman/listinfo/cplusplus-sig > > > -- Any opinions or advice expressed here do NOT reflect those of my employer Research In Motion Inc. Work Portfolio: http://careers.stackoverflow.com/nialldouglas/ SMime.p7s Description: S/MIME cryptographic signature ___ Cplusplus-sig mailing list Cplusplus-sig@python.org http://mail.python.org/mailman/listinfo/cplusplus-sig
