[pypy-dev] swig + pypy - object reference counting
Hi! I have a quite sophisticated program that can be summarized as follows: 1. Save a pypy object pointer inside C program. Here I call Py_XINCREF so that it does not get deleted. 2. Do some logic, move this reference around C code. 3. Return a python tuple via typemap, here I am probably supposed to return a borrowed reference. And in Python2 it works just fine. BUT. In pypy, for some reason, it causes segfault with following message: """ Fatal error in cpyext, CPython compatibility layer, calling PyTuple_SetItem Either report a bug or consider not using this particular extension RPython traceback: File "module_cpyext_api_1.c", line 28965, in PyTuple_SetItem File "module_cpyext_pyobject.c", line 1018, in CpyTypedescr_realize Segmentation fault """ If I call Py_XINCREF before returning the object, the crash does not happen and the memory does not seem to be leaking (at least not noticeably massive amounts of it). So it seems that PyPy is somewhat incompatible with Python2 in that matter. If you want I could send the code example that triggers the bug (it IS quite large app, which might have many more bugs apart from this, but still). Thank you, Alex. ___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] swig + pypy - object reference counting
Well,
the point is that first I make an owned copy of the object:
%typemap(in) void* {
Py_XINCREF($input);
$1 = $input;
}
Here is the storage struct:
struct event{
int code;
void* node_tx;
void* node_rx;
void* packet;
double signal_power;
double noise_power;
double BER;
struct event* next;
};
For the C-code python objects are just void*, so they are perfectly safe.
Now, when I fetch the objects by my own get function i have the following
typemap:
%typemap (out) event_t* {
if ($1 == NULL)
{
$result = Py_None;
Py_XINCREF($result);
}
else
{
$result = PyTuple_New(7);
PyTuple_SetItem($result, 0 , PyInt_FromLong($1->code));
PyTuple_SetItem($result, 1 , $1->node_tx);
PyTuple_SetItem($result, 2 , $1->node_rx);
PyTuple_SetItem($result, 3 , $1->packet);
#ifdef PYTHON_PYPY
Py_XINCREF($1->node_tx);
Py_XINCREF($1->node_rx);
Py_XINCREF($1->packet);
#endif
PyTuple_SetItem($result, 4 , PyFloat_FromDouble($1->signal_power));
PyTuple_SetItem($result, 5 , PyFloat_FromDouble($1->noise_power));
PyTuple_SetItem($result, 6 , PyFloat_FromDouble($1->BER));
free($1);
Py_XINCREF($result);
}
}
As you can see, in Python I do not need to INCREF object references, but in
PYPY I do, otherwise it crashes.
In the wrapper function it looks like this:
SWIGINTERN PyObject *_wrap_fetch_event(PyObject *SWIGUNUSEDPARM(self),
PyObject *args) {
PyObject *resultobj = 0;
event_t *result = 0 ;
if (!SWIG_Python_UnpackTuple(args,"fetch_event",0,0,0)) SWIG_fail;
result = (event_t *)fetch_event();
{
if (result == NULL)
{
resultobj = Py_None;
Py_XINCREF(resultobj);
}
else
{
resultobj = PyTuple_New(7);
#ifdef PYTHON_PYPY
Py_XINCREF($1->node_tx);
Py_XINCREF($1->node_rx);
Py_XINCREF($1->packet);
#endif
PyTuple_SetItem(resultobj, 0 , PyInt_FromLong(result->code));
PyTuple_SetItem(resultobj, 1 , result->node_tx);
PyTuple_SetItem(resultobj, 2 , result->node_rx);
PyTuple_SetItem(resultobj, 3 , result->packet);
PyTuple_SetItem(resultobj, 4 , PyFloat_FromDouble(result-
>signal_power));
PyTuple_SetItem(resultobj, 5 , PyFloat_FromDouble(result->noise_power));
PyTuple_SetItem(resultobj, 6 , PyFloat_FromDouble(result->BER));
free(result);
Py_XINCREF(resultobj);
}
}
return resultobj;
fail:
return NULL;
}
So essentially the same code works in different ways for python and pypy. IMHO
there is a bug somewhere, but I have not time ATM to find it. And yes, it
leaks memory like hell due to extra ref=(
On Wednesday 28 September 2011 11:11:07 Amaury Forgeot d'Arc wrote:
> 2011/9/28 Alex Pyattaev
>
> > Hi!
> > I have a quite sophisticated program that can be summarized as follows:
> > 1. Save a pypy object pointer inside C program. Here I call Py_XINCREF
> > so
> > that
> > it does not get deleted.
> > 2. Do some logic, move this reference around C code.
> > 3. Return a python tuple via typemap, here I am probably supposed to
> > return a
> > borrowed reference. And in Python2 it works just fine. BUT. In pypy, for
> > some
> > reason, it causes segfault with following message:
> > """
> > Fatal error in cpyext, CPython compatibility layer, calling
> > PyTuple_SetItem Either report a bug or consider not using this
> > particular extension
> >
> > RPython traceback:
> > File "module_cpyext_api_1.c", line 28965, in PyTuple_SetItem
> > File "module_cpyext_pyobject.c", line 1018, in CpyTypedescr_realize
> >
> > Segmentation fault
> > """
> > If I call Py_XINCREF before returning the object, the crash does not
> > happen and the memory does not seem to be leaking (at least not
> > noticeably massive amounts of it). So it seems that PyPy is somewhat
> > incompatible with Python2 in
> > that matter.
> > If you want I could send the code example that triggers the bug (it IS
> > quite
> > large app, which might have many more bugs apart from this, but still).
>
> Isn't PyTuple_SetItem supposed to "steal" the reference?
> In this case you'd better INCREF the object if it is globally shared.
___
pypy-dev mailing list
pypy-dev@python.org
http://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] PyPy 1.6 not working on Windows XP
1. Switch to linux. It helps. 2. To get a meaningful error log try to run the pypy from terminal. To do copy-paste you will need 3-rd party terminal, i.e. power shell. Then you'll be able to copy the error messages. Without them it is pretty much impossible to identify the problem. 3. Another option is to use debugger to run the suite step-by-step and see what happens. Choose your poison. Alex. On Wednesday 05 October 2011 13:44:14 Ram Rachum wrote: > Hey guys, > > I upgraded to PyPy 1.6 on my 2 Windows XP 32 bit machines. It crashes on > both system when running the GarlicSim test suite. > > It shows a Windows error dialog saying "pypy.exe has encountered a problem > and needs to close. We are sorry for the inconvenience." and giving this > data: > > AppName: pypy.exe AppVer: 0.0.0.0 ModName: kernel32.dll > ModVer: 5.1.2600.5512 Offset: 00040b0d > > > I can also open a dialog with a lot of data on the error (don't know how > useful it is) but Windows won't let me Ctrl-C it. > > > What can I do? > > > Thanks, > Ram. ___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] PyPy 1.6 not working on Windows XP
Generally, any binary-level debugger such as gdb or MSVC should work with pypy. At the very least you will find which operation crashed. If it is something really specific, for example sin/log/sign, it might be quite easy to map it back to python code. If it is not, it will be nearly impossible to find the original source line (at least I've failed when I tried). Another option is to edit the sources of the test suite adding print statements incrementally until you spot the place where it crashes. It is a slow, but very reliable way. That is of course if it is a particular segment of python code that crashes it. Also, could you send your exact environtment specs? I'll try to replicate it on a VM and see if it crashes for me too. I have an XP VM somewhere. PS: Sorry for my stupid joke about switching to linux. It was meant to cheer you up a bit. Alex. On Wednesday 05 October 2011 14:18:08 Ram Rachum wrote: > On Wed, Oct 5, 2011 at 2:11 PM, Amaury Forgeot d'Arc wrote: > > 2011/10/5 Ram Rachum : > > > I have hundreds of tests, and PyPy fails before a single one begins. > > > It > > > seems that PyPy crashes somewhere in nose's initialization. > > > Isn't there a way to find the last Python line run before the crash > > > > without > > > > > stepping with a finer granularity every time? > > > > Not without a debugger, I'm afraid > > > > -- > > Amaury Forgeot d'Arc > > How do I run the Nose test suite on Pypy with a debugger? I usually use Wing > IDE, but it doesn't support PyPy. I'm also aware of Nose's `--pdb` flag > which drops you into the debugger after an error, but it doesn't work here > because this crash seems to be happening at a lower level. So I don't know > how to start this in a debugger. > > > Ram. ___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] PyPy 1.6 not working on Windows XP
> If I use a Python debugger, can't I just step forward line by line, see > where I get the crash, and then isolate the offending line? The way pypy works - no you can not really do that. In Cpython it works somewhat better, but not in PYPY. Basically you have to use C debugger to locate the crash point, because the program that crashes can not really identify where it crashed precisely (well it can but pypy detects crash points very unprecisely). So in order to figure where exactly it crashes you have to use C debugger. OR, as I have said, add print statements until you localize the bug. *This may not work in some cases, as adding print's actually modifies the program=> it might not crash at all* Alex. On Wednesday 05 October 2011 14:27:45 Ram Rachum wrote: ___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] PyPy 1.6 not working on Windows XP
I've had a very similar stuff, as in something crashing only when run many times when I had a bug in a container type implemented in C. Basically, I had wrong refcount for the objets, which caused them to be freed by garbage collectore while they have been still used. Maybe something similar happens in the code that wraps windows API that handles file opening. That would explain why the bug never happens on linux. A good candidate would be incorrect refcount for the return value if the file does not exist. Try something like this: s="some_file" rets=[] for i in range(1000): rets.append(os.stat(s)) gc.collect() #Do something that uses lots of RAM (but a random amount, preferably in small blocks) print rets if it crashes then you have exactly that issue. 1000 might be not enough to toggle the crash though, as you need the OS to actually allocate different segments of memory for this to work. The more RAM you have the more cycles you need to toggle the crash. At least this approach helped me to debug extension modules written in C. BTW, for me on VM the test case does not crash. But I have SP2 windows there. On Wednesday 05 October 2011 19:37:07 Ram Rachum wrote: > On Wed, Oct 5, 2011 at 6:51 PM, Amaury Forgeot d'Arc wrote: > > 2011/10/5 Ram Rachum : > > > Okay, I've spent a few hours print-debugging, and I think I've > > > almost got it. > > > > > > The crash happens on a line: > > > st = os.stat(s) > > > > > > inside `os.path.isdir`, where `s` is a string 'C:\\Documents and > > > Settings\\User\\My Documents\\Python > > > Projects\\GarlicSim\\garlicsim\\src' This is a directory that > > > happens not to exist, but of course this is not> > > a > > > > > good reason to crash. > > > I have tried running `os.stat(s)` in the PyPy shell with that same > > > `s`, > > > > but > > > > > didn't get a crash there. I don't know why it's crashing in Nose but > > > not> > > in > > > > > the shell. > > > > > > Does anyone have a clue? > > > > it's possible that it's a RPython-level exception, or a bad handle > > because too many files wait for the garbage collector to close them. > > > > Can you give more information about the crash itself? > > - What are the last lines printed in the console? Try to disable > > "stdout capture" in Nose, by passing the -s option. > > This is the entire output: > > Preparing to run tests using Python 2.7.1 (080f42d5c4b4, Aug 23 2011, > 11:41:11) > [PyPy 1.6.0 with MSC v.1500 32 bit] > Running tests directly from GarlicSim repo. > Pypy doesn't have wxPython, not loading `garlicsim_wx` tests. > nose.config: INFO: Set working dir to C:\Documents and Settings\User\My > Documents\Python Projects\GarlicSim > nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$'] > nose.plugins.cover: INFO: Coverage report will include only packages: > ['garlicsim', 'garlicsim_lib', 'garlicsim_wx', 'test_garlicsim', > 'test_garlicsim_lib', 'test_garlicsim_wx', 'garlicsim', 'garlicsim_lib', > 'garlicsim_wx', 'test_garlicsim', 'test_garlicsim_lib', 'test_garlicsim_wx', > 'garlicsim', 'garlicsim_lib', 'garlicsim_wx', 'test_garlicsim', > 'test_garlicsim_lib', 'test_garlicsim_wx'] > > > > > - after the pypy process has exited, type "echo %ERRORLEVEL%" in the > > > same console, to print the exit code > > of the last process. Which number is it? > > -1073741819 > > > -- > > Amaury Forgeot d'Arc ___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] Success histories needed
I have a fully-functional wireless network simulation tool written in pypy+swig. Is that nice? Have couple papers to refer to as well. If you want I could write a small abstract on how it was so great to use pypy (which, in fact, was really great)? Alex. On Friday 07 October 2011 20:17:01 Maciej Fijalkowski wrote: > Hi > > We're gathering success stories for the website. Anyone feel like > providing name, company and some data how pypy worked for them (we > accept info how it didn't work on bugs.pypy.org all the time :) > > Cheers, > fijal > ___ > pypy-dev mailing list > pypy-dev@python.org > http://mail.python.org/mailman/listinfo/pypy-dev ___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] Success histories needed
Well... that took a bit long-ish because I was preparing a small conference we are having here in Tampere. Anyway, as promised. You are free to do whatever you like with the stuff (cut/remove stuff if it is too big) as long as there is my name somewhere nearby. PYPY + SWIG as a basis for simulation platform System-level simulations of communications networks are very resource-hungry, mostly due to complex statistical relations that have to be uncovered. For example, it is not enough to observe just one or two packets in the network to conclude if it is stable or not, the actual observation should last a lot longer then any feedback process in the network. If the buffer sizes are large, this can mean hundreds of thousands of packets. Wireless networks add one extra problem - the channel model is typically an $N^2$ complexity algorithm, and it has to be run for each and every symbol transmitted over the radio interface. Unfortunately, most of the existing simulation platforms for wireless networks do not allow for multiple sources to coexist and cooperate, and therefore are unsuitable for my research. A new simulation environment was to be devised. Originally, my simulator was implemented as a time-advance model with Matlab. Unfortunately, complex logic of the network in question made it so slow, that it was practically useless. The second implementation in C was scalable enough for a single wireless cell, but it was clearly unsuitable for implementation of complex scheduling algorithms. Generally, scheduling algorithms operate with such terms as sets, maps and dynamic priority queues. The relay selection algorithms, which are the prime topic, are also operating with sets and mappings. Unfortunately, such high- level data structures do not fit well into the paradigm of C/C++. Therefore, it was decided to split the model into two parts: the link-level model implemented with C and the control level implemented with Python. Obviously, the bulk of calculations was to be performed at the link-level, and the complex control logic was to be implemented with Python. Unfortunately, the performance of Python interpreter soon appeared to be a bottleneck, but redesigning the model again did not seem like a right thing to do. And here, following the Zen of Python, the one obvious solution popped out - PyPy. PyPy is essentially a JIT-compiler for Python, and it allows to run Python programs with speeds close to those approachable with pure C++. At the same time it retains the flexibility of the scripting language. As for the C part, the SWIG wrapper generator allows to maintain the linking between the python top-level and binary backend in a clean, efficient and cross-platform manner. It also allows for a nice trick - the wrappers can be also generated for Octave, which allows to verify the channel model parts against their analytical models in Matlab-like environment. Unfortunately, SWIG can not generate interfaces for Matlab yet, but in most cases I have actually found Octave better and faster for small prototyping and testing projects. Currently, the PyPy 1.6 runs the model about 200\% faster then standard CPython 2.7.2. Also it provides the unique opportunities in configuration, as most of the model scenario can be constructed with object constructors and for loops, rather then some freaky config files or infinite bulks of XML. Overall, I am very statisfied with PyPy experience, and I plan on trying it in even more demanding environment - as an experimental engine for a P2P protocol tracker. Best regards, Alexander Pyattaev, Researcher, Department of Communications Engineering, Tampere University of Technology, Finland On Friday 21 October 2011 16:36:10 Bea During wrote: > Hi there > > Alexander Pyattaev skrev 2011-10-21 15:53: > > I think i will finalize that thing on this weekend. Of course you will > > get permissions to do whatever you wish with it, I am not a microsoft > > fan=) > Thanks! We look forward to reading it - and publishing it ;-) > > Cheers > > Bea > > > On Wednesday 19 October 2011 14:36:15 Bea During wrote: > >> Hi there > >> > >> Maciej Fijalkowski skrev 2011-10-17 10:30: > >>> On Mon, Oct 17, 2011 at 10:17 AM, Alex > >>> Pyattaev> > > wrote: > >>>> I have a fully-functional wireless network simulation tool written > >>>> in > >>>> pypy+swig. Is that nice? Have couple papers to refer to as well. > >>>> If > >>>> you want I could write a small abstract on how it was so great to > >>>> use > >>>> pypy (which, in fact, was really great)? > >>> > >>> Please :) > >> > >> As Maciej says - please do ;-) > >> > >> And I think we may want to be able
Re: [pypy-dev] cppyy: C++ bindings for PyPy
Well, I have never been able to isolate it, and the project where it is triggered is rather close-sourced. long story short - make up a SWIG object inside a function, pass it as a pointer to the swig-wrapped function, then call gc.collect(). After that swig calls object's destructor and if the C code is using it, youre screwd. I am not sure if the bug persists on a small scale, however, it only shows during long simulation runs where thousands of objects like that are circulated. If it ever happens on a smaller case I'll let you know. Alex maanantai 23 huhtikuu 2012 10:52:04 Amaury Forgeot d'Arc kirjoitti: 2012/4/23 Alexander Pyattaev Right now the main problem with SWIG is the object ownership, i.e. with pypy one has to set thisown=False for all objects to avoid crashes related to GC code. The problem occurs only for long-living objects though. Do you have a sample code for this issue? -- Amaury Forgeot d'Arc ___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] cppyy: C++ bindings for PyPy
Distro is latest gentoo, amd64 with sse2 and sse3 (core i5). libssl is in 2 versions: dev-libs/openssl-0.9.8u:0.9.8 dev-libs/openssl-1.0.0h:0 so pick one that suits you better. Anyway, default binary pypy works fine. Currently i have only one typemap in my *.i file for swig, and it allows PyObject* pointers to go through to c++ side. Otherwise no special features are used, so migration should be pretty painless. And of course if there are bugs they would most likely arise, especially if they are in memory management, because the c++ code is executed millions of times during the program run. BR, Alex PS: sorry for using 2 emails - one is work other home, screwd company policy. maanantai 23 huhtikuu 2012 07:52:49 wlavrij...@lbl.gov kirjoitti: > Hi Alexander, > > > An important question from userland - would it be benefitial to switch > > from > > SWIG to the new interface? When can that be done? > > there are no many major differences between SWIG and this on the Python side > (nothing that can't be captured in a compatibility layer). I'm thinking of > such things like cvar that is needed in SWIG, but not in cppyy, and the use > of templates which need to be named in the .i's in SWIG but are have a more > "instantiable" syntax in cppyy (for when the Cling backend becomes > available). Also, Reflex can parse more code than can SWIG, so you'd have > to be careful in which direction you use things. > > > Right now the main problem with SWIG is the object ownership, i.e. with > > pypy one has to set thisown=False for all objects to avoid crashes > > related to GC code. The problem occurs only for long-living objects > > though. > > The GC and cppyy work fine for those objects clearly owned by PyPy. Of > course, if the C++ side takes ownership, that would need to be patched up > "by hand" or with a rule (if the C++ API is consistent). > > > PS: I would have gladly tested an alpha-beta quality version of the lib (i > > have some unittests for SWIG bindings, so they should work with new lib > > also i suppose), but I can not build pypy from sources, not enuf RAM =) > > Maybe someone could send me a build with new feature for testing? > > What distro (in particular, 32b or 64b and which version of libssl)? I've > never distributed a binary version of PyPy, but could give it a try. > > Best regards, > Wim___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
[pypy-dev] cppyy use case question
Hello, I am considering trying cppyy on a large project, and I have a question. I need to be able to add a reference to Python objects into the c++ classes. In essence: #Python code: x="My python object" inst = cpp_class() inst.add_data(x) inst.do_stuff()#This does not modify x in any way z=inst.get_data(x) So, cpp_class acts as a container for python objects, and it does not need to know what kind of object is inside. I was able to make it work with pypy through basic C API, using SWIG and typemaps by mapping void* into PyObject*, but that is somewhat hackish. I am now thinking how would the same logic look with cppyy extension. PS: building cppyy is pain in the ass... You guys seriously need to think about build logs =)___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] cppyy use case question
keskiviikko 30 toukokuu 2012 18:52:25 wlavrij...@lbl.gov kirjoitti: > Hi Alex, > > > So, cpp_class acts as a container for python objects, and it does not need > > to know what kind of object is inside. I was able to make it work with > > pypy through basic C API, using SWIG and typemaps by mapping void* into > > PyObject*, but that is somewhat hackish. I am now thinking how would the > > same logic look with cppyy extension. > > this should be supported directly (PyROOT does), but where in CPython > handing a PyObject* is trivial, in PyPy one has to be created. My best > guess is that the route would be cpyext to get such a PyObject*, then hand > that over. > > I don't think it can be hacked around (a void* argument will peel the > contained C++ object, which will fail, since it's a real Python object). > > Identity preservation (on the return) should be easy, and the C++ side > should keep the refcount up for the proper duration. > > Let me figure that one out. Is as good a reason as any to have some fun and > do some coding again for the first time in weeks. :P > Thanks! I have accumulated some understanding about the process, and the issue can be worked around by having a dictionary that would map ints into python objects and vice versa before those are sent to c++, so when c++ functions feth them back they can be retreived. However, this would incur certain overhead, which may be rather annoying. Anyway, the whole point was to test if it will be any faster then cpyext, so unless this feature is properly resolved the test does not make any massive amount of sense (mostly due to this extra overhead). I'll probably run it anyway though=) Thanks for looking into it, its good to hear that somebody cares! > > PS: > > building cppyy is pain in the ass... You guys seriously need to think > > about > > build logs =) > > Sorry (but this is why at CERN we have a global file system (afs) and simply > install there for all users :) ) ... So, is there anything in the > instructions that was missing and needs to be added? > With respect to build, I installed root from gentoo repository, and the problem there is that headers are in /usr/include/root and the actual libs are in /usr/lib/root, but the pypy build expects libs to be in /usr/include/root/lib if you specify SYSROOT=/usr/include/root. So instead I ended up having to symlink those together to replicate whatever pypy build was expecting. This is indeed a hax, but probably if you install root from source package it is not needed. Anyway, it should be benefitial to add a header and library check in the beginning of the build to be certain that it does not fail after 20 mins of compiling. But for now symlink works just fine. With respect to genreflex command, you could probably warn people that they should use gcc 4.3, not 4.6, cuz gccxml, used by genreflex, is not compatible with newer versions of STL, and crashes on standard headers. Also, any headers that are not absolutely necessary to define the interface to Python should be removed if possible, that does make life a lot easier, even with SWIG, because it makes wrappers smaller. Best Regards, Alex PS: If you want I could give you benchmark results of cppyy vs cpyext+SWIG when we get it running, probably it will be of some interest. I could give you the entire code, but it would not be much use since it requires a ton of other libraries.___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] Python FFI
So, when would users be able to test it? I'd like to compare the overhead with ctypes and cppyy for instance. I also wonder if I'd be able to send Python objects to the c++ domain for storage in a container (e.g. a vector). BR, Alex keskiviikko 30 toukokuu 2012 18:00:16 Robert Zaremba kirjoitti: > Thanks for clarification. I've been wondering why to dobule the work. > > I'm also really fascinated about this idea, and that i don't need to use > separate tool. > It looks really great! Thanks!___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] Python FFI
Okay, i see the issue now. Basically i was under an impression that FFI was somehow a more elaborate thing... Sorry for the bother, the actual thing i need is more like cppyy, but working =) Anyways, might be actually interesting to see it work with SWIG, since it uses C-like functions to wrap C++ classes. But, as you have said, it is pre-alpha, so no SWIG as of now... BR, Alex. sunnuntai 03 kesäkuu 2012 16:28:52 Armin Rigo kirjoitti: > Hi Alex, > > On Sun, Jun 3, 2012 at 10:16 AM, Alex Pyattaev wrote: > > So, when would users be able to test it? I'd like to compare the overhead > > with ctypes and cppyy for instance. > > The first version is ready for testing at https://github.com/arigo/ffi > . Note that the goal is not to interface with C++ but only with C. > In this pre-0.1-release, it supports the same things as a basic > ctypes, minus all the Windows calling convention stuff. We're going > to work now on non-ctypes issues like structs without a fixed known > layout, macros, etc. (listed in the README.md file). > > Reading just the file names in 'ctypes/test/test_*.py', I should > stress again that the scale of the project is completely different > than ctypes: it's meant as a minimal interface to C offering a C-like > feeling, with absolutely no extra features on top. > > See the 'demo/readdir.py' file for a quick introduction. If you > already know independently Python, C, and (for this example) did at > some point read "man openat" on Linux, then this code should be > self-explanatory and kind of obvious. > > > A bientôt, > > Armin.___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] cppyy use case question
Please, mention in the manual, for the mortals, that the reflection library has to be linked with -lReflex, so that they do not have to figure that out on their own=) BR, Alex. sunnuntai 03 kesäkuu 2012 02:33:15 Alex Pyattaev kirjoitti: > good you found a workaround, as using cpyext from cppyy is turning out to be > a bigger can of worms that I expected. :} Will take a tad bit longer: I > have some working code, but only post-translation. For some reason, the > resulting objects look nothing as expected in the test and as passed, they > go into the CPython interpreter (this is a test, after all), which then > bombs. Cpyext is not needed with cppyy as far as I understand, is it? > Post-translation, it's looking fine, though, and working as expected. > > Performance ... is absolutely deplorable: slower than PyROOT on CPython! Performance was originally the point, right? Or am I missing something (lol)? > My guess is that in the micro-benchmark, the JIT gets no chance of making a > trace out of the loop iteration, since the loop iteration requires the > PyObject in every call (the PyObject that I'm passing is the iterator > variable itself). I'll have to look into more detail I am giving you my benchmark code, right now it uses dictionary to map pyobject id into objects when they leave c++ domain. Obviously, you add about O(2*log(N)) overhead to the code, but that should not be too much anyway. If that can be done in rpython, it could probably be made to work massively faster. Like a module that does object caching for the storage of objects that would be sent to c++ domain. Then in c++ one can do as simple as have an long int index that can be used to resolve back to byobject when the job is done. It also does classical SWIG typemap to do the same job. to use swig use SConstruct build. Or just use the run.sh script to get all the stuff immediately. > > With respect to genreflex command, you could probably warn people that > > they > > should use gcc 4.3, not 4.6, cuz gccxml, used by genreflex, is not > > compatible with newer versions of STL, and crashes on standard headers. > > That shouldn't be ... on my box I have gcc 4.6.2 and that does not pose any > problems. What version of gccxml do you have? Mine's at 0.9.0. mine is 0.9.0-2009.05.16, but it does crash with gcc 4.6. maybe you have older gcc in a different slot? > > Also, any headers > > that are not absolutely necessary to define the interface to Python should > > be removed if possible, that does make life a lot easier, even with SWIG, > > because it makes wrappers smaller. > > Have you looked into providing a selection.xml to genreflex to select the > needed classes? Also, another common option is to collect all desired > headers in a single header file (with simple #include's) and feed that to > genreflex, together with a selection.xml that simply uses "*" patterns. The point is that if the header is included, and it confuses the parser, then the patterns have no effect. So it is better not to have extra headers like Qt library=) Alex. ___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] cppyy use case question
Okay, with respect to the SWIG + pypy, works for me, and about 4x slower then cppyy or cpython ( which show same results more or less). time cppyy ./cppyy_timings.py real0m0.265s user0m0.225s sys 0m0.040s time pypy ./pypy_timings.py real0m2.424s user0m2.338s sys 0m0.086s time python2 ./pypy_timings.py real0m0.414s user0m0.390s sys 0m0.024s So apparently pypy is dramatically bad with SWIG. cppyy seems to work nice. I have, however, encountered an interesting problem. Say I have a dictionary built. I'd like to know which symbols/classes are actually available (if any), but I can not do that. The implication is that in case your library does not have any symbols in it after SUCCESSFUL build, there is no way to figure out why and how that happened. Please advise. Also I can bench the entire thing on a real, quite large app that makes intensive use of pypy and also runs on python2 if needed, but for that we would need to figure why no classes are visible in the dictionary=) BR, Alex sunnuntai 03 kesäkuu 2012 16:44:19 wlavrij...@lbl.gov kirjoitti: > Hi Alex, > > >> Performance ... is absolutely deplorable: slower than PyROOT on CPython! > > > > Performance was originally the point, right? Or am I missing something > > (lol)? > well, remember that there are two problems here: 1) since there are no > PyObject's as such in pypy-c (they have different internal structure), one > has the create them, to have a PyObject* pointer to them; and keep track of > them to mix the reference counting scheme with the garbage collection scheme > (this is all done in/by cpyext). And 2) now that the object is created, > pinning down a variable in the .py code, the JIT can't unbox it. > > The example I used, which was like so: > >for i in range(N): >a.add_data(i) > > and is rather pathological, of course. But then, all micro-benches are. > > Conversely, doing: > >l = [] >for i in range(N): >l.append(i) >a.add_data(l) > > is very speedy, presumably (I don't know, guessing here as it may also be > that there is another optimization for the list that I can't prevent even > with the append() calls) because now the cpyext code is in a loop in and > of itself, so e.g. lookup of the type of the objects in the container (all > the same here) can become constant. > > > I am giving you my benchmark code > > Which clearly is also pathological. :) I can't manage to run the SWIG test > from pypy, though? I tried by adding cpyext, but no dice? > > Anyway, you'll be happy to know that: > > CPPYY > Created container > No more items > > real 0m2.236s > user 0m2.156s > sys 0m0.071s > > Cpython > Created container > No more items > > real 0m2.758s > user 0m2.574s > sys 0m0.174s > > (I verified with your debug=1 that the objects appear.) > > But that improvement is (I'm sure) only b/c all the other stuff you do > around it with manipulating the dict on the python side. That does get > JIT'ed just fine. > > Btw., I had to jack up N by 10x, otherwise all numbers were just noise. :} > > > right now it uses dictionary to map > > pyobject id into objects when they leave c++ domain. Obviously, you add > > about O(2*log(N)) overhead to the code, but that should not be too much > > anyway. If that can be done in rpython, it could probably be made to work > > massively faster. Like a module that does object caching for the storage > > of objects that would be sent to c++ domain. Then in c++ one can do as > > simple as have an long int index that can be used to resolve back to > > byobject when the job is done. > > From a quick read of the cpyext code, that is rather similar to what is > there. > > But as per above, the problem isn't with the extra overhead, the problem is > with the JIT being prevented to do any work (and with pypy-c being slower > than CPython in pure interpreted mode). > > > mine is 0.9.0-2009.05.16, but it does crash with gcc 4.6. maybe you have > > older gcc in a different slot? > > I do, but those should not be accessible from my pypy work environment. I > did rebuild it. Not sure whether that matters. I'll update the docs showing > how a different set of headers from another compiler can be selected. > > > The point is that if the header is included, and it confuses the parser, > > then the patterns have no effect. So it is better not to have extra > > headers like Qt library=) > > Qt is a story in and of itself, of course. :) Actually, what folks here do, > is to link any code from external C++ through python to Qt (C++) by making > use of the fact that both ends can handle opaque pointers, represented as > long values. (The combination of addressof and bind_object in cppyy.) > > Best regards, > Wim___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] cppyy use case question
I think I've figured it with the missing classes. When supplied with multiple headers, genreflex apparently only saves the last one of them...Obviously, that can be worked around, but still seems like a bug or just me not finding some manual. BR, Alex. sunnuntai 03 kesäkuu 2012 16:44:19 wlavrij...@lbl.gov kirjoitti: > Hi Alex, > > >> Performance ... is absolutely deplorable: slower than PyROOT on CPython! > > > > Performance was originally the point, right? Or am I missing something > > (lol)? > well, remember that there are two problems here: 1) since there are no > PyObject's as such in pypy-c (they have different internal structure), one > has the create them, to have a PyObject* pointer to them; and keep track of > them to mix the reference counting scheme with the garbage collection scheme > (this is all done in/by cpyext). And 2) now that the object is created, > pinning down a variable in the .py code, the JIT can't unbox it. > > The example I used, which was like so: > >for i in range(N): >a.add_data(i) > > and is rather pathological, of course. But then, all micro-benches are. > > Conversely, doing: > >l = [] >for i in range(N): >l.append(i) >a.add_data(l) > > is very speedy, presumably (I don't know, guessing here as it may also be > that there is another optimization for the list that I can't prevent even > with the append() calls) because now the cpyext code is in a loop in and > of itself, so e.g. lookup of the type of the objects in the container (all > the same here) can become constant. > > > I am giving you my benchmark code > > Which clearly is also pathological. :) I can't manage to run the SWIG test > from pypy, though? I tried by adding cpyext, but no dice? > > Anyway, you'll be happy to know that: > > CPPYY > Created container > No more items > > real 0m2.236s > user 0m2.156s > sys 0m0.071s > > Cpython > Created container > No more items > > real 0m2.758s > user 0m2.574s > sys 0m0.174s > > (I verified with your debug=1 that the objects appear.) > > But that improvement is (I'm sure) only b/c all the other stuff you do > around it with manipulating the dict on the python side. That does get > JIT'ed just fine. > > Btw., I had to jack up N by 10x, otherwise all numbers were just noise. :} > > > right now it uses dictionary to map > > pyobject id into objects when they leave c++ domain. Obviously, you add > > about O(2*log(N)) overhead to the code, but that should not be too much > > anyway. If that can be done in rpython, it could probably be made to work > > massively faster. Like a module that does object caching for the storage > > of objects that would be sent to c++ domain. Then in c++ one can do as > > simple as have an long int index that can be used to resolve back to > > byobject when the job is done. > > From a quick read of the cpyext code, that is rather similar to what is > there. > > But as per above, the problem isn't with the extra overhead, the problem is > with the JIT being prevented to do any work (and with pypy-c being slower > than CPython in pure interpreted mode). > > > mine is 0.9.0-2009.05.16, but it does crash with gcc 4.6. maybe you have > > older gcc in a different slot? > > I do, but those should not be accessible from my pypy work environment. I > did rebuild it. Not sure whether that matters. I'll update the docs showing > how a different set of headers from another compiler can be selected. > > > The point is that if the header is included, and it confuses the parser, > > then the patterns have no effect. So it is better not to have extra > > headers like Qt library=) > > Qt is a story in and of itself, of course. :) Actually, what folks here do, > is to link any code from external C++ through python to Qt (C++) by making > use of the fact that both ends can handle opaque pointers, represented as > long values. (The combination of addressof and bind_object in cppyy.) > > Best regards, > Wim___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] cppyy use case question
Well, as far as I can see from where I am, the only real good that can come from interfacing with PyObject is the capability to write container classes in C++. On the other hand, it seems that it is not very often needed. For the case when objects just need to be LINKED from the C++ code as references, pure python/Rpython abstraction layer that would keep a backup copy of the object (which can be discarded from the python side) seems like a better idea. The main benefit is that there will be no refcounting problem and no slate pointers - if C++ returns some key that is not available then exception can be raised instead of segfault as with PyObject*. Also, no extra headers will be needed on C++ side, and the datatype can be an integer. Why not? As far as I understand, modifying python objects in C++ with pypy is massively stupid. By the way - the only reason i use C++ in my project is because I have a massive amount of legacy code that uses several C libraries and Qt, as well as openGL, so porting that bunch to python can be quite a hassle. The only problem is that Python side has to send certain requests to C++, those should be cached, processed and reported back up when done. Obviously, request object itself could travel along with the request data for code clarity, but there can just as well be an int. The only problem with ints is the lifetime of the references in the mapping structure, which would have to be managed explicitly in such case. If you are interested, I could send you some thoughts on that issue specifically, as I was planning to do this kind of thing at some point. BR, Alex. tiistai 05 kesäkuu 2012 09:54:25 Armin Rigo kirjoitti: > Hi Wim, > > On Tue, Jun 5, 2012 at 1:53 AM, wrote: > > Overall, I'm happy enough with the result (I'm going to need handling of > > PyObject*'s later for my own nefarious purposes). > > Maybe we can motivate you to look for and fix inefficiencies in > cpyext :-) As far as I know it's still fully designed for "it works > and we don't care about speed". See for example > https://bugs.pypy.org/issue1121 which tries to push optimizations in > cpyext but with no-one really interested from the core group. > > But IMHO the main slow part that should be optimized (probably by > being rewritten from scratch) is the correspondance between the proxy > PyObject structures and the real PyPy objects, which is exactly the > part that interests you too. It should be possible at least to cope > with only one dict lookup instead of at least two to do the roundtrip > "pypy obj -> cpyext obj -> pypy obj". > > > A bientôt, > > Armin.___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] cppyy use case question
> If the goal > is "only" to pass around PyObjects as black boxes, as fast as > reasonably possible, then the easiest route would be to not use cpyext > at all, but instead try to come up with your own mapping logic. This > looks quite easier for experimentation purposes. And if you end up > with a good and fast solution we can always try later to reuse it in > cpyext. This is precisely the goal for me. The requests in my system are basically a C++ classes (structs, essentially), that are created and submitted to C++ core as pointers, they also hold a pointer to the PyObject that submitted the request. When the request is served, the pyobject can be notified about that without having to go through any sort of lookups. > > 1) a pair of functions to register and unregister a GC pointer that > lives outside the GC That might actually be a good idea. In that case a certain object can tell GC that it wants to have a valid external pointer, and when the object is destroyed, a __del__ function can remove that registration with GC. If anyone uses that pointer afterwards it is probably his/her own problem. I think also that SWIG's idea with c++ side "owning" python objects is rather lame. Basically, if an object is owned by c++ and not cleaned properly all kinds of strange things can happen... BR, Alex.___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] cppyy use case question
Great, that actually works. torstai 07 kesäkuu 2012 15:34:33 wlavrij...@lbl.gov kirjoitti: > Hi Alex, > > > With respect to build, I installed root from gentoo repository, and the > > problem there is that headers are in /usr/include/root and the actual libs > > are in /usr/lib/root, but the pypy build expects libs to be in > > /usr/include/root/lib if you specify SYSROOT=/usr/include/root. > > now picking up include and libdirs via root-config, but then it does require > that tool to be available through PATH (it should be under $ROOTSYS/bin). > If not (e.g. standalone Reflex), it will still fall back to > $ROOTSYS/include etc. > > Best regards, > Wim___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] cpyext performance
And now for something completely different: Would not it make dramatically more sense to just finish the cppyy branch and get a working extension-making scheme that actually works? I have a project now that uses C++ extensions and python quite extensively, and the only thing stopping me from migrating to cppyy from SWIG is the fact that cppyy is still so unfinished and requires rebuild of pypy with root libraries and all the other tedious things, that slow down deployment. BR, Alex___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
