Robert Bradshaw wrote: > On Jul 12, 2009, at 8:24 PM, Kurt Smith wrote: > >> In the midst of GSoC, I wanted to float an idea about a Cython >> profiler, similar to the current Python profiler. Specifically I >> wanted to see if there's interest, what sorts of features would be >> desired, and, broadly, some implementation ideas.
+1, totally interested. >> I think the arguments in favor are clear: >> >> Often people use Cython incrementally -- take a .py file, profile it, >> put the hotspots in Cython. If fast enough -- you're done. If not, >> continue to put more slow python code into Cython, or determine if the >> Cython code should be massaged to yield better performance. In the >> latter case it is difficult to zone in on the Cython hotspot, for it >> is not possible for the Python profiler to see it. Other external >> tools must be used (gprof?, valgrind, mac os x-specific profiler), >> which is inconvenient at best; other times they are just plain >> unavailable. The main problem I always had here is the tool barrier. You can use cProfile for profiling Python code, but it can't look into the Cython functions. Profiling Cython itself will show you loads of time spent inside the main parser function, without further detail. That's not quite the right level of granularity for optimisations. And you can use callgrind for C-level profiling, which works very well thanks to kcachegrind, but is much too low-level for the Python stuff. So all you see is the internal interpreter calls instead of the code you are really looking at. And line-level profiling obviously means C-lines, not Cython lines. Can be ok sometimes, but it's still more low-level than necessary, and hard to follow in the code you write. Having something in-between, maybe something that interfaces with cProfile and Python's stats module would make things a lot easier. There's already a conversion script that allows you to look at Python profiling dumps with kcachegrind, so it might not even be too hard to make it work with both, so that you could get profiling information at the Python, Cython and C level. http://ddaa.net/blog/python/lsprof-calltree http://oubiwann.blogspot.com/2006/08/python-and-kcachegrind.html http://www.gnome.org/~johan/lsprofcalltree.py >> I'd like to see a profiling option to the cython compiler that would >> allow the user to, as seamlessly as possible, run the compiled >> extension module with profiling enabled, and have the user interaction >> be just like the current python profiler, specifically with meaningful >> output for the Cython-level functions. If possible, I'd like to have >> the cython profiling code make use of the python cProfiler module so >> profiling isn't any different between pure Python & Cython compiled >> code. I don't know how feasible hooking up with the python cProfiler >> is, so it is just in the pie-in-the-sky stage right now. >> >> Another tantalizing possibility would be line-by-line profiling, >> beyond the the regular function-level profiling. > > After talking a bit about it at the last Sage days, and looking (a > tiny bit) into how this stuff works, this really is quite feasible, > and would be awesome to have. > >> What other features would be desired? What user interaction features >> would be good to have? >> >> Implementation-wise, it could be done as its own transform, enabled by >> compile-time switch. > > Certainly we would want to enable this with a compile-time switch > (and, if it can be done not to verbosely, maybe even via macros). > I'm not sure transformations would be the approach to take here > (seems not the right level)--I'd probably add some stuff to the Code > object and maybe even latch onto the emit_marker for the line-by-line. Absolutely. After all, profiling is not more than collecting information like "I'm here, time is XYZ". We emit tons of error handling code that contains line information, so this would just be one line more for each line of Cython source, and likely similar to the code generated for the ref-nanny. I'd also certainly go for emit_marker() here (which simply needs to get fixed wherever it's not accurate enough to mark each line). We need to make sure that this only emits code inside of C function bodies, though. >> Someone else (Robert? in some thread?) mentioned somewhere about >> getting debugger-support in Cython, allowing one to step through >> Cython code just like pdb (one can use gdb currently, of course). I >> imagine the implementation issues for a Cython debugger would overlap >> a good deal with the profiler (it would be its own separate tranform, >> obviously). > > Yep. I don't have any idea how pdb works, but I'd imagine if we can > provide fake frames (the code is there for exceptions and locals()) > it wouldn't be too hard to hook up. Making gdb friendlier is an > orthogonal but still very useful idea to pursue as well, especially > for code that interfaces with external libraries. It would be good to look into the Python extensions to gdb first, I'd say. Something like that might work well enough already. After all, when you debug, you *must* see the C code, so that's different from profiling. Stefan _______________________________________________ Cython-dev mailing list [email protected] http://codespeak.net/mailman/listinfo/cython-dev
