On Tue, Jul 14, 2009 at 4:25 AM, Robert Bradshaw<[email protected]> wrote: > On Jul 12, 2009, at 8:24 PM, Kurt Smith wrote: > >> In the midst of GSoC, I wanted to float an idea about a Cython >> profiler, similar to the current Python profiler. Specifically I >> wanted to see if there's interest, what sorts of features would be >> desired, and, broadly, some implementation ideas. >> >> I think the arguments in favor are clear: >> >> Often people use Cython incrementally -- take a .py file, profile it, >> put the hotspots in Cython. If fast enough -- you're done. If not, >> continue to put more slow python code into Cython, or determine if the >> Cython code should be massaged to yield better performance. In the >> latter case it is difficult to zone in on the Cython hotspot, for it >> is not possible for the Python profiler to see it. Other external >> tools must be used (gprof?, valgrind, mac os x-specific profiler), >> which is inconvenient at best; other times they are just plain >> unavailable. >> >> I'd like to see a profiling option to the cython compiler that would >> allow the user to, as seamlessly as possible, run the compiled >> extension module with profiling enabled, and have the user interaction >> be just like the current python profiler, specifically with meaningful >> output for the Cython-level functions. If possible, I'd like to have >> the cython profiling code make use of the python cProfiler module so >> profiling isn't any different between pure Python & Cython compiled >> code. I don't know how feasible hooking up with the python cProfiler >> is, so it is just in the pie-in-the-sky stage right now. >> >> Another tantalizing possibility would be line-by-line profiling, >> beyond the the regular function-level profiling. > > After talking a bit about it at the last Sage days, and looking (a > tiny bit) into how this stuff works, this really is quite feasible, > and would be awesome to have.
Good! I'm at a 2-week workshop currently; I'll hopefully get to respond in more detail this weekend. Here's what I have for now. > >> What other features would be desired? What user interaction features >> would be good to have? >> >> Implementation-wise, it could be done as its own transform, enabled by >> compile-time switch. > > Certainly we would want to enable this with a compile-time switch > (and, if it can be done not to verbosely, maybe even via macros). > I'm not sure transformations would be the approach to take here > (seems not the right level)--I'd probably add some stuff to the Code > object and maybe even latch onto the emit_marker for the line-by-line. > One reason I thought of a transform is to deal with other transforms that might modify the python statements in such a way that makes it difficult to reproduce the 'generated-C -> python-statement' mapping at the code-generation phase. Not having looked much at this part of Cython (the Code object that is) maybe I'm way off here. For, e.g. the 'with' transform, would it be easy to figure out all the generated C code that corresponds to the original 'with' statement in the Python source through the Code object (essentially at code generation phase, right)? Or is it mixed around enough that this would be difficult to sort out? My basic thinking is something like: Have the profiling transform inject 'marker' lines at the necessary level of detail (for function-level or line-by-line level profiling). This would come fairly early in the pipeline before mangling transforms take place. The markers would contain all the necessary info -- the source line text, location, etc. At code generation phase, these marker lines would be turned into profiling code (macros as you mention, or something else if too complex). >> Someone else (Robert? in some thread?) mentioned somewhere about >> getting debugger-support in Cython, allowing one to step through >> Cython code just like pdb (one can use gdb currently, of course). I >> imagine the implementation issues for a Cython debugger would overlap >> a good deal with the profiler (it would be its own separate tranform, >> obviously). > > Yep. I don't have any idea how pdb works, but I'd imagine if we can > provide fake frames (the code is there for exceptions and locals()) > it wouldn't be too hard to hook up. Making gdb friendlier is an > orthogonal but still very useful idea to pursue as well, especially > for code that interfaces with external libraries. > I misspoke somewhat -- I meant to say that one can do C-level debugging with gdb right now if necessary, but we need a pdb-like debugging for Cython-level source code. Without looking under the hood at pdb either, I think the fake frame idea is a good start. >> I'll put each in a CEP, hopefully soon (and after the benefits of >> comments, etc.) > > That would be good. I'll try to post my thoughts/findings there as well. > Great. I won't be able to get to it anytime in the next 2 weeks, but soon thereafter. Thanks for the thoughts. Kurt _______________________________________________ Cython-dev mailing list [email protected] http://codespeak.net/mailman/listinfo/cython-dev
