On Jul 12, 2009, at 8:24 PM, Kurt Smith wrote: > In the midst of GSoC, I wanted to float an idea about a Cython > profiler, similar to the current Python profiler. Specifically I > wanted to see if there's interest, what sorts of features would be > desired, and, broadly, some implementation ideas. > > I think the arguments in favor are clear: > > Often people use Cython incrementally -- take a .py file, profile it, > put the hotspots in Cython. If fast enough -- you're done. If not, > continue to put more slow python code into Cython, or determine if the > Cython code should be massaged to yield better performance. In the > latter case it is difficult to zone in on the Cython hotspot, for it > is not possible for the Python profiler to see it. Other external > tools must be used (gprof?, valgrind, mac os x-specific profiler), > which is inconvenient at best; other times they are just plain > unavailable. > > I'd like to see a profiling option to the cython compiler that would > allow the user to, as seamlessly as possible, run the compiled > extension module with profiling enabled, and have the user interaction > be just like the current python profiler, specifically with meaningful > output for the Cython-level functions. If possible, I'd like to have > the cython profiling code make use of the python cProfiler module so > profiling isn't any different between pure Python & Cython compiled > code. I don't know how feasible hooking up with the python cProfiler > is, so it is just in the pie-in-the-sky stage right now. > > Another tantalizing possibility would be line-by-line profiling, > beyond the the regular function-level profiling.
After talking a bit about it at the last Sage days, and looking (a tiny bit) into how this stuff works, this really is quite feasible, and would be awesome to have. > What other features would be desired? What user interaction features > would be good to have? > > Implementation-wise, it could be done as its own transform, enabled by > compile-time switch. Certainly we would want to enable this with a compile-time switch (and, if it can be done not to verbosely, maybe even via macros). I'm not sure transformations would be the approach to take here (seems not the right level)--I'd probably add some stuff to the Code object and maybe even latch onto the emit_marker for the line-by-line. > Someone else (Robert? in some thread?) mentioned somewhere about > getting debugger-support in Cython, allowing one to step through > Cython code just like pdb (one can use gdb currently, of course). I > imagine the implementation issues for a Cython debugger would overlap > a good deal with the profiler (it would be its own separate tranform, > obviously). Yep. I don't have any idea how pdb works, but I'd imagine if we can provide fake frames (the code is there for exceptions and locals()) it wouldn't be too hard to hook up. Making gdb friendlier is an orthogonal but still very useful idea to pursue as well, especially for code that interfaces with external libraries. > I'll put each in a CEP, hopefully soon (and after the benefits of > comments, etc.) That would be good. I'll try to post my thoughts/findings there as well. - Robert _______________________________________________ Cython-dev mailing list [email protected] http://codespeak.net/mailman/listinfo/cython-dev
