Hey Jake, On Thu, Jan 21, 2010 at 10:48 AM, Jake McGuire <mcgu...@google.com> wrote: > On Thu, Jan 21, 2010 at 10:19 AM, Reid Kleckner <r...@mit.edu> wrote: >> On Thu, Jan 21, 2010 at 12:27 PM, Jake McGuire <mcgu...@google.com> wrote: >>> On Wed, Jan 20, 2010 at 2:27 PM, Collin Winter <collinwin...@google.com> >>> wrote: >>>> Profiling >>>> --------- >>>> >>>> Unladen Swallow integrates with oProfile 0.9.4 and newer [#oprofile]_ to >>>> support >>>> assembly-level profiling on Linux systems. This means that oProfile will >>>> correctly symbolize JIT-compiled functions in its reports. >>> >>> Do the current python profiling tools (profile/cProfile/pstats) still >>> work with Unladen Swallow? >> >> Sort of. They disable the use of JITed code, so they don't quite work >> the way you would want them to. Checking tstate->c_tracefunc every >> line generated too much code. They still give you a rough idea of >> where your application hotspots are, though, which I think is >> acceptable. > > Hmm. So cProfile doesn't break, but it causes code to run under a > completely different execution model so the numbers it produces are > not connected to reality? > > We've found the call graph and associated execution time information > from cProfile to be extremely useful for understanding performance > issues and tracking down regressions. Giving that up would be a huge > blow.
FWIW, cProfile's call graph information is still perfectly accurate, but you're right: turning on cProfile does trigger execution under a different codepath. That's regrettable, but instrumentation-based profiling is always going to introduce skew into your numbers. That's why we opted to improve oProfile, since we believe sampling-based profiling to be a better model. Profiling was problematic to support in machine code because in Python, you can turn profiling on from user code at arbitrary points. To correctly support that, we would need to add lots of hooks to the generated code to check whether profiling is enabled, and if so, call out to the profiler. Those "is profiling enabled now?" checks are (almost) always going to be false, which means we spend cycles for no real benefit. Can YouTube use oProfile for profiling, or is instrumented profiling critical? oProfile does have its downsides for profiling user code: you see all the C-language support functions, not just the pure-Python functions. That extra data might be useful, but it's probably more information than most people want. YouTube might want it, though. Assuming YouTube can't use oProfile as-is, there are some options: - Write a script around oProfile's reporting tool to strip out all C functions from the report. Enhance oProfile to fix any deficiencies compared to cProfile's reporting. - Develop a sampling profiler for Python that only samples pure-Python functions, ignoring C code (but including JIT-compiled Python code). - Add the necessary profiling hooks to JITted code to better support cProfile, but add a command-line flag (something explicit like -O3) that removes the hooks and activates the current behaviour (or something even more restrictive, possibly). - Initially compile Python code without the hooks, but have a trip-wire set to detect the installation of profiling hooks. When profiling hooks are installed, purge all machine code from the system and recompile all hot functions to include the profiling hooks. Thoughts? Collin Winter _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
