Re: [Python-Dev] PyCharm debugger became 40x faster on Python 3.6 thanks to PEP 523
On 03/25/2017 08:57 PM, Terry Reedy wrote: > On 3/25/2017 8:56 AM, Serhiy Storchaka wrote: >> On 25.03.17 12:04, Victor Stinner wrote: >>> https://blog.jetbrains.com/pycharm/2017/03/inside-the-debugger-interview-with-elizaveta-shashkova/ >>> >>> >>> "What changed in Python 3.6 to allow this? >>> >>> The new frame evaluation API was introduced to CPython in PEP 523 and it >>> allows to specify a per-interpreter function pointer to handle the >>> evaluation of frames." >>> >>> Nice! >> >> Awesome! Any chance that pdb can utilize similar technique? Or this >> doesn't make sense for pdb? > > According to the bdb.Bdb docstring, pdb implements a command-line user interface on top of bdb, while bdb.Bdb "takes care of the details of the trace facility". idlelib.debugger similarly implements > a GUI user interface on top of bdb. I am sure that there are other debuggers that build directly or indirectly (via pdb) on bdb. So the question is whether bdb can be enhanced or augmented with a > C-coded _bdb or other new module. > > As I understand it, sys.settrace results in an execution break and function call at each point in the bytecode corresponding to the beginning of a (logical?) line. This add much overhead. In return, > a trace-based debugger allows one to flexibly control stop and go execution either with preset breakpoints* or with interactive commands: step (one line), step into (a function frame), step over (a > function frame), or go to next breakpoint. The last is implemented by the debugger automatically stepping at each break call unless the line is in the existing breakpoint list. > > * Breakpoints can be defined either in an associated editor or with breakpoint commands in the debugger when execution is stopped. > > PEP 523 envisioned an alternate non-trace implementation of 'go to next breakpoint' by a debugger going "as far as to dynamically rewrite bytecode prior to execution to inject e.g. breakpoints in the > bytecode." > https://www.python.org/dev/peps/pep-0523/#debugging > > A debugger doing this could either eliminate the other 'go' commands (easiest) or implement them by either setting temporary breakpoints or temporarily turning tracing on. > > I presume it should be possible to make bdb.Bdb use bytecode breakpoints or add a new class with a similar API. Then any bdb-based debugger to be modified to make the speedup available. pdb-clone, an extension to pdb, gets about those same performance gains over pdb while still using sys.settrace(). pdb-clone runs at a speed of less than twice the speed of the interpreter when pdb runs at about 80 times the speed of the interpreter. See some performance measurements at https://bitbucket.org/xdegaye/pdb-clone/wiki/Performances.md Given those results, it is not clear how one would get a boost of a factor 40 by implementing PEP 523 for the pdb debugger as pdb could already be very close to the speed of the interpreter mostly by implementing in a C extension module the bdb.Bdb methods that check whether the debugger should take control. Setting a trace function with sys.settrace() adds the following incompressible overhead: * 15-20 % overhead: computed goto are not used in the ceval loop when tracing is active. * The trace function receives all the PyTrace_LINE events (even when the frame f_trace is NULL :(). The interpreter calls _PyCode_CheckLineNumber() for each of these events and the processing in this function is the one that is costly. An optimization is done in pdb-clone that swaps the trace function with a profiler function whenever possible (i.e. when there is no need to trace the lines of the function) to avoid calling _PyCode_CheckLineNumber() (the profiler still gets PyTrace_C_CALL events but there is not such overhead with these events). The performance gain obtained with this scheme is about 30%. I think that the main point here is not whether to switch from sys.settrace() to PEP 523, but first to implement the stop_here() bdb.Bdb method in a C extension module. Xavier ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PyCharm debugger became 40x faster on Python 3.6 thanks to PEP 523
On 3/25/2017 8:56 AM, Serhiy Storchaka wrote: On 25.03.17 12:04, Victor Stinner wrote: https://blog.jetbrains.com/pycharm/2017/03/inside-the-debugger-interview-with-elizaveta-shashkova/ "What changed in Python 3.6 to allow this? The new frame evaluation API was introduced to CPython in PEP 523 and it allows to specify a per-interpreter function pointer to handle the evaluation of frames." Nice! Awesome! Any chance that pdb can utilize similar technique? Or this doesn't make sense for pdb? According to the bdb.Bdb docstring, pdb implements a command-line user interface on top of bdb, while bdb.Bdb "takes care of the details of the trace facility". idlelib.debugger similarly implements a GUI user interface on top of bdb. I am sure that there are other debuggers that build directly or indirectly (via pdb) on bdb. So the question is whether bdb can be enhanced or augmented with a C-coded _bdb or other new module. As I understand it, sys.settrace results in an execution break and function call at each point in the bytecode corresponding to the beginning of a (logical?) line. This add much overhead. In return, a trace-based debugger allows one to flexibly control stop and go execution either with preset breakpoints* or with interactive commands: step (one line), step into (a function frame), step over (a function frame), or go to next breakpoint. The last is implemented by the debugger automatically stepping at each break call unless the line is in the existing breakpoint list. * Breakpoints can be defined either in an associated editor or with breakpoint commands in the debugger when execution is stopped. PEP 523 envisioned an alternate non-trace implementation of 'go to next breakpoint' by a debugger going "as far as to dynamically rewrite bytecode prior to execution to inject e.g. breakpoints in the bytecode." https://www.python.org/dev/peps/pep-0523/#debugging A debugger doing this could either eliminate the other 'go' commands (easiest) or implement them by either setting temporary breakpoints or temporarily turning tracing on. I presume it should be possible to make bdb.Bdb use bytecode breakpoints or add a new class with a similar API. Then any bdb-based debugger to be modified to make the speedup available. -- Terry Jan Reedy ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PyCharm debugger became 40x faster on Python 3.6 thanks to PEP 523
On Sat, 25 Mar 2017 at 05:58 Serhiy Storchaka wrote: > On 25.03.17 12:04, Victor Stinner wrote: > > > https://blog.jetbrains.com/pycharm/2017/03/inside-the-debugger-interview-with-elizaveta-shashkova/ > > > > "What changed in Python 3.6 to allow this? > > > > The new frame evaluation API was introduced to CPython in PEP 523 and it > > allows to specify a per-interpreter function pointer to handle the > > evaluation of frames." > > > > Nice! > > Awesome! Any chance that pdb can utilize similar technique? Or this > doesn't make sense for pdb? > I guess it's possible. It probably depends on how you're using the debugger. It sounds like PyCharm is injecting bytecode for specified breakpoints and so I suspect the speed is only there when you press "debug" and are not stepping through line-by-line. Getting gdb to have the same level of sophistication might not be too bad as long as you keep the hook simple and you're okay injected new bytecode just before a frame begins execution. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PyCharm debugger became 40x faster on Python 3.6 thanks to PEP 523
On Sat, 25 Mar 2017 at 03:05 Victor Stinner wrote: > > https://blog.jetbrains.com/pycharm/2017/03/inside-the-debugger-interview-with-elizaveta-shashkova/ > > "What changed in Python 3.6 to allow this? > > The new frame evaluation API was introduced to CPython in PEP 523 and it > allows to specify a per-interpreter function pointer to handle the > evaluation of frames." > > Nice! > I just wanted to publicly thank Microsoft for paying for that PEP. :) It came out of the Pyjion work that Dino and I got to spend work time on. The hook is also used by the Python workload in Visual Studio 2017 Preview, so even if no JITs ever use the hook at least debuggers are finding it useful. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PyCharm debugger became 40x faster on Python 3.6 thanks to PEP 523
On 25.03.17 12:04, Victor Stinner wrote: https://blog.jetbrains.com/pycharm/2017/03/inside-the-debugger-interview-with-elizaveta-shashkova/ "What changed in Python 3.6 to allow this? The new frame evaluation API was introduced to CPython in PEP 523 and it allows to specify a per-interpreter function pointer to handle the evaluation of frames." Nice! Awesome! Any chance that pdb can utilize similar technique? Or this doesn't make sense for pdb? ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PyCharm debugger became 40x faster on Python 3.6 thanks to PEP 523
https://blog.jetbrains.com/pycharm/2017/03/inside-the-debugger-interview-with-elizaveta-shashkova/ "What changed in Python 3.6 to allow this? The new frame evaluation API was introduced to CPython in PEP 523 and it allows to specify a per-interpreter function pointer to handle the evaluation of frames." Nice! Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
