[issue13044] pdb throws AttributeError at end of debugging session

[Xavier de Gaye]

Xavier de Gaye  added the comment:

See also issue 33458 that deals with the same problem when pdb is run as 
'python -m pdb some_main.py' instead of by inserting a breakpoint with 
'pdb.set_trace()'.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13044] pdb throws AttributeError at end of debugging session

[Xavier de Gaye]

Xavier de Gaye  added the comment:

Actually the segfault can be avoided by running the garbage collection before 
_PyState_ClearModules() in PyImport_Cleanup() and one can trace the C 
destructor with the attached patch global_destructors.diff applied on top of PR 
6730.

_PyState_ClearModules() clears the state->modules_by_index list and causes the 
readline module to be non-functional. The same problem occurs in 
test_create_at_shutdown_without_encoding() in test_io.py when it is run with 
the io module: the C destructor fails before hitting the "LookupError: unknown 
encoding: ascii" error message because the garbage collection is made after 
_PyState_ClearModules() and the _io module is non-functional 
(PyState_FindModule() returns NULL).

So the destructors can be traced during finalization provided the tracer does 
not access any of the sys attributes that are set to None at the start of 
PyImport_Cleanup().  This is true for the globals of the modules that are only 
owned by sys.modules at the time 'weaklist' is about to be built in 
PyImport_Cleanup(). But it is not the case of '_ag' and that explains at last 
why the destructor was called and failed:
'_ag' is a global of the types module which is owned (imported) by the inspect 
module, which is owned (imported) by the bdb module which is owned by 
PyThreadState through its c_profileobj member. So the types module, and all 
modules imported directly or indirectly by pdb, can only be cleared at the end 
of PyImport_Cleanup() when 'weaklist' is browsed for the last modules to clear. 
At that point any module may be non-functional and tracing must not be allowed.

The global_destructors.diff patch ensures that no access is made by the pdb and 
bdb modules (but imported modules have not been checked) to the the sys 
attributes that are set to None at the start of PyImport_Cleanup():
* Pdb.lookupmodule() accesses sys.path and is used to set breakpoints so we 
forbid setting breakpoint when sys.path is None.
* All the values of sys.modules are set to None when 'weaklist' is built. This 
causes the lazy imports to fail and the patch removes them except the lazy 
import of pydoc in pdb.py because pydoc imports threading and when threading 
has been imported wait_for_thread_shutdown() in Py_FinalizeEx() runs the 
threading._shutdown() Python code which is traced (this happens in Python 3.6 
too whenever threading is imported). Since it is annoying to systematically 
trace this function and pydoc is a heavy module, pdb.help() is disabled on 
Python finalization.

--
Added file: https://bugs.python.org/file47581/global_destructors.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13044] pdb throws AttributeError at end of debugging session

[Xavier de Gaye]

Change by Xavier de Gaye :


Added file: https://bugs.python.org/file47578/lazy_import.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13044] pdb throws AttributeError at end of debugging session

[Xavier de Gaye]

Change by Xavier de Gaye :


Added file: https://bugs.python.org/file47579/gdb_backtrace.txt

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13044] pdb throws AttributeError at end of debugging session

[Xavier de Gaye]

Xavier de Gaye  added the comment:

Nosying Serhiy as this post attempts to answer one of the questions raised in 
msg315588 in issue 33328:
> Is it good that the debugger is enabled at the shutdown stage?

This post has four different sections showing that the successive and 
incremental attempts at fixing the current code to have a destructor traced in 
the shutdown stage may lead to a segfault.

Section 1 - Current status:
---
A run with Python 3.6.5 of the following code proposed in the previous 
msg176993 gives now a different output and the C destructor is still not 
traced, but now it is not even called:

1 class C:
2 def __del__(self):
3 print('deleted')
4
5 c = C()
6 import pdb; pdb.set_trace()
7 x = 1

$ python test_destructor.py
> test_destructor.py(7)()
-> x = 1
(Pdb) step
--Return--
> test_destructor.py(7)()->None
-> x = 1
(Pdb) step
--Call--
Exception ignored in: 
Traceback (most recent call last):
  File "/usr/lib/python3.6/types.py", line 27, in _ag
  File "/usr/lib/python3.6/bdb.py", line 53, in trace_dispatch
  File "/usr/lib/python3.6/bdb.py", line 85, in dispatch_call
  File "/usr/lib/python3.6/pdb.py", line 251, in user_call
  File "/usr/lib/python3.6/pdb.py", line 351, in interaction
  File "/usr/lib/python3.6/pdb.py", line 1453, in print_stack_entry
  File "/usr/lib/python3.6/bdb.py", line 394, in format_stack_entry
TypeError: 'NoneType' object is not callable

Section 2 - Fix using PR 6730:
--
PR 6730 in issue 33446 fixes a bug in the pdb module that leaks a reference to 
frame->f_locals through its curframe_locals attribute.
There is no major change when the current Python master branch is run with PR 
6730, the destructor is still not traced and not called.

Section 3 - Fix a bdb leak:
---
After returning from the '__main__' module, the Bdb.bdb instance keeps a 
reference to the module through its botframe attribute and prevents the 
corresponding frame from being deallocated at the end of 
_PyEval_EvalCodeWithName() (the frame reference count is 2 instead of 1).  The 
attached bdb_leak.diff patch fixes this, and when PR 6730 and bdb_leak.diff are 
run, the destructor is now traced but this fails with an exception in bdb:

$ /home/xavier/src/python/master/python test_destructor.py
> ./test_destructor.py(7)()
-> x = 1
(Pdb) step
--Return--
> ./test_destructor.py(7)()->None
-> x = 1
(Pdb) step
Exception ignored in: 
Traceback (most recent call last):
  File "test_destructor.py", line 3, in __del__
  File "test_destructor.py", line 3, in __del__
  File "/home/xavier/src/python/master/Lib/bdb.py", line 88, in trace_dispatch
  File "/home/xavier/src/python/master/Lib/bdb.py", line 115, in dispatch_line
  File "/home/xavier/src/python/master/Lib/pdb.py", line 262, in user_line
  File "/home/xavier/src/python/master/Lib/pdb.py", line 352, in interaction
  File "/home/xavier/src/python/master/Lib/pdb.py", line 1454, in 
print_stack_entry
  File "/home/xavier/src/python/master/Lib/bdb.py", line 544, in 
format_stack_entry
ImportError: sys.meta_path is None, Python is likely shutting down

Section 4 - Remove the lazy import:
---
The attached lazy_import.diff patch includes the changes in bdb_leak.diff patch 
and replaces the lazy imports causing the previous ImportError with imports at 
the start of the bdb module.  Running test_destructor.py fails now with a 
segfault:

$ /home/xavier/src/python/master/python test_destructor.py
> ./test_destructor.py(7)()
-> x = 1
(Pdb) step
--Return--
> ./test_destructor.py(7)()->None
-> x = 1
(Pdb) step
> ./test_destructor.py(3)__del__()
-> print('deleted')
Segmentation fault (core dumped)

The gdb back trace is attached at gdb_backtrace.txt.

Conclusion:
---
It seems that tracing should be prevented in the shutdown stage and that 
multiple little bugs have hidden that fact (when tracing is done with pdb).
Not sure if tracing with PyEval_SetTrace() could be allowed in that stage.
BTW coverage.py uses PyEval_SetTrace() and I have noticed that coverage.py does 
not trace either the destructor in test_destructor.py (nosying Ned).

--
keywords: +patch
nosy: +nedbat, serhiy.storchaka
Added file: https://bugs.python.org/file47577/bdb_leak.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13044] pdb throws AttributeError at end of debugging session

[Xavier de Gaye]

Xavier de Gaye  added the comment:

I can reproduce this bug with python 3.6.5 using akl's debug.py. The exception 
is now on 3.6.5:

(Pdb) next
--Call--
Exception ignored in: 
Traceback (most recent call last):
  File "/usr/lib/python3.6/types.py", line 27, in _ag
  File "/usr/lib/python3.6/bdb.py", line 53, in trace_dispatch
  File "/usr/lib/python3.6/bdb.py", line 85, in dispatch_call
  File "/usr/lib/python3.6/pdb.py", line 251, in user_call
  File "/usr/lib/python3.6/pdb.py", line 351, in interaction
  File "/usr/lib/python3.6/pdb.py", line 1453, in print_stack_entry
  File "/usr/lib/python3.6/bdb.py", line 394, in format_stack_entry
TypeError: 'NoneType' object is not callable

Issue 33328 is a duplicate of this issue.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13044] pdb throws AttributeError at end of debugging session

[Davin Potts]

Changes by Davin Potts :


--
nosy: +davin

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13044] pdb throws AttributeError at end of debugging session

[Charles F. Bearden]

Charles F. Bearden added the comment:

One further observation: the exception is thrown only if a breakpoint (apart 
from the call to pdb.set_trace) is set. If no breakpoint is set, the exception 
is not raised.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13044] pdb throws AttributeError at end of debugging session

[Charles F. Bearden]

Charles F. Bearden added the comment:

I can reproduce this bug on Ubuntu 14.04.2 LTS with Python 2.7.6 using akl's 
debug.py.

--
nosy: +cfbearden

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13044] pdb throws AttributeError at end of debugging session

[Xavier de Gaye]

Xavier de Gaye added the comment:

> On finalizing pdb can stop working at some time, but debugging on
> finalization stage can be still useful in certain cases.

Agreed that debugging on finalization stage is useful. Debugging on
finalization stage does not seem to work though:

===
class C:
def __del__(self):
print('deleted')

c = C()
import pdb; pdb.set_trace()
x = 1
===
$ python3 bar.py
> /tmp/bar.py(7)()
-> x = 1
(Pdb) step
--Return--
> /tmp/bar.py(7)()->None
-> x = 1
(Pdb) step
deleted
$
===


Maybe this is another issue.


> Xavier, your proposition noticeably changes current behavior as I understand
> it.  For now set_trace() works as breakpoint set up by program, you can go
> out of debugged function and it's very convenient from my perspective.

This proposition only sets the global trace function to None on
returning from the bottom frame, i.e. the oldest frame in the stack. So you can
still use  set_trace() as a hard-coded breakpoint and continue the debugging
session out of the debugged function.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13044] pdb throws AttributeError at end of debugging session

[Andrew Svetlov]

Andrew Svetlov added the comment:

Do we need to fix it at all?
On finalizing pdb can stop working at some time, but debugging on finalization 
stage can be still useful in certain cases.

Xavier, your proposition noticeably changes current behavior as I understand it.
For now set_trace() works as breakpoint set up by program, you can go out of 
debugged function and it's very convenient from my perspective.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13044] pdb throws AttributeError at end of debugging session

[Andrew Svetlov]

Changes by Andrew Svetlov :


--
nosy: +asvetlov

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13044] pdb throws AttributeError at end of debugging session

[Xavier de Gaye]

Xavier de Gaye added the comment:

The run, runeval and runcall methods run the debugging session in a
try/finally clause and set the global trace function to None in the
finally clause. But set_trace does not run in a try/finally, hence the
problem. A possible fix is to ensure that the bottom frame has a local
trace function and to set the global trace function to None when
returning from that frame. See how this is fixed at
http://code.google.com/p/pdb-clone/source/detail?r=ee22a278c743ea9449a3e7618acc7c33cb11bb26&name=2.7

--
nosy: +xdegaye

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13044] pdb throws AttributeError at end of debugging session

[Meador Inge]

Changes by Meador Inge :


--
nosy: +meador.inge

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13044] pdb throws AttributeError at end of debugging session

[Amaury Forgeot d'Arc]

Amaury Forgeot d'Arc  added the comment:

By modifying a bit the Python intepreter, I got this traceback:

Traceback (most recent call last):
  File "/home/amauryfa/python/cpython2.7/Lib/_weakrefset.py", line 38, in 
_remove
def _remove(item, selfref=ref(self)):
  File "/home/amauryfa/python/cpython2.7/Lib/bdb.py", line 50, in trace_dispatch
return self.dispatch_call(frame, arg)
  File "/home/amauryfa/python/cpython2.7/Lib/bdb.py", line 76, in dispatch_call
if not (self.stop_here(frame) or self.break_anywhere(frame)):
  File "/home/amauryfa/python/cpython2.7/Lib/bdb.py", line 147, in 
break_anywhere
return self.canonic(frame.f_code.co_filename) in self.breaks
  File "/home/amauryfa/python/cpython2.7/Lib/bdb.py", line 33, in canonic
canonic = os.path.abspath(filename)
AttributeError: 'NoneType' object has no attribute 'path'

Here is the gdb stack when the error is printed:

#0  PyErr_Print () at Python/pythonrun.c:1040
#1  0x00480c1d in handle_callback (object=) at 
Objects/weakrefobject.c:884
#2  PyObject_ClearWeakRefs (object=) at 
Objects/weakrefobject.c:966
#3  0x00423a37 in class_dealloc (op=0x77f68890) at 
Objects/classobject.c:193
#4  0x0046a7ed in tupledealloc (op=0x77f89410) at 
Objects/tupleobject.c:220
#5  0x00423afb in class_dealloc (op=0x77f7ed50) at 
Objects/classobject.c:194
#6  0x004270cb in instance_dealloc (inst=0x77f8a518) at 
Objects/classobject.c:670
#7  0x0044de47 in insertdict (mp=0x7f7800, key='environ', 
hash=-5347984860299468300, value=None)
at Objects/dictobject.c:530
#8  0x00450287 in PyDict_SetItem (op=
{'WTERMSIG': None, 'lseek': None, 'EX_IOERR': None, 'EX_NOHOST': None, 
'seteuid': None, 'pathsep': None, 'execle': None, 'major': None, '_Environ': 
None, 'fstatvfs': None, 'uname': None, 'kill': None, 'urandom': None, 'execlp': 
None, 'getegid': None, 'getresgid': None, 'EX_OSFILE': None, 'umask': None, 
'linesep': None, 'fchmod': None, 'lchown': None, 'setgid': None, 'tmpnam': 
None, 'devnull': None, 'EX_NOINPUT': None, 'makedev': None, 'fstat': None, 
'getlogin': None, 'O_CREAT': None, 'dup2': None, 'read': None, '__file__': 
None, 'getppid': None, 'fchown': None, 'getloadavg': None, 'WIFSTOPPED': None, 
'getpgrp': None, '_spawnvef': None, 'TMP_MAX': None, 'utime': None, 'execl': 
None, 'F_OK': None, '_make_stat_result': None, 'name': None, 'fsync': None, 
'tcsetpgrp': None, 'statvfs': None, 'setreuid': None, 'remove': None, 
'setegid': None, 'P_NOWAITO': None, '_copy_reg': None, 'execv': None, 'spawnv': 
None, 'spawnvpe': None, 'EX_OSERR': None, 'ttyname': None, 'pardir': None, 
 'tempnam': None, 'tmpfile': None, 'sep...(truncated), key='environ', 
value=None) at Objects/dictobject.c:775
#9  0x0045268e in _PyModule_Clear (m=) at 
Objects/moduleobject.c:138
#10 0x004bfa17 in PyImport_Cleanup () at Python/import.c:498
#11 0x004cd9ef in Py_Finalize () at Python/pythonrun.c:447

So, when the interpreter shuts down, the debugger is still active...
Python 3 does not seem to be affected by this issue. And indeed if I make 
UserDict a new-style class, I don't reproduce the error on 2.7 either.

A possible fix is to reset the trace function to None in Py_Finalize.

--
nosy: +amaury.forgeotdarc

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13044] pdb throws AttributeError at end of debugging session

[akl]

New submission from akl :

Using Python 2.7.1 on OpenBSD-current and 2.7.2 on Arch Linux, pdb throws a 
(harmless, it appears) exception at the end of a debugging session.  This does 
not happen for me using Python 2.5 or 2.6 or 3.2.

Console Session (on linux):

$ python2 --version
Python 2.7.2

$ python2 debug.py 
> /tmp/debug.py(4)()
-> a = 1
(Pdb) n
> /tmp/debug.py(5)()
-> b = 2
(Pdb) n
> /tmp/debug.py(6)()
-> sum = a + b
(Pdb) n
--Return--
> /tmp/debug.py(6)()->None
-> sum = a + b
(Pdb) n
Exception AttributeError: "'NoneType' object has no attribute 'path'" in 
 ignore

--
components: Library (Lib)
files: debug.py
messages: 144525
nosy: akl
priority: normal
severity: normal
status: open
title: pdb throws AttributeError at end of debugging session
type: behavior
versions: Python 2.7
Added file: http://bugs.python.org/file23242/debug.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com