[issue17922] Crash in clear_weakref
Jan Safranek added the comment: On 05/09/2013 09:07 AM, Jan Safranek wrote: Jan Safranek added the comment: On 05/07/2013 05:32 PM, Antoine Pitrou wrote: Jan, one possibility would be for Pegasus to stop unloading Python, it seems. It is always possibility. Actually, Pegasus plugin is just a shared object (.so) and the .so is linked with Python. Pegasus calls dlopen() and dlclose() on it. After dlclose(), the plugin is removed from memory. Unfortunately, libpython2.7.so stays loaded, at least /proc/XXX/mems says so. If there was a way to unload libpython2.7.so from memory too... libpython2.7.so is not unloaded because python extensions, e.g. /usr/lib64/python2.7/lib-dynload/_heapq.so depend on it. And _heapq.so was dlopenened by Python and it was not dlclosed - glibc does not unload it. It seems that Py_Finalize() does not even close opened shared objects. Isn't it a bug? Jan -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17922 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17922] Crash in clear_weakref
Jan Safranek added the comment: On 05/07/2013 06:06 PM, Antoine Pitrou wrote: a significant amount of static data inside CPython actually survives Py_Finalize :-/ As a solution, would it be possible to wipe all registered types in Py_Finalize? Jan -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17922 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17922] Crash in clear_weakref
Jan Safranek added the comment: On 05/07/2013 05:32 PM, Antoine Pitrou wrote: Jan, one possibility would be for Pegasus to stop unloading Python, it seems. It is always possibility. Actually, Pegasus plugin is just a shared object (.so) and the .so is linked with Python. Pegasus calls dlopen() and dlclose() on it. After dlclose(), the plugin is removed from memory. Unfortunately, libpython2.7.so stays loaded, at least /proc/XXX/mems says so. If there was a way to unload libpython2.7.so from memory too... Jan -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17922 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17922] Crash in clear_weakref
New submission from Jan Safranek: I have Python 2.7.4 running on Fedora Rawhide and I get segmentation fault with following backtrace: #0 0x7f73f69ca5f1 in clear_weakref (self=0x7f73ff515c00) at Objects/weakrefobject.c:56 #1 weakref_dealloc (self=0x7f73ff515c00) at Objects/weakrefobject.c:106 #2 0x7f73f698ea27 in PyList_SetItem (op=optimized out, i=optimized out, newitem=optimized out) at Objects/listobject.c:218 #3 0x7f73f69ba9db in add_subclass (type=type@entry=0x7f73e00456b0, base=optimized out) at Objects/typeobject.c:4155 #4 0x7f73f69c440e in PyType_Ready (type=type@entry=0x7f73e00456b0) at Objects/typeobject.c:4120 #5 0x7f73f69c6d4b in type_new (metatype=optimized out, args=optimized out, kwds=optimized out) at Objects/typeobject.c:2467 #6 0x7f73f69be7d3 in type_call (type=0x7f73f6cdad00 PyType_Type, args=0x7f73f61e1550, kwds=0x0) at Objects/typeobject.c:725 #7 0x7f73f6954833 in PyObject_Call (func=func@entry=0x7f73f6cdad00 PyType_Type, arg=arg@entry=0x7f73f61e1550, kw=kw@entry=0x0) at Objects/abstract.c:2529 #8 0x7f73f69553c9 in PyObject_CallFunctionObjArgs (callable=callable@entry=0x7f73f6cdad00 PyType_Type) at Objects/abstract.c:2760 #9 0x7f73f6a06bf3 in build_class (name=optimized out, bases=0x7f73f61e3910, methods=0x7f73e0045590) at Python/ceval.c:4632 #10 PyEval_EvalFrameEx (f=f@entry=0x7f73e0043a40, throwflag=throwflag@entry=0) at Python/ceval.c:1928 #11 0x7f73f6a0b46d in PyEval_EvalCodeEx (co=co@entry=0x7f73f61f50b0, globals=globals@entry=0x7f73e003bf00, locals=locals@entry=0x7f73e003bf00, args=args@entry=0x0, argcount=argcount@entry=0, kws=kws@entry=0x0, kwcount=kwcount@entry=0, defs=defs@entry=0x0, defcount=defcount@entry=0, closure=closure@entry=0x0) at Python/ceval.c:3253 #12 0x7f73f6a0b5a2 in PyEval_EvalCode (co=co@entry=0x7f73f61f50b0, globals=globals@entry=0x7f73e003bf00, locals=locals@entry=0x7f73e003bf00) at Python/ceval.c:667 #13 0x7f73f6a22cfc in PyImport_ExecCodeModuleEx (name=name@entry=0x7f73e003d760 warnings, co=co@entry=0x7f73f61f50b0, pathname=pathname@entry=0x7f73e003ac90 /usr/local/lib/python2.7/warnings.pyc) at Python/import.c:709 #14 0x7f73f6a2305e in load_source_module (name=0x7f73e003d760 warnings, pathname=0x7f73e003ac90 /usr/local/lib/python2.7/warnings.pyc, fp=optimized out) at Python/import.c:1099 #15 0x7f73f6a23f59 in import_submodule (mod=mod@entry=0x7f73f6cd2ec0 _Py_NoneStruct, subname=subname@entry=0x7f73e003d760 warnings, fullname=fullname@entry=0x7f73e003d760 warnings) at Python/import.c:2700 #16 0x7f73f6a24b93 in load_next (p_buflen=synthetic pointer, buf=0x7f73e003d760 warnings, p_name=synthetic pointer, altmod=0x7f73f6cd2ec0 _Py_NoneStruct, mod=0x7f73f6cd2ec0 _Py_NoneStruct) at Python/import.c:2515 #17 import_module_level (locals=optimized out, level=optimized out, fromlist=0x7f73f6cd2ec0 _Py_NoneStruct, globals=optimized out, name=0x0) at Python/import.c:2224 #18 PyImport_ImportModuleLevel (name=0x7f73ff54fbf4 warnings, globals=optimized out, locals=optimized out, fromlist=0x7f73f6cd2ec0 _Py_NoneStruct, level=optimized out) at Python/import.c:2288 #19 0x7f73f6a033af in builtin___import__ (self=optimized out, args=optimized out, kwds=optimized out) at Python/bltinmodule.c:49 ... #61 0x7f73f6a37dbc in initsite () at Python/pythonrun.c:721 #62 Py_InitializeEx (install_sigs=1) at Python/pythonrun.c:265 (full back trace is attached, it's quite long). (gdb) py-bt #10 Frame 0x7f4bf8043df0, for file /usr/lib64/python2.7/warnings.py, line 281, in module () class WarningMessage(object): #23 Frame 0x7f4bf803d300, for file /usr/lib64/python2.7/posixpath.py, line 17, in module () import warnings #36 Frame 0x7f4bf8024fc0, for file /usr/lib64/python2.7/os.py, line 49, in module () import posixpath as path #49 Frame 0x7f4bf801c520, for file /usr/lib64/python2.7/site.py, line 62, in module () import os I get the same crash with vanilla Python 2.7.4 without Fedora patches. Python 2.7.3 works well and doesn't crash. -- components: Interpreter Core files: full-bt.txt messages: 188629 nosy: jsafrane priority: normal severity: normal status: open title: Crash in clear_weakref type: crash versions: Python 2.7 Added file: http://bugs.python.org/file30161/full-bt.txt ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17922 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17922] Crash in clear_weakref
Jan Safranek added the comment: I can reproduce the crash in very unusual setup: 1. OpenPegasus (http://www.openpegasus.org/), for this bug we may consider it just a network daemon, listening on TCP port. When a request comes, it is eventually processed by a provider (= something like plugin). 2. cmpi-bindings ([1], [2]), which allows to write these plugins in Python + some other python modules, but IMHO not relevant (e.g. pywbem [3]) 1: https://github.com/kkaempf/cmpi-bindings 2: http://sourceforge.net/apps/mediawiki/pywbem/index.php?title=Provider_Home 3: http://sourceforge.net/apps/mediawiki/pywbem/index.php?title=Main_Page Now, if the Pegasus daemon gets a request, it calls cmpi-bindings, which creates embedded Python [4], loads the python plugin, and processes the request. If the plugin is idle for 15 minutes, it is unloaded by Pegasus (= the embedded Python is destroyed). So far everything works like charm. But, when new request arrives *after* the unload, Pegaasus calls cmpi-bindings again, which tries to create the embedded Python for second time and here I get the crash. [4]: python initialization/shutdown: https://github.com/kkaempf/cmpi-bindings/blob/master/src/target_python.c, TargetInitialize() and TargetCleanup(), some marcos are generated by swig from https://github.com/kkaempf/cmpi-bindings/blob/master/swig/cmpi.i I haven't been able to reproduce the crash with simpler setup (and I have tried, believe me). It is also possible that the Python initialization/shutdown in cmpi-bindings is wrong, but I am not able to find any bug here. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17922 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17922] Crash in clear_weakref
Jan Safranek added the comment: Bisecting Python mercurial repository, I found the patch which causes the crash: changeset: 80762:7e771f0363e2 branch: 2.7 parent: 80758:29627bd5b333 user:Antoine Pitrou solip...@pitrou.net date:Sat Dec 08 21:15:26 2012 +0100 summary: Issue #16602: When a weakref's target was part of a long deallocation chain, the object could remain reachable through its weakref even though its refcount had dropped to zero. If I revert the patch in Python 2.7.4, my setup works fine, without any crash. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17922 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17922] Crash in clear_weakref
Jan Safranek added the comment: First, have you seen the following message on that bug report: http://bugs.python.org/issue16602#msg177180 I'm reading it now... I searched for PyWeakref_GET_OBJECT in cmpi-bindings and both occurrences generated by SWIG and both look safe. Is it hidden/wrapped by any other macro? Sorry, I don't know much about python internals and extension development, I'm not author of cmpi-bindings. And I'm attaching stack trace with --with-pydebug. Debugging with gdb is quite a problem, I have gdb linked with distribution Python 2.7.4 and it doesn't cooperate with my custom built python, which I have in LD_LIBRARY_PATH (so Pegasus gets the right one when loading providers). -- Added file: http://bugs.python.org/file30163/full-bt.txt ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17922 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17922] Crash in clear_weakref
Jan Safranek added the comment:
Could you try to inspect the `self` variable inside weakref_dealloc,
especially `self-wr_object` and its Py_TYPE() value? Also, what is the
value of Py_REFCNT(self-wr_object)?
in weakref_dealloc at Objects/weakrefobject.c:106:
(gdb) p *self
$1 = {_ob_next = 0x0, _ob_prev = 0x0, ob_refcnt = 0, ob_type = 0x7fdb8ffc91a0
_PyWeakref_RefType}
(gdb) p *((PyWeakReference*)self)
$7 = {_ob_next = 0x0, _ob_prev = 0x0, ob_refcnt = 0, ob_type = 0x7fdb8ffc91a0
_PyWeakref_RefType, wr_object = 0x7fdb9c30bc00 swigpyobject_type.9541,
wr_callback = 0x0, hash = -1,
wr_prev = 0x0, wr_next = 0x0}
(gdb) p *((PyWeakReference*)self)-wr_object
$9 = {_ob_next = 0x0, _ob_prev = 0x0, ob_refcnt = 0, ob_type = 0x0}
If I am reading Py_TYPE right, Py_TYPE(self-wr_object) must be 0 (=NULL).
swigpyobject_type.9541 seems to be PyTypeObject generated by SWIG in
cmpi-bindings, I'll dig into it. Please let me know if there is anything
suspicious or worth checking.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17922
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17922] Crash in clear_weakref
Jan Safranek added the comment: Right. But this is an embedded interpreter, and SWIG does not call PyType_Ready() again; the old type is returned instead. Python crashes in Py_Initialize(). SWIG_init() is called right after it. So even if SWIG calls PyType_Ready, it would be too late. Why python remembers SWIG types after Py_Finalize() in the first place? I want to destroy it and start with fresh instance. Jan -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17922 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
