[issue9263] Try to print repr() when an C-level assert fails (in the garbage collector, beyond?)

[STINNER Victor]

STINNER Victor  added the comment:


New changeset c9b3fc6b59b625c36c31ad437253e7140938af1a by Victor Stinner in 
branch '3.6':
bpo-9263: _PyObject_Dump() detects freed memory (GH-10061) (GH-10662) (GH-10663)
https://github.com/python/cpython/commit/c9b3fc6b59b625c36c31ad437253e7140938af1a


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9263] Try to print repr() when an C-level assert fails (in the garbage collector, beyond?)

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +9916

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9263] Try to print repr() when an C-level assert fails (in the garbage collector, beyond?)

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 95036ea25d47f0081bda2ba96ea327f3375cb6a4 by Victor Stinner in 
branch '3.7':
[3.7] bpo-9263: _PyObject_Dump() detects freed memory (GH-10061) (GH-10662)
https://github.com/python/cpython/commit/95036ea25d47f0081bda2ba96ea327f3375cb6a4


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9263] Try to print repr() when an C-level assert fails (in the garbage collector, beyond?)

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +9915

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9263] Try to print repr() when an C-level assert fails (in the garbage collector, beyond?)

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 2cf5d32fd9e61488e8b0be55a2e92a752ba8b06b by Victor Stinner in 
branch 'master':
bpo-9263: Fix _PyObject_Dump() for freed object (#10661)
https://github.com/python/cpython/commit/2cf5d32fd9e61488e8b0be55a2e92a752ba8b06b


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9263] Try to print repr() when an C-level assert fails (in the garbage collector, beyond?)

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +9914

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9263] Try to print repr() when an C-level assert fails (in the garbage collector, beyond?)

[Dave Malcolm]

Dave Malcolm  added the comment:

Thanks!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9263] Try to print repr() when an C-level assert fails (in the garbage collector, beyond?)

[STINNER Victor]

STINNER Victor  added the comment:

I pushed the change and even more, so I consider that the issue can now be 
closed... 8 years later! Thank you very much Dave Malcolm for this nice idea, 
and for its implementation. Thanks Bohuslav "Slavek" Kabrda for the rebase in 
2013, and thanks to my colleagues who rebased the patch frequently since 2013 
in the Fedora package!

Maybe some people (like me?) want to use _PyObject_ASSERT() in more places, but 
I consider that we don't need to leave this issue open just for that.


I took the 00170-gc-assertions.patch rebased on Python 3.7.1 by my colleagues 
for the Fedora package, and I rebased it on master. I modified more functions 
in object.c and typeobject.c to use _PyObject_ASSERT(). I tried to not replace 
all assert(), but only when it's revelant.

I added code to detect if the object memory has been freed to avoid derefering 
0xdbdbdbdbdbdbdbdb pointers which is very likely to cause a segmantation fault. 
It should reduce the risk of crash when dumping the faulty object.


Dave Malcolm:
> - Only tested on gcc-4.4.3 so far; the __STRING(expr) and __PRETTY_FUNCTION__ 
> look non-portable.

I used Py_STRINGIFY() and __func__ in the final patch. __func__ is part of the 
C99 standard which is now required since Python 3.6: see PEP 7.


Dave Malcolm:
> - no test case; I thought about using ctypes to extract PyObject_IncRef from 
> the implementation, but this is likely to lead to brittle test cases.  
> Alternatively, is xxmodule to be used for this kind of thing?

I reworked the unit test to not use ctypes, but write the crashing code in C 
instead.


Antoine Pitrou:
> How about turning these asserts into Py_FatalError()s and then enabling 
> Victor's faulthandler extension?

Done.

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9263] Try to print repr() when an C-level assert fails (in the garbage collector, beyond?)

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 50fe3f8913c503e63f4cfb8ddcf8641ef7ad0722 by Victor Stinner in 
branch 'master':
bpo-9263: _PyXXX_CheckConsistency() use _PyObject_ASSERT() (GH-10108)
https://github.com/python/cpython/commit/50fe3f8913c503e63f4cfb8ddcf8641ef7ad0722


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9263] Try to print repr() when an C-level assert fails (in the garbage collector, beyond?)

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 0862505a0377c12e8004b2eb8de0555f26ce9530 by Victor Stinner in 
branch 'master':
bpo-9263: Use _PyObject_ASSERT() in typeobject.c (GH-10111)
https://github.com/python/cpython/commit/0862505a0377c12e8004b2eb8de0555f26ce9530


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9263] Try to print repr() when an C-level assert fails (in the garbage collector, beyond?)

[STINNER Victor]

STINNER Victor  added the comment:


New changeset a4b2bc70f69d93d8252861b455052c051b7167ae by Victor Stinner in 
branch 'master':
bpo-9263: Use _PyObject_ASSERT() in gcmodule.c (GH-10112)
https://github.com/python/cpython/commit/a4b2bc70f69d93d8252861b455052c051b7167ae


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9263] Try to print repr() when an C-level assert fails (in the garbage collector, beyond?)

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 24702044afb1d4ad7568bf6aa7450b14dc44a38f by Victor Stinner in 
branch 'master':
bpo-9263: Use _PyObject_ASSERT() in object.c (GH-10110)
https://github.com/python/cpython/commit/24702044afb1d4ad7568bf6aa7450b14dc44a38f


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9263] Try to print repr() when an C-level assert fails (in the garbage collector, beyond?)

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 3ec9af75f6825a32f369ee182a388c365db241b6 by Victor Stinner in 
branch 'master':
bpo-9263: _Py_NegativeRefcount() use _PyObject_AssertFailed() (GH-10109)
https://github.com/python/cpython/commit/3ec9af75f6825a32f369ee182a388c365db241b6


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9263] Try to print repr() when an C-level assert fails (in the garbage collector, beyond?)

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +9444

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9263] Try to print repr() when an C-level assert fails (in the garbage collector, beyond?)

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +9443

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9263] Try to print repr() when an C-level assert fails (in the garbage collector, beyond?)

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +9442

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9263] Try to print repr() when an C-level assert fails (in the garbage collector, beyond?)

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +9441

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9263] Try to print repr() when an C-level assert fails (in the garbage collector, beyond?)

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +9440

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9263] Try to print repr() when an C-level assert fails (in the garbage collector, beyond?)

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 626bff856840f471e98ec627043f780c111a063d by Victor Stinner in 
branch 'master':
bpo-9263: Dump Python object on GC assertion failure (GH-10062)
https://github.com/python/cpython/commit/626bff856840f471e98ec627043f780c111a063d


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9263] Try to print repr() when an C-level assert fails (in the garbage collector, beyond?)

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9263] Try to print repr() when an C-level assert fails (in the garbage collector, beyond?)

[STINNER Victor]

STINNER Victor  added the comment:

00170-gc-assertions.patch is used in the Fedora package of Python.

I converted the patch to a pull request: PR 10062.

My PR only uses the new assertion macro once. I plan to write more changes to 
use the new assertion macros in more places, once the first PR is merged.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9263] Try to print repr() when an C-level assert fails (in the garbage collector, beyond?)

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +9398

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9263] Try to print repr() when an C-level assert fails (in the garbage collector, beyond?)

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 82af0b63b07aa8d92b50098e382b458143cfc677 by Victor Stinner in 
branch 'master':
bpo-9263: _PyObject_Dump() detects freed memory (GH-10061)
https://github.com/python/cpython/commit/82af0b63b07aa8d92b50098e382b458143cfc677


--
nosy: +vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9263] Try to print repr() when an C-level assert fails (in the garbage collector, beyond?)

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +9397

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9263] Try to print repr() when an C-level assert fails (in the garbage collector, beyond?)

[Bohuslav Slavek Kabrda]

Bohuslav Slavek Kabrda added the comment:

I'm currently patching Python 3.3.2 with this, so I thought it might be nice to 
attach an up-to-date patch. The only notable difference is that I added 
self.preclean() at the beginning of test_refcount_errors - without it, running 
test suite produced a huge number of these lines:

Exception AttributeError: 'GCCallbackTests' object has no attribute 'visit' 
in bound method GCCallbackTests.cb1 of __main__.GCCallbackTests 
testMethod=test_refcount_errors ignored

--
nosy: +bkabrda
Added file: http://bugs.python.org/file30532/00170-gc-assertions.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9263
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9263] Try to print repr() when an C-level assert fails (in the garbage collector, beyond?)

[Dave Malcolm]

Changes by Dave Malcolm dmalc...@redhat.com:


--
nosy: +loewis

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9263
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9263] Try to print repr() when an C-level assert fails (in the garbage collector, beyond?)

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

How about turning these asserts into Py_FatalError()s and then enabling 
Victor's faulthandler extension?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9263
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9263] Try to print repr() when an C-level assert fails (in the garbage collector, beyond?)

[Nick Coghlan]

Nick Coghlan ncogh...@gmail.com added the comment:

I'd suggest calling Py_FatalError rather than calling abort() directly in 
_PyObject_AssertFailed, but otherwise this looks like a nice improvement over 
standard C asserts for state invariants that may be broken by buggy C 
extensions.

For the tests, take a look at test.script_helper - it provides some convenience 
wrappers for spawning subprocesses for tests that would cause problems if run 
in the current process.

--
nosy: +ncoghlan

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9263
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9263] Try to print repr() when an C-level assert fails (in the garbage collector, beyond?)

[Dave Malcolm]

Dave Malcolm dmalc...@redhat.com added the comment:

Attaching updated version of the patch.

I've added a selftest which (in a sacrificial subprocess) abuses ctypes to 
break an ob_refcnt, and then triggers a garbage collection.

I also changed the printing to stderr to directly use fprintf and fflush to 
ensure that data leaves the process before abort kills it (not sure if this is 
a cross-platform or unicode no-no, though).

--
Added file: 
http://bugs.python.org/file20343/py3k-objdump-on-gcmodule-assertions-2011-01-10-001.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9263
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9263] Try to print repr() when an C-level assert fails (in the garbage collector, beyond?)

[Dave Malcolm]

Dave Malcolm dmalc...@redhat.com added the comment:

As above, but I added an extra call to fflush in case the call to 
_PyObject_Dump leads to a segfault.

--
Added file: 
http://bugs.python.org/file20344/py3k-objdump-on-gcmodule-assertions-2011-01-10-002.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9263
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9263] Try to print repr() when an C-level assert fails (in the garbage collector, beyond?)

[Antoine Pitrou]

Changes by Antoine Pitrou pit...@free.fr:


--
versions: +Python 3.3 -Python 3.2

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9263
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9263] Try to print repr() when an C-level assert fails (in the garbage collector, beyond?)

[Dave Malcolm]

Changes by Dave Malcolm dmalc...@redhat.com:


--
nosy: +pitrou

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9263
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9263] Try to print repr() when an C-level assert fails (in the garbage collector, beyond?)

[Dave Malcolm]

Dave Malcolm dmalc...@redhat.com added the comment:

Attaching a simplified version of the patch; I got rid of the callbacks.

Still doesn't have test cases.

I suspect that the use of __STRING and __PRETTY_FUNCTION__ may be compatibility 
issues.  I believe that __FILE__ and __LINE__ and standard C though.

--
Added file: 
http://bugs.python.org/file19777/py3k-objdump-on-gcmodule-assertions-2010-11-22-001.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9263
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9263] Try to print repr() when an C-level assert fails (in the garbage collector, beyond?)

[Dave Malcolm]

New submission from Dave Malcolm dmalc...@redhat.com:

Modules/gcmodule.c contains various assertions which can fail due to reference 
counting errors elsewhere in either python, or an extension module.  These can 
be difficult to track down.

In the hope of maximizing the information from crash reports, the attached 
patch (against py3k) introduces a new assertion macro to Objects/object.h and 
Objects.c, which provides a richer debug message.  In particular, it identifies 
which object has the issue, and can more clearly spell out the problem.

The patch replaces all uses of assert() in Modules/gcmodule.c for which a 
specific object has an issue (e.g. bogus reference count).

The implementation may play somewhat fast-and-loose with rules about object 
invariants: you might have an only partially valid object, but the process is 
about to abort, so it seems acceptable to try to glean extra information on 
stderr.  (This may turn an abort into a segfault, of course)

Caveats:
- exact name of the API probably could be better
- I don't yet have a specific use for the callback idea; I was thinking of 
trying to display all objects that reference that object.  Might need a void* 
closure to be useful.  Might be a useless complication.
- Only tested on gcc-4.4.3 so far; the __STRING(expr) and __PRETTY_FUNCTION__ 
look non-portable.
- no test case; I thought about using ctypes to extract PyObject_IncRef from 
the implementation, but this is likely to lead to brittle test cases.  
Alternatively, is xxmodule to be used for this kind of thing?

Thoughts?

--
components: Interpreter Core
files: py3k-repr-on-gcmodule-assertions.patch
keywords: patch
messages: 110341
nosy: dmalcolm
priority: normal
severity: normal
stage: patch review
status: open
title: Try to print repr() when an C-level assert fails (in the garbage 
collector, beyond?)
type: feature request
versions: Python 3.2
Added file: 
http://bugs.python.org/file18007/py3k-repr-on-gcmodule-assertions.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9263
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com