[issue36389] Add gc.enable_object_debugger(): detect corrupted Python objects in the GC

Change by STINNER Victor : -- pull_requests: +16367 pull_request: https://github.com/python/cpython/pull/16816 ___ Python tracker ___

[issue36389] Add gc.enable_object_debugger(): detect corrupted Python objects in the GC

STINNER Victor added the comment: New changeset f82ce5b1b12873b65927149a016be6a7c65e979d by Victor Stinner in branch '3.8': [3.8] bpo-36389: Backport debug enhancements from master (GH-16796) https://github.com/python/cpython/commit/f82ce5b1b12873b65927149a016be6a7c65e979d --

[issue36389] Add gc.enable_object_debugger(): detect corrupted Python objects in the GC

Change by STINNER Victor : -- pull_requests: +16352 pull_request: https://github.com/python/cpython/pull/16796 ___ Python tracker ___

[issue36389] Add gc.enable_object_debugger(): detect corrupted Python objects in the GC

STINNER Victor added the comment: New changeset 7775349895088a7ae68cecf0c74cf817f15e2c74 by Victor Stinner in branch 'master': bpo-36389: Add newline to _PyObject_AssertFailed() (GH-16629) https://github.com/python/cpython/commit/7775349895088a7ae68cecf0c74cf817f15e2c74 --

[issue36389] Add gc.enable_object_debugger(): detect corrupted Python objects in the GC

Change by STINNER Victor : -- pull_requests: +16215 pull_request: https://github.com/python/cpython/pull/16629 ___ Python tracker ___

[issue36389] Add gc.enable_object_debugger(): detect corrupted Python objects in the GC

STINNER Victor added the comment: New changeset 60ec6efd96d95476fe5e38c491491add04f026e5 by Victor Stinner in branch 'master': bpo-36389: Fix _PyBytesWriter in release mode (GH-16624) https://github.com/python/cpython/commit/60ec6efd96d95476fe5e38c491491add04f026e5 --

[issue36389] Add gc.enable_object_debugger(): detect corrupted Python objects in the GC

Change by STINNER Victor : -- pull_requests: +16211 pull_request: https://github.com/python/cpython/pull/16624 ___ Python tracker ___

[issue36389] Add gc.enable_object_debugger(): detect corrupted Python objects in the GC

STINNER Victor added the comment: New changeset 6876257eaabdb30f27ebcbd7d2557278ce2e5705 by Victor Stinner in branch 'master': bpo-36389: _PyObject_CheckConsistency() available in release mode (GH-16612) https://github.com/python/cpython/commit/6876257eaabdb30f27ebcbd7d2557278ce2e5705

[issue36389] Add gc.enable_object_debugger(): detect corrupted Python objects in the GC

STINNER Victor added the comment: I created bpo-38392 "Ensure that objects entering the GC are valid". -- ___ Python tracker ___

[issue36389] Add gc.enable_object_debugger(): detect corrupted Python objects in the GC

STINNER Victor added the comment: Update: I added an assertion which should help to detect some kind of bugs in debug mode: commit d91d4de31745fc1ed4c7e6c208917827c9c472b6 Author: Victor Stinner Date: Mon Sep 9 17:44:59 2019 +0200 bpo-38070: visit_decref() calls _PyObject_IsFreed()

[issue36389] Add gc.enable_object_debugger(): detect corrupted Python objects in the GC

Change by STINNER Victor : -- pull_requests: +16200 pull_request: https://github.com/python/cpython/pull/16612 ___ Python tracker ___

[issue36389] Add gc.enable_object_debugger(): detect corrupted Python objects in the GC

STINNER Victor added the comment: Ah by the way, this issue was mostly motivated by a customer issue, but the bug disappeared from customer's production. Moreover, Python 3.8 now allows to use debug build without having to recompile all C extensions:

[issue36389] Add gc.enable_object_debugger(): detect corrupted Python objects in the GC

STINNER Victor added the comment: I opened this issue, because I was convinced that it would be easy to implement checks faster than gc.setthreshold(), but I failed to write efficient tests which detect the bugs that I listed above. My approach was basically to check all objects tracked by

[issue36389] Add gc.enable_object_debugger(): detect corrupted Python objects in the GC

STINNER Victor added the comment: New changeset 0810fa79885276114d1a94e2ce61da367ebb1ffc by Victor Stinner in branch 'master': bpo-36389: Cleanup gc.set_threshold() (GH-12844) https://github.com/python/cpython/commit/0810fa79885276114d1a94e2ce61da367ebb1ffc --

[issue36389] Add gc.enable_object_debugger(): detect corrupted Python objects in the GC

Change by STINNER Victor : -- pull_requests: +12769 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue36389] Add gc.enable_object_debugger(): detect corrupted Python objects in the GC

STINNER Victor added the comment: New changeset 0fc91eef34a1d9194904fa093c9fbd711af0f26c by Victor Stinner in branch 'master': bpo-36389: Add _PyObject_CheckConsistency() function (GH-12803) https://github.com/python/cpython/commit/0fc91eef34a1d9194904fa093c9fbd711af0f26c --

[issue36389] Add gc.enable_object_debugger(): detect corrupted Python objects in the GC

Change by STINNER Victor : -- pull_requests: +12730 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue36389] Add gc.enable_object_debugger(): detect corrupted Python objects in the GC

STINNER Victor added the comment: New changeset 9e23f0a27cb8bf6e4ea1d2aef36a91502282bbc9 by Victor Stinner in branch '3.7': [3.7] bpo-36389: _PyObject_IsFreed() now also detects uninitialized memory (GH-12770) (GH-12788)

[issue36389] Add gc.enable_object_debugger(): detect corrupted Python objects in the GC

Change by STINNER Victor : -- pull_requests: +12715 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue36389] Add gc.enable_object_debugger(): detect corrupted Python objects in the GC

STINNER Victor added the comment: Currently, I'm using the following patch to try to detect bpo-33803 bug using my GC object debugger. It's still a work-in-progress. diff --git a/Lib/site.py b/Lib/site.py index ad1146332b..e184080b19 100644 --- a/Lib/site.py +++ b/Lib/site.py @@ -638,3

[issue36389] Add gc.enable_object_debugger(): detect corrupted Python objects in the GC

STINNER Victor added the comment: New changeset 4c409beb4c360a73d054f37807d3daad58d1b567 by Victor Stinner in branch 'master': bpo-36389: Change PyMem_SetupDebugHooks() constants (GH-12782) https://github.com/python/cpython/commit/4c409beb4c360a73d054f37807d3daad58d1b567 --

[issue36389] Add gc.enable_object_debugger(): detect corrupted Python objects in the GC

Change by STINNER Victor : -- pull_requests: +12709 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue36389] Add gc.enable_object_debugger(): detect corrupted Python objects in the GC

Change by STINNER Victor : -- pull_requests: +12707 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue36389] Add gc.enable_object_debugger(): detect corrupted Python objects in the GC

STINNER Victor added the comment: New changeset 2b00db68554422ec37faba2a80179a0172df6349 by Victor Stinner in branch 'master': bpo-36389: _PyObject_IsFreed() now also detects uninitialized memory (GH-12770) https://github.com/python/cpython/commit/2b00db68554422ec37faba2a80179a0172df6349

[issue36389] Add gc.enable_object_debugger(): detect corrupted Python objects in the GC

STINNER Victor added the comment: bpo-33803 bug can be reintroduced using the following patch: diff --git a/Python/hamt.c b/Python/hamt.c index 67af04c437..67da8ec22c 100644 --- a/Python/hamt.c +++ b/Python/hamt.c @@ -2478,8 +2478,10 @@ hamt_alloc(void) if (o == NULL) { return

[issue36389] Add gc.enable_object_debugger(): detect corrupted Python objects in the GC

Change by STINNER Victor : -- pull_requests: +12699 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue36389] Add gc.enable_object_debugger(): detect corrupted Python objects in the GC

STINNER Victor added the comment: Do you think that a gc.is_object_debugger_enabled() function would be needed? The tracemalloc module has 3 functions: * start(), stop() * is_tracing() The faulthandler module has 3 functions: * enable() / disable() * is_enabled() --

[issue36389] Add gc.enable_object_debugger(): detect corrupted Python objects in the GC

STINNER Victor added the comment: I modified my PR to reuse tp_traverse. Inada-san: would you mind to review my change? -- ___ Python tracker ___

[issue36389] Add gc.enable_object_debugger(): detect corrupted Python objects in the GC

STINNER Victor added the comment: > I don't think calling APIs like _PyDict_CheckConsistency() is super useful. I looked a multiple old issues which contain "visit_decref". Most of them are really strange crashes and were closed with a message like "we don't have enough info to debug,

[issue36389] Add gc.enable_object_debugger(): detect corrupted Python objects in the GC

Inada Naoki added the comment: I don't think calling APIs like _PyDict_CheckConsistency() is super useful. Can the PR find bugs like issue33803 quickly? I think calling tp_traverse is better. static int check_object(PyObject *obj, void *unused) { _PyObject_ASSERT(obj, Py_REFCNT(obj) >

[issue36389] Add gc.enable_object_debugger(): detect corrupted Python objects in the GC

STINNER Victor added the comment: > It is better to not use assert(foo && bar). Use instead two separate asserts: > assrte(foo) and assert(bar). Hum, I looked at my PR and I'm not sure that I added such new assertion. Note: "assert" on calling assert(_PyDict_CheckConsistency(mp)) is only

[issue36389] Add gc.enable_object_debugger(): detect corrupted Python objects in the GC

Serhiy Storchaka added the comment: It is better to not use assert(foo && bar). Use instead two separate asserts: assrte(foo) and assert(bar). -- ___ Python tracker ___

[issue36389] Add gc.enable_object_debugger(): detect corrupted Python objects in the GC

STINNER Victor added the comment: I'm not sure if I should include an unit test. WIP patch for that: diff --git a/Modules/_testcapimodule.c b/Modules/_testcapimodule.c index 350ef77163..9c0d0cf41a 100644 --- a/Modules/_testcapimodule.c +++ b/Modules/_testcapimodule.c @@ -4718,6 +4718,18 @@

[issue36389] Add gc.enable_object_debugger(): detect corrupted Python objects in the GC

STINNER Victor added the comment: Hum, _PyType_CheckConsistency() fails on the following assertion during Python finalization: ASSERT(type->tp_mro != NULL && PyTuple_Check(type->tp_mro)); Error: --- /home/vstinner/prog/python/master/python: No module named asyncio.__main__; 'asyncio'

[issue36389] Add gc.enable_object_debugger(): detect corrupted Python objects in the GC

Change by STINNER Victor : -- nosy: +inada.naoki ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue36389] Add gc.enable_object_debugger(): detect corrupted Python objects in the GC

Change by STINNER Victor : -- keywords: +patch pull_requests: +12432 stage: -> patch review ___ Python tracker ___ ___

[issue36389] Add gc.enable_object_debugger(): detect corrupted Python objects in the GC

New submission from STINNER Victor : That's the follow-up of a thread that I started on python-dev in June 2018: [Python-Dev] Idea: reduce GC threshold in development mode (-X dev) https://mail.python.org/pipermail/python-dev/2018-June/153857.html When an application crash during a