[issue31061] asyncio segfault when using threadpool and "_asyncio" native module

Alexander Mohr added the comment: hmm, may be my fault due to docker image tagging issue. Will redeploy and update if the issue persists. If I don't reply again sorry for the noise. -- ___ Python tracker

[issue31061] asyncio segfault when using threadpool and "_asyncio" native module

Alexander Mohr added the comment: bad news, I just got a crash in the same place (updating defaultdict) after running for a week with the fixes from this and inada naoki's patches. I think the threadpool may be leaking threads too as I had > 40 threads after running for a week when I use no

[issue31061] asyncio segfault when using threadpool and "_asyncio" native module

Yury Selivanov added the comment: Thank you, Alexander. Bugs like this are notoriously hard to fix, you saved us from hours, if not days, of debugging. -- resolution: -> fixed stage: -> resolved status: open -> closed ___ Python tracker

[issue31061] asyncio segfault when using threadpool and "_asyncio" native module

Alexander Mohr added the comment: I've verified that this along with the changes in 31095 resolve the crashes I've been seeing in our production environment -- ___ Python tracker

[issue31061] asyncio segfault when using threadpool and "_asyncio" native module

INADA Naoki added the comment: New changeset f142e85d22ba135d5205280240f3a2fe1df2649f by INADA Naoki in branch '3.6': bpo-31061: fix crash in asyncio speedup module (GH-2984) https://github.com/python/cpython/commit/f142e85d22ba135d5205280240f3a2fe1df2649f --

[issue31061] asyncio segfault when using threadpool and "_asyncio" native module

Changes by INADA Naoki : -- pull_requests: +3023 ___ Python tracker ___ ___

[issue31061] asyncio segfault when using threadpool and "_asyncio" native module

INADA Naoki added the comment: New changeset de34cbe9cdaaf7b85fed86f99c2fd071e1a7b1d2 by INADA Naoki (Alexander Mohr) in branch 'master': bpo-31061: fix crash in asyncio speedup module (GH-2966) https://github.com/python/cpython/commit/de34cbe9cdaaf7b85fed86f99c2fd071e1a7b1d2 --

[issue31061] asyncio segfault when using threadpool and "_asyncio" native module

Alexander Mohr added the comment: ok, created: https://github.com/python/cpython/pull/2966 there are some other deallocs in there, mind verifying the rest? -- pull_requests: +3014 ___ Python tracker

[issue31061] asyncio segfault when using threadpool and "_asyncio" native module

Yury Selivanov added the comment: Alexander, I think this is it, you found it! Can you open a PR against master with a fix? I'll review & merge it. -- ___ Python tracker

[issue31061] asyncio segfault when using threadpool and "_asyncio" native module

Changes by Antoine Pitrou : -- stage: resolved -> ___ Python tracker ___ ___

[issue31061] asyncio segfault when using threadpool and "_asyncio" native module

Alexander Mohr added the comment: I'm hoping this is the fix: --- Modules/_asynciomodule.c.orig 2017-07-31 12:16:16.0 -0700 +++ Modules/_asynciomodule.c2017-07-31 13:08:52.0 -0700 @@ -953,15 +953,18 @@ FutureObj_dealloc(PyObject *self) { FutureObj *fut =

[issue31061] asyncio segfault when using threadpool and "_asyncio" native module

Alexander Mohr added the comment: another core had a different gc object: $1 = {ob_base = {ob_base = {_ob_next = 0x7f801eac3158, _ob_prev = 0x7f801eab95a0, ob_refcnt = 41, ob_type = 0x7f80238e76e0 }, ob_size = 0}, tp_name = 0x7f801e8967af "_asyncio.Task", tp_basicsize = 128, tp_itemsize = 0,

[issue31061] asyncio segfault when using threadpool and "_asyncio" native module

Alexander Mohr added the comment: oh, so this is looking like an asyncio issue, the "gc" that is causing the crash is: (gdb) print *FROM_GC(gc)->ob_type $8 = {ob_base = {ob_base = {_ob_next = 0x7f80c8aafc88, _ob_prev = 0x7f80c8aafd00, ob_refcnt = 7, ob_type = 0x7f80cd8c86e0 }, ob_size = 0},

[issue31061] asyncio segfault when using threadpool and "_asyncio" native module

Alexander Mohr added the comment: btw got slightly difference stacktrace on second core file -- Added file: http://bugs.python.org/file47051/python crash2.txt ___ Python tracker

[issue31061] asyncio segfault when using threadpool and "_asyncio" native module

Alexander Mohr added the comment: hmm, how would I do that? btw I'm not 100% sure this is due to asyncio. -- ___ Python tracker ___

[issue31061] asyncio segfault when using threadpool and "_asyncio" native module

Alexander Mohr added the comment: this is the comment on the assert: /* Python's cyclic gc should never see an incoming refcount * of 0: if something decref'ed to 0, it should have been * deallocated immediately at that time. * Possible cause (if the assert triggers): a tp_dealloc *

[issue31061] asyncio segfault when using threadpool and "_asyncio" native module

Yury Selivanov added the comment: >ok got a full debug core file, let me know what other information I can >provide. Thank you. Does the crash happen with python versions of Tasks and Future? -- ___ Python tracker

[issue31061] asyncio segfault when using threadpool and "_asyncio" native module

Alexander Mohr added the comment: ok got a full debug core file, let me know what other information I can provide. -- status: closed -> open Added file: http://bugs.python.org/file47049/python crash.txt ___ Python tracker

[issue31061] asyncio segfault when using threadpool and "_asyncio" native module

Yury Selivanov added the comment: Thank you for trying to get to the bottom of this! -- ___ Python tracker ___

[issue31061] asyncio segfault when using threadpool and "_asyncio" native module

Alexander Mohr added the comment: the problem with this crash is that it only happens periodically in our prod environment :( If I try running the exact same docker container with the same inputs locally it doesn't reproduce, so frustrating. I've created a whole workflow now for deploying

[issue31061] asyncio segfault when using threadpool and "_asyncio" native module

Yury Selivanov added the comment: > so looks like disabling the _asyncio module just caused the crash to happen > less often, closing and will continue investigating after a get a core file Interesting. So it crashes even with pure Python Future/Task implementations? Another idea to debug

[issue31061] asyncio segfault when using threadpool and "_asyncio" native module

Alexander Mohr added the comment: so looks like disabling the _asyncio module just caused the crash to happen less often, closing and will continue investigating after a get a core file -- stage: -> resolved status: open -> closed ___ Python

[issue31061] asyncio segfault when using threadpool and "_asyncio" native module

Yury Selivanov added the comment: Interesting. A code to reproduce this crash would help tremendously. So far this is the first time I hear about this crash. -- nosy: +inada.naoki ___ Python tracker

[issue31061] asyncio segfault when using threadpool and "_asyncio" native module

Alexander Mohr added the comment: btw I've seen this issue in 3.5.2 + 3.6.2 on debian jessie + stretch -- ___ Python tracker ___

[issue31061] asyncio segfault when using threadpool and "_asyncio" native module

New submission from Alexander Mohr: I have a project in a prod environment which heavily uses asyncio and a threadpool. It uses the threadpool to run CPU heavy tasks (in this case populating a defaultdict) to avoid blocking the main thread (no async code in thread). For some time now my