[issue33608] [subinterpreters] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.

Eric Snow added the comment: Thanks for taking a look Victor! That info is helpful. It will likely be a few days before I can sort this out. Once I have addressed the problem I'll re-merge. I plan on using the "buildbot-custom" branch to make sure the buildbots are happy with the change

[issue33608] [subinterpreters] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.

STINNER Victor added the comment: > That's okay, Victor. Thanks for jumping on this. I'll take a look when I > get a chance. >From what I saw, your first commit was enough to reproduce the crash. If I recall correctly, Antoine Pitrou modified the GIL so threads exit immediately when

[issue33608] [subinterpreters] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.

Eric Snow added the comment: That's okay, Victor. Thanks for jumping on this. I'll take a look when I get a chance. -- ___ Python tracker ___

[issue33608] [subinterpreters] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.

STINNER Victor added the comment: For curious people, Pablo Galindo spent a few hours to investigate https://bugs.python.org/issue36114 and I spent a few more hours to finish the analysis. The bug indirectly crashed my laptop :-) https://twitter.com/VictorStinner/status/1102528982036201478

[issue33608] [subinterpreters] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.

STINNER Victor added the comment: New changeset 4d61e6e3b802399be62a521d6fa785698cb670b5 by Victor Stinner in branch 'master': Revert: bpo-33608: Factor out a private, per-interpreter _Py_AddPendingCall(). (GH-11617) (GH-12159)

[issue33608] [subinterpreters] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.

STINNER Victor added the comment: Hi Eric, I'm sorry but I had to revert your recent work. It introduced regressions in at least in test_io and test_multiprocessing_spawn on Windows and FreeBSD. I simply don't have the bandwidth to investigate this regression yet, whereas we really need the

[issue33608] [subinterpreters] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.

STINNER Victor added the comment: > The commit ef4ac967e2f3a9a18330cc6abe14adb4bc3d0465 introduced a regression > in > test.test_multiprocessing_spawn.WithThreadsTestManagerRestart.test_rapid_restart: > bpo-36114. There is a very similar bug on Windows: bpo-36116. I'm now also quite

[issue33608] [subinterpreters] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.

Change by STINNER Victor : -- pull_requests: +12157 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue33608] [subinterpreters] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.

STINNER Victor added the comment: The commit ef4ac967e2f3a9a18330cc6abe14adb4bc3d0465 introduced a regression in test.test_multiprocessing_spawn.WithThreadsTestManagerRestart.test_rapid_restart: bpo-36114. Would it be possible to revert it until the bug is properly understood and fixed?

[issue33608] [subinterpreters] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.

STINNER Victor added the comment: > I suspect changes for this issue may be creating test_io failures on my > windows builders, (...) I created bpo-36177 to track this regression. -- ___ Python tracker

[issue33608] [subinterpreters] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.

David Bolen added the comment: If I can help with testing or builder access or anything just let me know. It appears that I can pretty reliably trigger the error through just the individual tests (test_daemon_threads_shutdown_std{out,err}_deadlock) in isolation, which is definitely less

[issue33608] [subinterpreters] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.

Eric Snow added the comment: I've merged the PR for hoisting eval_breaker before the ceval loop starts. @Raymond, see if that addresses the performance regression you've been seeing. I'll close this issue after I've sorted out the buildbot issues David mentioned (on his Windows builders).

[issue33608] [subinterpreters] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.

Eric Snow added the comment: New changeset bda918bf65a88560ec453aaba0758a9c0d49b449 by Eric Snow in branch 'master': bpo-33608: Simplify ceval's DISPATCH by hoisting eval_breaker ahead of time. (gh-12062) https://github.com/python/cpython/commit/bda918bf65a88560ec453aaba0758a9c0d49b449

[issue33608] [subinterpreters] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.

Eric Snow added the comment: New changeset b05b711a2cef6c6c381e01069dedac372e0b9fb2 by Eric Snow in branch 'master': bpo-33608: Use _Py_AddPendingCall() in _PyCrossInterpreterData_Release(). (gh-12024) https://github.com/python/cpython/commit/b05b711a2cef6c6c381e01069dedac372e0b9fb2

[issue33608] [subinterpreters] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.

David Bolen added the comment: I suspect changes for this issue may be creating test_io failures on my windows builders, most notably my x86 Windows 7 builder where test_io has been failing both attempts on almost all builds. It fails with a lock failure during interpreter shutdown, and

[issue33608] [subinterpreters] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.

Change by Eric Snow : -- pull_requests: +12086 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue33608] [subinterpreters] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.

Eric Snow added the comment: FYI, I have a couple of small follow-up changes to land before I close this issue. -- ___ Python tracker ___

[issue33608] [subinterpreters] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.

STINNER Victor added the comment: > ...so it doesn't appear that my PR introduces a performance regression. IMHO there is no performance regression at all. Just noice in the result which doesn't come with std dev. -- ___ Python tracker

[issue33608] [subinterpreters] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.

Eric Snow added the comment: ...so it doesn't appear that my PR introduces a performance regression. -- ___ Python tracker ___ ___

[issue33608] [subinterpreters] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.

Eric Snow added the comment: Here's what I did (more or less): $ python3 -m pip install --user perf $ python3 -m perf system tune $ git clone g...@github.com:python/performance.git $ git clone g...@github.com:python/cpython.git $ cd cpython $ git checkout

[issue33608] [subinterpreters] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.

Eric Snow added the comment: At least, it *might* be a performance regression. Here are the two commits I tried: after: ef4ac967e2f3a9a18330cc6abe14adb4bc3d0465 (PR #11617, from this issue) before: 463572c8beb59fd9d6850440af48a5c5f4c0c0c9 (the one right before) After building each (a

[issue33608] [subinterpreters] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.

Eric Snow added the comment: GH-11617 has introduces a slight performance regression which will need to be resolved. If I can't find the problem right away then I'll probably temporarily revert the commit. See https://mail.python.org/pipermail/python-dev/2019-February/156451.html.

[issue33608] [subinterpreters] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.

Change by Eric Snow : -- pull_requests: +12057 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue33608] [subinterpreters] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.

Eric Snow added the comment: New changeset ef4ac967e2f3a9a18330cc6abe14adb4bc3d0465 by Eric Snow in branch 'master': bpo-33608: Factor out a private, per-interpreter _Py_AddPendingCall(). (GH-11617) https://github.com/python/cpython/commit/ef4ac967e2f3a9a18330cc6abe14adb4bc3d0465

[issue33608] [subinterpreters] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.

Change by Eric Snow : -- pull_requests: +11360, 11361 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue33608] [subinterpreters] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.

Change by Eric Snow : -- pull_requests: +11360, 11361, 11362 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue33608] [subinterpreters] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.

Change by Eric Snow : -- pull_requests: +11360 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue33608] [subinterpreters] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.

Change by Johan Dahlin : -- nosy: +Johan Dahlin ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue33608] [subinterpreters] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.

Eric Snow added the comment: @Neil, we're definitely on the same page. In fact, in a world where subinterpreters do not share a GIL, we can't ever use an object in one interpreter that was created in another (due to thread safety on refcounts). The role of "tightly controlling"

[issue33608] [subinterpreters] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.

Neil Schemenauer added the comment: I would suggest that sharing of objects between interpreters should be stamped out. Could we have some #ifdef debug checking that would warn or assert so this doesn't happen? I know currently we share a lot of objects. However, in the long term, that

[issue33608] [subinterpreters] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.

Change by Eric Snow : -- keywords: +patch pull_requests: +8758 stage: needs patch -> patch review ___ Python tracker ___ ___

[issue33608] [subinterpreters] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.

Change by Eric Snow : -- nosy: +emilyemorehouse ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue33608] [subinterpreters] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.

Nick Coghlan added the comment: Adding a low level callback based mechanism to ask another interpreter to do work seems like a good way to go to me. As an example of why that might be needed, consider the case of sharing a buffer exporting object with another

[issue33608] [subinterpreters] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.

Change by STINNER Victor : -- title: Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed. -> [subinterpreters] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.