[issue32280] Expose `_PyRuntime` through a section name

2019-06-14 Thread Eric Snow
Eric Snow added the comment: (Sorry for the delay!) Is the need for a named section due to the fact that _PyRuntime is part of the "internal" C-API (hidden behind the Py_BUILD_CORE macro)? I.E. would it help to build with Py_BUILD_CORE? Regarding the GIL: The plan for 3.9 is t

[issue34651] Disallow fork in a subinterpreter.

2019-06-14 Thread Eric Snow
Eric Snow added the comment: FYI, I plan on looking into this either today or next Friday. -- ___ Python tracker <https://bugs.python.org/issue34651> ___ ___

[issue12857] Expose called function on frame object

2019-06-14 Thread Eric Snow
Change by Eric Snow : Removed file: https://bugs.python.org/file48418/122.pdf ___ Python tracker <https://bugs.python.org/issue12857> ___ ___ Python-bugs-list mailin

[issue37186] Everyone uses GIL wrong! = DEADLOCK

2019-06-07 Thread Eric Snow
Eric Snow added the comment: The situation with subinterpreters and the gilstate API is a known problem (see issues #10915 and #15751). We plan on fixing that in the near future for 3.9 (and probably for 3.8). Feel free to chime in on those earlier issues. Note that the subinterpreter

[issue15751] Support subinterpreters in the GIL state API

2019-06-07 Thread Eric Snow
Change by Eric Snow : -- versions: +Python 3.8, Python 3.9 -Python 3.6 ___ Python tracker <https://bugs.python.org/issue15751> ___ ___ Python-bugs-list mailin

[issue10915] Make the PyGILState API compatible with multiple interpreters

2019-06-07 Thread Eric Snow
Change by Eric Snow : -- versions: +Python 3.9 ___ Python tracker <https://bugs.python.org/issue10915> ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue33608] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.

2019-06-03 Thread Eric Snow
Eric Snow added the comment: Also, thanks for the suggestions, Victor! -- ___ Python tracker <https://bugs.python.org/issue33608> ___ ___ Python-bugs-list mailin

[issue33608] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.

2019-06-03 Thread Eric Snow
Eric Snow added the comment: Well, for this PR I actually disabled the execution of pending calls during runtime finialization. I'd hoped it would help, but it did not. :( Regardless, I'll need to look even more closely at what is different during runtime finalization with my PR

[issue33608] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.

2019-06-03 Thread Eric Snow
Eric Snow added the comment: I had really hoped to get this in for 3.8, as I may need it. However, realistically I doubt it will take just a few minutes to resolve this. With the beta 1 targeted for today it makes sense to revert my change. :( (I may still try to convince our fearless RM

[issue33608] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.

2019-06-03 Thread Eric Snow
Eric Snow added the comment: FWIW, the failures before were in test_io and test_multiprocessing_spawn, not test_asyncio. -- ___ Python tracker <https://bugs.python.org/issue33

[issue33608] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.

2019-06-03 Thread Eric Snow
Eric Snow added the comment: Please wait on reverting. I'm going to investigate. @koobs, yeah, I'll send you a key. Thanks! :) -- ___ Python tracker <https://bugs.python.org/issue33

[issue33608] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.

2019-06-01 Thread Eric Snow
Eric Snow added the comment: FYI, after merging that PR I realized that the COMPUTE_EVAL_BREAKER macro isn't quite right. While the following scenario worked before, now it doesn't: 1. interpreter A: _PyEval_AddPendingCall() causes the global eval breaker to be set 2. interpreter B

[issue33608] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.

2019-06-01 Thread Eric Snow
Eric Snow added the comment: So far so good. :) I'll keep an eye on things and if the buildbots are still happy then I'll add back _PyEval_FinishPendingCalls() in _Py_FinalizeEx() in a few days. -- ___ Python tracker <https://bugs.python.

[issue33608] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.

2019-06-01 Thread Eric Snow
Eric Snow added the comment: New changeset 6a150bcaeb190d1731b38ab9c7a5d1a352847ddc by Eric Snow in branch 'master': bpo-33608: Factor out a private, per-interpreter _Py_AddPendingCall(). (gh-13714) https://github.com/python/cpython/commit/6a150bcaeb190d1731b38ab9c7a5d1a352847ddc

[issue33608] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.

2019-06-01 Thread Eric Snow
Eric Snow added the comment: I've made a few tweaks and Victor did some cleanup, so I'm going to try the PR again. At first I'm also going to disable the _PyEval_FinishPendingCalls() call in _Py_FinalizeEx() and then enable it is a separate PR. Also, I've opened bpo-37127 specifically

[issue33608] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.

2019-06-01 Thread Eric Snow
Eric Snow added the comment: That's really helpful, Pavel! Thanks for investigating so thoroughly. I'm going to double check all the places I've made the assumption that "tstate" isn't NULL and likewise for "tstate->interp". Is there an issue open for the

[issue37127] Handling pending calls during runtime finalization may cause problems.

2019-06-01 Thread Eric Snow
Eric Snow added the comment: Also, someone did manage to investigate and identify a likely cause: https://bugs.python.org/issue33608#msg342791 -- ___ Python tracker <https://bugs.python.org/issue37

[issue37127] Handling pending calls during runtime finalization may cause problems.

2019-06-01 Thread Eric Snow
New submission from Eric Snow : In Python/lifecycle.c (Py_FinalizeEx) we call _Py_FinishPendingCalls(), right after we stop all non-daemon Python threads but before we've actually started finalizing the runtime state. That call looks for any remaining pending calls (for the main interpreter

[issue36818] Add PyInterpreterState.runtime.

2019-05-31 Thread Eric Snow
Change by Eric Snow : -- resolution: rejected -> fixed ___ Python tracker <https://bugs.python.org/issue36818> ___ ___ Python-bugs-list mailing list Unsubscrib

[issue33608] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.

2019-05-31 Thread Eric Snow
Change by Eric Snow : -- pull_requests: +13601 pull_request: https://github.com/python/cpython/pull/13714 ___ Python tracker <https://bugs.python.org/issue33

[issue36818] Add PyInterpreterState.runtime.

2019-05-31 Thread Eric Snow
Eric Snow added the comment: New changeset 396e0a8d9dc65453cb9d53500d0a620602656cfe by Eric Snow in branch 'master': bpo-36818: Add PyInterpreterState.runtime field. (gh-13129) https://github.com/python/cpython/commit/396e0a8d9dc65453cb9d53500d0a620602656cfe

[issue36839] Support the buffer protocol in code objects

2019-05-30 Thread Eric Snow
Eric Snow added the comment: FWIW, I don't see the problem with supporting any read-only "buffer" object, rather than just bytes objects, for the string of bytes in a code object. That's all that Dino is proposing. The change is not invasive and solves a real need. The

[issue37088] Add a way to schedule a function to be called from the main thread

2019-05-29 Thread Eric Snow
Eric Snow added the comment: If this is about signals, it isn't enough just to run on the main thread. It also has to be the main interpreter. -- ___ Python tracker <https://bugs.python.org/issue37

[issue37088] Add a way to schedule a function to be called from the main thread

2019-05-29 Thread Eric Snow
Eric Snow added the comment: > Well, it's absolutely the same mechanism that signal handlers use. Antoine changed signals a while back so they no longer use the pending calls machinery. -- ___ Python tracker <https://bugs.python.org/issu

[issue37088] Add a way to schedule a function to be called from the main thread

2019-05-29 Thread Eric Snow
Eric Snow added the comment: Note that I'm working on making pending calls per-interpreter (see issue #33608 and https://github.com/python/cpython/pull/12360 (since reverted)). As to exposing Py_AddPendingCall() as sys.addpendingcall, that might be opening a can of worms. Injecting code

[issue36876] Global C variables are a problem.

2019-05-23 Thread Eric Snow
Change by Eric Snow : -- pull_requests: +13445 ___ Python tracker <https://bugs.python.org/issue36876> ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue24932] Use proper command line parsing in _testembed

2019-05-18 Thread Eric Snow
Change by Eric Snow : -- nosy: +vstinner ___ Python tracker <https://bugs.python.org/issue24932> ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue36876] Global C variables are a problem.

2019-05-16 Thread Eric Snow
Change by Eric Snow : -- keywords: +patch pull_requests: +13283 stage: needs patch -> patch review ___ Python tracker <https://bugs.python.org/issu

[issue36876] Global C variables are a problem.

2019-05-10 Thread Eric Snow
Eric Snow added the comment: Also, Tools/c-globals/ignored-globals.txt is a bit out of date (some vars have been removed, renamed, or moved to another file). That should get cleaned up. It might also make sense to update check-c-globals.py to verify that all variables in ignored

[issue36877] [meta] Move fields from _PyRuntimeState to PyInterpreterState.

2019-05-10 Thread Eric Snow
Eric Snow added the comment: In conjunction with #36876, there are a bunch of currently ignored globals (in Tools/c-globals/ignored-globals.txt) that should actually be moved to per-interpreter runtime state. This mostly applies to any globals that point to one or more objects, since

[issue36877] [meta] Move fields from _PyRuntimeState to PyInterpreterState.

2019-05-10 Thread Eric Snow
Eric Snow added the comment: FYI, I've already started some of this work: * #36737 warnings * #36854 gc * #33608 pending calls * #10915 & #15751 gilstate Other bits I'm planning on: * the rest of the global "ceval" state * the memory allocators * the GIL Note that, to ma

[issue36877] [meta] Move fields from _PyRuntimeState to PyInterpreterState.

2019-05-10 Thread Eric Snow
New submission from Eric Snow : We have quite a bit of global state the runtime that effectively breaks the isolation between interpreters. Some of it exists as "global" C variables (see #36876) and the rest as fields on _PyRuntimeState. The offending state should

[issue36876] Global C variables are a problem.

2019-05-10 Thread Eric Snow
New submission from Eric Snow : We still have a bunch of "global" C variables (static globals, static locals, maybe thread-local storage) in our code-base that break the isolation between interpreters. I added Tools/c-globals/check-c-globals.py a while back to help identify such

[issue36737] Warnings operate out of global runtime state.

2019-05-10 Thread Eric Snow
Eric Snow added the comment: New changeset 86ea58149c3e83f402cecd17e6a536865fb06ce1 by Eric Snow in branch 'master': bpo-36737: Use the module state C-API for warnings. (gh-13159) https://github.com/python/cpython/commit/86ea58149c3e83f402cecd17e6a536865fb06ce1

[issue36737] Warnings operate out of global runtime state.

2019-05-10 Thread Eric Snow
Change by Eric Snow : -- assignee: -> eric.snow resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker <https://bugs.python

[issue36854] GC operates out of global runtime state.

2019-05-09 Thread Eric Snow
Change by Eric Snow : -- keywords: +patch pull_requests: +13130 stage: -> patch review ___ Python tracker <https://bugs.python.org/issue36854> ___ ___ Python-

[issue24554] GC should happen when a subinterpreter is destroyed

2019-05-09 Thread Eric Snow
Eric Snow added the comment: FYI, issue #36854 is about moving GC runtime state from _PyRuntimeState to PyInterpreterState. However, that doesn't trigger any collection when the interpreter is finalized. So there is more to be done here

[issue36854] GC operates out of global runtime state.

2019-05-08 Thread Eric Snow
New submission from Eric Snow : (also see #24554) We need to move GC state from _PyRuntimeState to PyInterpreterState. -- assignee: eric.snow components: Interpreter Core messages: 341911 nosy: eric.snow, pablogsal priority: normal severity: normal status: open title: GC operates out

[issue36737] Warnings operate out of global runtime state.

2019-05-07 Thread Eric Snow
Change by Eric Snow : -- keywords: +patch pull_requests: +13078 stage: needs patch -> patch review ___ Python tracker <https://bugs.python.org/issu

[issue36835] Move the warnings runtime state into per-interpreter state.

2019-05-07 Thread Eric Snow
Eric Snow added the comment: dupe of #36737 -- resolution: -> duplicate stage: patch review -> resolved status: open -> closed ___ Python tracker <https://bugs.python.or

[issue36835] Move the warnings runtime state into per-interpreter state.

2019-05-07 Thread Eric Snow
Change by Eric Snow : -- keywords: +patch pull_requests: +13076 stage: needs patch -> patch review ___ Python tracker <https://bugs.python.org/issu

[issue36835] Move the warnings runtime state into per-interpreter state.

2019-05-07 Thread Eric Snow
New submission from Eric Snow : Currently the warnings module uses runtime-global state (PyRuntimeState.warnings). That should be moved down to per-interpreter state. There are three possible places: 1. the module's "module state" 2. the module's __dict__ 3. PyInterpreterStat

[issue36818] Add PyInterpreterState.runtime.

2019-05-06 Thread Eric Snow
New submission from Eric Snow : Currently we use the _PyRuntime static global to access the runtime state in various places. At the same time, in thread contexts we get access to the thread state from Thread-Local Storage and the interpreter state by indirection from there. We should do

[issue36818] Add PyInterpreterState.runtime.

2019-05-06 Thread Eric Snow
Change by Eric Snow : -- keywords: +patch pull_requests: +13042 ___ Python tracker <https://bugs.python.org/issue36818> ___ ___ Python-bugs-list mailin

[issue36809] Crash for test test_importlib

2019-05-06 Thread Eric Snow
Change by Eric Snow : -- nosy: +brett.cannon, eric.snow ___ Python tracker <https://bugs.python.org/issue36809> ___ ___ Python-bugs-list mailing list Unsub

[issue36772] Let lru_cache be used as a decorator with no arguments

2019-05-04 Thread Eric Snow
Eric Snow added the comment: As to the issue of positional vs. keyword arguments, keyword arguments make the implementation easier in some cases. Otherwise I haven't seen positional arguments cause much of a problem. -- ___ Python tracker

[issue36772] Let lru_cache be used as a decorator with no arguments

2019-05-04 Thread Eric Snow
Eric Snow added the comment: FWIW, I've followed this pattern (one function is both decorator and factory) in my own code for quite a while. I've never found it confusing nor has anyone else (that I'm aware) that has used those decorators. One reason I've done decorators this way

[issue36737] Warnings operate out of global runtime state.

2019-04-27 Thread Eric Snow
Eric Snow added the comment: Good point. Also, the whole idea of inheriting things (settings, some copied objects, etc.) into subinterpreters is interesting. My initial reaction is that folks would appreciate that feature, at least for a handful of things. It's not critical, but is worth

[issue36710] Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable

2019-04-27 Thread Eric Snow
Eric Snow added the comment: FWIW, I don't mean to side-track this issue. If we want to have any further discussion about broader solutions then let's take this to capi-sig. In fact, I've started a thread there. I'd post the link, but I think it got stuck in moderation

[issue36710] Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable

2019-04-27 Thread Eric Snow
Eric Snow added the comment: I don't think this change is the right way to go (yet), but something related might be. First, let's be clear on the status quo for CPython. (This has gotten long, but I want to be clear.) Status Quo For simplicity sake, let's say nearly all

[issue36710] Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable

2019-04-27 Thread Eric Snow
Eric Snow added the comment: FWIW, PEP 554 is part of a larger project that I've been working on (slowly) for several years now. [1] The concrete objective is to leverage subinterpreters as the mechanism by which we can achieve multi-core parallelism in Python code. Moving the GIL

[issue36737] Warnings operate out of global runtime state.

2019-04-27 Thread Eric Snow
New submission from Eric Snow : (See Include/internal/pycore_warnings.h and Python/_warnings.c.) The warnings module's state (filters, default action, etc.) is currently stored at the level of the global runtime. That's a problem for the following reasons: * Python objects are getting

[issue36710] Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable

2019-04-24 Thread Eric Snow
Change by Eric Snow : -- nosy: +eric.snow ___ Python tracker <https://bugs.python.org/issue36710> ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue36666] threading.Thread should have way to catch an exception thrown within

2019-04-19 Thread Eric Snow
Eric Snow added the comment: Here's a basic decorator along those lines, similar to one that I've used on occasion: def as_thread(target): def _target(): try: t.result = target() except Exception as exc: t.failure = exc

[issue31954] Don't prevent dict optimization by coupling with OrderedDict

2019-04-15 Thread Eric Snow
Eric Snow added the comment: Please don't miss the fact that the main reason for mirroring the dict table is to get O(1) node lookup (in the linked list). Otherwise most lookup-dependent operations, like __delitem__(), would become O(n); whereas in the pure-Python implementation they are O

[issue33608] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.

2019-04-12 Thread Eric Snow
Eric Snow added the comment: @Victor, I set up a FreeBSD 12.0 VM (in Hyper-v) and made sure core files were getting generated for segfaults. Then I cloned the cpython repo, built it (using GCC), and ran regrtest as you recommended. It generated no core files after half an hour. I

[issue33608] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.

2019-04-12 Thread Eric Snow
Eric Snow added the comment: New changeset b75b1a3504a0cea6fac6ecba44c10b2629577025 by Eric Snow in branch 'master': bpo-33608: Revert "Factor out a private, per-interpreter _Py_AddPendingCall()." (gh-12806) https://github.com/python/cpython/commit/b75b1a3504a0cea6fac6ecba44c10b

[issue33608] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.

2019-04-12 Thread Eric Snow
Change by Eric Snow : -- pull_requests: +12733 ___ Python tracker <https://bugs.python.org/issue33608> ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue33608] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.

2019-04-12 Thread Eric Snow
Eric Snow added the comment: Thanks for checking, Victor. Don't feel bad about your results, nor about not checking sooner. :) We'll get this sorted out. For now I'll revert. This is not code that changes very often, so there isn't much benefit to keeping it merged. Testing against

[issue33608] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.

2019-04-12 Thread Eric Snow
Eric Snow added the comment: New changeset f13c5c8b9401a9dc19e95d8b420ee100ac022208 by Eric Snow in branch 'master': bpo-33608: Factor out a private, per-interpreter _Py_AddPendingCall(). (gh-12360) https://github.com/python/cpython/commit/f13c5c8b9401a9dc19e95d8b420ee100ac022208

[issue19476] Add a dedicated specification for module "reloading" to the language reference

2019-04-10 Thread Eric Snow
Eric Snow added the comment: Thanks for checking in, Cheryl! Clearly no one picked up this banner. :) Furthermore, I'm not aware of any reload-related complaints coming from the community. I know of only a couple major use cases for reload(): refresh parts of a running app during

[issue32280] Expose `_PyRuntime` through a section name

2019-04-05 Thread Eric Snow
Change by Eric Snow : -- versions: +Python 3.8 -Python 3.7 ___ Python tracker <https://bugs.python.org/issue32280> ___ ___ Python-bugs-list mailing list Unsub

[issue36487] Make C-API docs clear about what the "main" interpreter is

2019-04-05 Thread Eric Snow
Eric Snow added the comment: I should have added something like this earlier, but here are key points to consider covering: * "main" interpreter is the original, created when the runtime initializes * historically almost always the only Python interpreter in a process * this i

[issue36157] Document PyInterpreterState_Main().

2019-04-01 Thread Eric Snow
Eric Snow added the comment: Thanks for working on this, Joannah! :) -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker <https://bugs.python.or

[issue36487] Make C-API docs clear about what the "main" interpreter is

2019-04-01 Thread Eric Snow
Change by Eric Snow : -- nosy: +eric.snow ___ Python tracker <https://bugs.python.org/issue36487> ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue33356] Windows 10 buildbot: test__xxsubinterpreters.test_already_running() fails randomly

2019-03-29 Thread Eric Snow
Eric Snow added the comment: I'll take a look when I get a chance. -- ___ Python tracker <https://bugs.python.org/issue33356> ___ ___ Python-bugs-list mailin

[issue36427] Document that PyEval_RestoreThread and PyGILState_Ensure can terminate the calling thread

2019-03-29 Thread Eric Snow
Eric Snow added the comment: > Currently PyEval_RestoreThread and its callers (mainly PyGILState_Ensure) > can terminate the thread if the interpreter is finalizing: s/interpreter/runtime/ Most likely (guaranteed?) it will be in the main interpreter, but it is actually not tri

[issue36479] Exit threads when interpreter is finalizing rather than runtime.

2019-03-29 Thread Eric Snow
New submission from Eric Snow : Currently when a thread acquires the GIL, it subsequently exits if the runtime is finalizing. This helps with some cases like with stopping daemon threads. This behavior should instead be triggered by the thread's interpreter finalizing rather than

[issue36427] Document that PyEval_RestoreThread and PyGILState_Ensure can terminate the calling thread

2019-03-29 Thread Eric Snow
Eric Snow added the comment: > > if (_Py_IsFinalizing() && !_Py_CURRENTLY_FINALIZING(tstate)) > > _Py_IsFinalizing() check is redundant :-) Not really: * _Py_IsFinalizing() checks if the runtime is finalizing * _Py_CURRENTLY_FINALIZING checks if the current threa

[issue36427] Document that PyEval_RestoreThread and PyGILState_Ensure can terminate the calling thread

2019-03-29 Thread Eric Snow
Eric Snow added the comment: Related: * #36475: "PyEval_AcquireLock() and PyEval_AcquireThread() do not handle runtime finalization properly." * #36476: "Runtime finalization assumes all other threads have exited." -- ___ Py

[issue36157] Document PyInterpreterState_Main().

2019-03-29 Thread Eric Snow
Eric Snow added the comment: As I noted on the PR, this might be a good chance to make sure the C-API docs are clear about what the "main" interpreter is. -- ___ Python tracker <https://bugs.python.o

[issue36097] Use only public C-API in _xxsubinterpreters module.

2019-03-29 Thread Eric Snow
Change by Eric Snow : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker <https://bugs.python.or

[issue36225] Lingering subinterpreters should be implicitly cleared on shutdown

2019-03-29 Thread Eric Snow
Eric Snow added the comment: test__xxsubinterpreters is a great place for tests that exercise use of subinterpreters, including most lifecycle operations. There are also one or two subinterpreter-related tests in test_embed. However, for this issue the interplay with runtime finalization

[issue36225] Lingering subinterpreters should be implicitly cleared on shutdown

2019-03-29 Thread Eric Snow
Eric Snow added the comment: Interestingly, I noticed this independently today. :) Here's what I wrote in #36477 (which I've closed as a duplicate): When using subinterpreters, any that exist when Py_FinalizeEx() is called do not appear to get cleaned up during runtime finalization. Maybe

[issue36477] Subinterpreters are not finalized during runtime finalization.

2019-03-29 Thread Eric Snow
Change by Eric Snow : -- resolution: -> duplicate stage: needs patch -> resolved status: open -> closed superseder: -> Lingering subinterpreters should be implicitly cleared on shutdown ___ Python tracker <https://bugs.python

[issue36469] Stuck during interpreter exit, attempting to take the GIL

2019-03-29 Thread Eric Snow
Eric Snow added the comment: @Remy, aside from the recommendations I've made, I'm not sure what else we can do to help. Before we close the issue, I'd really like to ensure that one of those threads is holding the GIL still. It would definitely be a problem if a thread exited while still

[issue36476] Runtime finalization assumes all other threads have exited.

2019-03-29 Thread Eric Snow
Eric Snow added the comment: FYI, I've opened issue36477 to deal with the subinterpreters case. -- ___ Python tracker <https://bugs.python.org/issue36

[issue36477] Subinterpreters are not finalized during runtime finalization.

2019-03-29 Thread Eric Snow
New submission from Eric Snow : When using subinterpreters, any that exist when Py_FinalizeEx() is called do not appear to get cleaned up during runtime finalization. Maybe I've been looking at the code too much and I'm missing something. :) This really isn't a problem except for embedders

[issue36469] Stuck during interpreter exit, attempting to take the GIL

2019-03-29 Thread Eric Snow
Eric Snow added the comment: I've also opened issues #36476 and #36477. -- ___ Python tracker <https://bugs.python.org/issue36469> ___ ___ Python-bugs-list m

[issue36476] Runtime finalization assumes all other threads have exited.

2019-03-29 Thread Eric Snow
New submission from Eric Snow : Among the first 3 things that happen in Py_FinalizeEx() are, in order: 1. wait for all non-daemon threads (of the main interpreter) to finish 2. call registered atexit funcs 3. mark the runtime as finalizing At that point the only remaining Python threads

[issue36475] PyEval_AcquireLock() and PyEval_AcquireThread() do not handle runtime finalization properly.

2019-03-29 Thread Eric Snow
Change by Eric Snow : -- components: +Interpreter Core ___ Python tracker <https://bugs.python.org/issue36475> ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue36469] Stuck during interpreter exit, attempting to take the GIL

2019-03-29 Thread Eric Snow
Eric Snow added the comment: Looking at the stack traces for all your threads (super helpful, BTW), I saw 4 groups: * (1) main thread (blocked on GIL during runtime finalization) + Thread 1 * (12) blocked on GIL in call to PyEval_RestoreThread() (socket, select, time.sleep(), file.read

[issue36469] Stuck during interpreter exit, attempting to take the GIL

2019-03-29 Thread Eric Snow
Eric Snow added the comment: I've opened issue36475 for the two C-API functions that do not cause daemon threads to exit. -- ___ Python tracker <https://bugs.python.org/issue36

[issue36475] PyEval_AcquireLock() and PyEval_AcquireThread() do not handle runtime finalization properly.

2019-03-29 Thread Eric Snow
New submission from Eric Snow : Daemon threads keep running until they finish or until finalization starts. For the latter, there is a check right after the thread acquires the GIL which causes the thread to exit if runtime finalization has started. [1] However, there are functions

[issue36469] Stuck during interpreter exit, attempting to take the GIL

2019-03-29 Thread Eric Snow
Eric Snow added the comment: Here are some things that would likely help: * in the main thread stop your daemon threads if you hit SystemExit * make sure daemon threads release/acquire the GIL frequently (so they notice finalization) * make sure daemon threads otherwise exit promptly

[issue36469] Stuck during interpreter exit, attempting to take the GIL

2019-03-29 Thread Eric Snow
Eric Snow added the comment: At this point I think it's likely that the problem relates to how daemon threads are handled during runtime finalization. What normally happens in the main thread of the "python3" executable is this: 1. Python runtime initializes 2. main interpreter i

[issue36469] Stuck during interpreter exit, attempting to take the GIL

2019-03-29 Thread Eric Snow
Eric Snow added the comment: * Are any signals (or signal handlers) involved? * Are you monkeypatching any builtins, builtin/stdlib modules (or any parts of those)? Keep in mind that a number of bugs have been fixed in later releases related to the various things I've asked about

[issue30703] Non-reentrant signal handler (test_multiprocessing_forkserver hangs)

2019-03-29 Thread Eric Snow
Change by Eric Snow : -- nosy: +eric.snow ___ Python tracker <https://bugs.python.org/issue30703> ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue36469] Stuck during interpreter exit, attempting to take the GIL

2019-03-29 Thread Eric Snow
Eric Snow added the comment: Also: * are any of your threads daemon threads? -- ___ Python tracker <https://bugs.python.org/issue36469> ___ ___ Python-bug

[issue36469] Stuck during interpreter exit, attempting to take the GIL

2019-03-29 Thread Eric Snow
Eric Snow added the comment: As Inada-san indicated, the problem might be resolved already. So without the option of reproducing the problem, it will be hard to resolve this. Here's some information you could provide that might help narrow down the scope a bit: * what (stdlib/third-party

[issue36450] 3D Array Assignment to all 2D sub-arrays

2019-03-27 Thread Eric Snow
Eric Snow added the comment: In Python, multiplication on a list does not make copies of the values in the original list. So what you have done is equivalent to the following: a = [0, 0] b = [a, a] M = [b, b] Hence: >>> M[0] is M[1] True >>> M[0][0] is M[0][1

[issue32312] Create Py_AtExitRegister C API

2019-03-27 Thread Eric Snow
Eric Snow added the comment: > Neil Schemenauer added the comment: > Regarding m_traverse, maybe the list of finalizers should be stored somewhere > in the interpreter > state, not in the atexit module state. That would make more sense to me. > atexit could merely be &g

[issue36124] Provide convenient C API for storing per-interpreter state

2019-03-15 Thread Eric Snow
Eric Snow added the comment: @arigo, thanks for nudging us here. :) Let me know if there's anything else that would help here. -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracke

[issue36124] Provide convenient C API for storing per-interpreter state

2019-03-15 Thread Eric Snow
Eric Snow added the comment: New changeset d2fdd1fedf6b9dc785cf5025b548a989faed089a by Eric Snow in branch 'master': bpo-36124: Add PyInterpreterState.dict. (gh-12132) https://github.com/python/cpython/commit/d2fdd1fedf6b9dc785cf5025b548a989faed089a

[issue36097] Use only public C-API in _xxsubinterpreters module.

2019-03-15 Thread Eric Snow
Eric Snow added the comment: New changeset c11183cdcff6af13c4339fdcce84ab63f7930ddc by Eric Snow in branch 'master': bpo-36097: Use only public C-API in the_xxsubinterpreters module (adding as necessary). (gh-12359) https://github.com/python/cpython/commit

[issue33608] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.

2019-03-15 Thread Eric Snow
Change by Eric Snow : -- pull_requests: +12327 ___ Python tracker <https://bugs.python.org/issue33608> ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue36097] Use only public C-API in _xxsubinterpreters module.

2019-03-15 Thread Eric Snow
Change by Eric Snow : -- pull_requests: +12326 stage: needs patch -> patch review ___ Python tracker <https://bugs.python.org/issue36097> ___ ___ Python-

[issue33608] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.

2019-03-15 Thread Eric Snow
Eric Snow added the comment: New changeset 842a2f07f2f08a935ef470bfdaeef40f87490cfc by Eric Snow in branch 'master': bpo-33608: Deal with pending calls relative to runtime shutdown. (gh-12246) https://github.com/python/cpython/commit/842a2f07f2f08a935ef470bfdaeef40f87490cfc

[issue33608] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.

2019-03-08 Thread Eric Snow
Eric Snow added the comment: New changeset 8479a3426eb7d1840473f7788e639954363ed37e by Eric Snow in branch 'master': bpo-33608: Make sure locks in the runtime are properly re-created. (gh-12245) https://github.com/python/cpython/commit/8479a3426eb7d1840473f7788e639954363ed37e

[issue33608] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.

2019-03-08 Thread Eric Snow
Eric Snow added the comment: New changeset 5be45a6105d656c551adeee7770afdc3b806fbb5 by Eric Snow in branch 'master': bpo-33608: Minor cleanup related to pending calls. (gh-12247) https://github.com/python/cpython/commit/5be45a6105d656c551adeee7770afdc3b806fbb5

<    2   3   4   5   6   7   8   9   10   11   >