Eric Snow added the comment:
(Sorry for the delay!)
Is the need for a named section due to the fact that _PyRuntime is part of the
"internal" C-API (hidden behind the Py_BUILD_CORE macro)? I.E. would it help
to build with Py_BUILD_CORE?
Regarding the GIL:
The plan for 3.9 is t
Eric Snow added the comment:
FYI, I plan on looking into this either today or next Friday.
--
___
Python tracker
<https://bugs.python.org/issue34651>
___
___
Change by Eric Snow :
Removed file: https://bugs.python.org/file48418/122.pdf
___
Python tracker
<https://bugs.python.org/issue12857>
___
___
Python-bugs-list mailin
Eric Snow added the comment:
The situation with subinterpreters and the gilstate API is a known problem (see
issues #10915 and #15751). We plan on fixing that in the near future for 3.9
(and probably for 3.8). Feel free to chime in on those earlier issues. Note
that the subinterpreter
Change by Eric Snow :
--
versions: +Python 3.8, Python 3.9 -Python 3.6
___
Python tracker
<https://bugs.python.org/issue15751>
___
___
Python-bugs-list mailin
Change by Eric Snow :
--
versions: +Python 3.9
___
Python tracker
<https://bugs.python.org/issue10915>
___
___
Python-bugs-list mailing list
Unsubscribe:
Eric Snow added the comment:
Also, thanks for the suggestions, Victor!
--
___
Python tracker
<https://bugs.python.org/issue33608>
___
___
Python-bugs-list mailin
Eric Snow added the comment:
Well, for this PR I actually disabled the execution of pending calls during
runtime finialization. I'd hoped it would help, but it did not. :(
Regardless, I'll need to look even more closely at what is different during
runtime finalization with my PR
Eric Snow added the comment:
I had really hoped to get this in for 3.8, as I may need it. However,
realistically I doubt it will take just a few minutes to resolve this. With
the beta 1 targeted for today it makes sense to revert my change. :(
(I may still try to convince our fearless RM
Eric Snow added the comment:
FWIW, the failures before were in test_io and test_multiprocessing_spawn, not
test_asyncio.
--
___
Python tracker
<https://bugs.python.org/issue33
Eric Snow added the comment:
Please wait on reverting. I'm going to investigate.
@koobs, yeah, I'll send you a key. Thanks! :)
--
___
Python tracker
<https://bugs.python.org/issue33
Eric Snow added the comment:
FYI, after merging that PR I realized that the COMPUTE_EVAL_BREAKER macro isn't
quite right. While the following scenario worked before, now it doesn't:
1. interpreter A: _PyEval_AddPendingCall() causes the global
eval breaker to be set
2. interpreter B
Eric Snow added the comment:
So far so good. :) I'll keep an eye on things and if the buildbots are still
happy then I'll add back _PyEval_FinishPendingCalls() in _Py_FinalizeEx() in a
few days.
--
___
Python tracker
<https://bugs.python.
Eric Snow added the comment:
New changeset 6a150bcaeb190d1731b38ab9c7a5d1a352847ddc by Eric Snow in branch
'master':
bpo-33608: Factor out a private, per-interpreter _Py_AddPendingCall().
(gh-13714)
https://github.com/python/cpython/commit/6a150bcaeb190d1731b38ab9c7a5d1a352847ddc
Eric Snow added the comment:
I've made a few tweaks and Victor did some cleanup, so I'm going to try the PR
again. At first I'm also going to disable the _PyEval_FinishPendingCalls()
call in _Py_FinalizeEx() and then enable it is a separate PR.
Also, I've opened bpo-37127 specifically
Eric Snow added the comment:
That's really helpful, Pavel! Thanks for investigating so thoroughly. I'm
going to double check all the places I've made the assumption that "tstate"
isn't NULL and likewise for "tstate->interp".
Is there an issue open for the
Eric Snow added the comment:
Also, someone did manage to investigate and identify a likely cause:
https://bugs.python.org/issue33608#msg342791
--
___
Python tracker
<https://bugs.python.org/issue37
New submission from Eric Snow :
In Python/lifecycle.c (Py_FinalizeEx) we call _Py_FinishPendingCalls(), right
after we stop all non-daemon Python threads but before we've actually started
finalizing the runtime state. That call looks for any remaining pending calls
(for the main interpreter
Change by Eric Snow :
--
resolution: rejected -> fixed
___
Python tracker
<https://bugs.python.org/issue36818>
___
___
Python-bugs-list mailing list
Unsubscrib
Change by Eric Snow :
--
pull_requests: +13601
pull_request: https://github.com/python/cpython/pull/13714
___
Python tracker
<https://bugs.python.org/issue33
Eric Snow added the comment:
New changeset 396e0a8d9dc65453cb9d53500d0a620602656cfe by Eric Snow in branch
'master':
bpo-36818: Add PyInterpreterState.runtime field. (gh-13129)
https://github.com/python/cpython/commit/396e0a8d9dc65453cb9d53500d0a620602656cfe
Eric Snow added the comment:
FWIW, I don't see the problem with supporting any read-only "buffer" object,
rather than just bytes objects, for the string of bytes in a code object.
That's all that Dino is proposing. The change is not invasive and solves a
real need. The
Eric Snow added the comment:
If this is about signals, it isn't enough just to run on the main thread. It
also has to be the main interpreter.
--
___
Python tracker
<https://bugs.python.org/issue37
Eric Snow added the comment:
> Well, it's absolutely the same mechanism that signal handlers use.
Antoine changed signals a while back so they no longer use the pending calls
machinery.
--
___
Python tracker
<https://bugs.python.org/issu
Eric Snow added the comment:
Note that I'm working on making pending calls per-interpreter (see issue #33608
and https://github.com/python/cpython/pull/12360 (since reverted)).
As to exposing Py_AddPendingCall() as sys.addpendingcall, that might be opening
a can of worms. Injecting code
Change by Eric Snow :
--
pull_requests: +13445
___
Python tracker
<https://bugs.python.org/issue36876>
___
___
Python-bugs-list mailing list
Unsubscribe:
Change by Eric Snow :
--
nosy: +vstinner
___
Python tracker
<https://bugs.python.org/issue24932>
___
___
Python-bugs-list mailing list
Unsubscribe:
Change by Eric Snow :
--
keywords: +patch
pull_requests: +13283
stage: needs patch -> patch review
___
Python tracker
<https://bugs.python.org/issu
Eric Snow added the comment:
Also, Tools/c-globals/ignored-globals.txt is a bit out of date (some vars have
been removed, renamed, or moved to another file). That should get cleaned up.
It might also make sense to update check-c-globals.py to verify that all
variables in ignored
Eric Snow added the comment:
In conjunction with #36876, there are a bunch of currently ignored globals (in
Tools/c-globals/ignored-globals.txt) that should actually be moved to
per-interpreter runtime state. This mostly applies to any globals that point
to one or more objects, since
Eric Snow added the comment:
FYI, I've already started some of this work:
* #36737 warnings
* #36854 gc
* #33608 pending calls
* #10915 & #15751 gilstate
Other bits I'm planning on:
* the rest of the global "ceval" state
* the memory allocators
* the GIL
Note that, to ma
New submission from Eric Snow :
We have quite a bit of global state the runtime that effectively breaks the
isolation between interpreters. Some of it exists as "global" C variables (see
#36876) and the rest as fields on _PyRuntimeState. The offending state should
New submission from Eric Snow :
We still have a bunch of "global" C variables (static globals, static locals,
maybe thread-local storage) in our code-base that break the isolation between
interpreters. I added Tools/c-globals/check-c-globals.py a while back to help
identify such
Eric Snow added the comment:
New changeset 86ea58149c3e83f402cecd17e6a536865fb06ce1 by Eric Snow in branch
'master':
bpo-36737: Use the module state C-API for warnings. (gh-13159)
https://github.com/python/cpython/commit/86ea58149c3e83f402cecd17e6a536865fb06ce1
Change by Eric Snow :
--
assignee: -> eric.snow
resolution: -> fixed
stage: patch review -> resolved
status: open -> closed
___
Python tracker
<https://bugs.python
Change by Eric Snow :
--
keywords: +patch
pull_requests: +13130
stage: -> patch review
___
Python tracker
<https://bugs.python.org/issue36854>
___
___
Python-
Eric Snow added the comment:
FYI, issue #36854 is about moving GC runtime state from _PyRuntimeState to
PyInterpreterState. However, that doesn't trigger any collection when the
interpreter is finalized. So there is more to be done here
New submission from Eric Snow :
(also see #24554)
We need to move GC state from _PyRuntimeState to PyInterpreterState.
--
assignee: eric.snow
components: Interpreter Core
messages: 341911
nosy: eric.snow, pablogsal
priority: normal
severity: normal
status: open
title: GC operates out
Change by Eric Snow :
--
keywords: +patch
pull_requests: +13078
stage: needs patch -> patch review
___
Python tracker
<https://bugs.python.org/issu
Eric Snow added the comment:
dupe of #36737
--
resolution: -> duplicate
stage: patch review -> resolved
status: open -> closed
___
Python tracker
<https://bugs.python.or
Change by Eric Snow :
--
keywords: +patch
pull_requests: +13076
stage: needs patch -> patch review
___
Python tracker
<https://bugs.python.org/issu
New submission from Eric Snow :
Currently the warnings module uses runtime-global state
(PyRuntimeState.warnings). That should be moved down to per-interpreter state.
There are three possible places:
1. the module's "module state"
2. the module's __dict__
3. PyInterpreterStat
New submission from Eric Snow :
Currently we use the _PyRuntime static global to access the runtime state in
various places. At the same time, in thread contexts we get access to the
thread state from Thread-Local Storage and the interpreter state by indirection
from there. We should do
Change by Eric Snow :
--
keywords: +patch
pull_requests: +13042
___
Python tracker
<https://bugs.python.org/issue36818>
___
___
Python-bugs-list mailin
Change by Eric Snow :
--
nosy: +brett.cannon, eric.snow
___
Python tracker
<https://bugs.python.org/issue36809>
___
___
Python-bugs-list mailing list
Unsub
Eric Snow added the comment:
As to the issue of positional vs. keyword arguments, keyword arguments make the
implementation easier in some cases. Otherwise I haven't seen positional
arguments cause much of a problem.
--
___
Python tracker
Eric Snow added the comment:
FWIW, I've followed this pattern (one function is both decorator and factory)
in my own code for quite a while. I've never found it confusing nor has anyone
else (that I'm aware) that has used those decorators.
One reason I've done decorators this way
Eric Snow added the comment:
Good point.
Also, the whole idea of inheriting things (settings, some copied objects, etc.)
into subinterpreters is interesting. My initial reaction is that folks would
appreciate that feature, at least for a handful of things. It's not critical,
but is worth
Eric Snow added the comment:
FWIW, I don't mean to side-track this issue. If we want to have any further
discussion about broader solutions then let's take this to capi-sig. In fact,
I've started a thread there. I'd post the link, but I think it got stuck in
moderation
Eric Snow added the comment:
I don't think this change is the right way to go (yet), but something related
might be. First, let's be clear on the status quo for CPython. (This has
gotten long, but I want to be clear.)
Status Quo
For simplicity sake, let's say nearly all
Eric Snow added the comment:
FWIW, PEP 554 is part of a larger project that I've been working on (slowly)
for several years now. [1] The concrete objective is to leverage
subinterpreters as the mechanism by which we can achieve multi-core parallelism
in Python code. Moving the GIL
New submission from Eric Snow :
(See Include/internal/pycore_warnings.h and Python/_warnings.c.)
The warnings module's state (filters, default action, etc.) is currently stored
at the level of the global runtime. That's a problem for the following reasons:
* Python objects are getting
Change by Eric Snow :
--
nosy: +eric.snow
___
Python tracker
<https://bugs.python.org/issue36710>
___
___
Python-bugs-list mailing list
Unsubscribe:
Eric Snow added the comment:
Here's a basic decorator along those lines, similar to one that I've used on
occasion:
def as_thread(target):
def _target():
try:
t.result = target()
except Exception as exc:
t.failure = exc
Eric Snow added the comment:
Please don't miss the fact that the main reason for mirroring the dict table is
to get O(1) node lookup (in the linked list). Otherwise most lookup-dependent
operations, like __delitem__(), would become O(n); whereas in the pure-Python
implementation they are O
Eric Snow added the comment:
@Victor, I set up a FreeBSD 12.0 VM (in Hyper-v) and made sure core files were
getting generated for segfaults. Then I cloned the cpython repo, built it
(using GCC), and ran regrtest as you recommended. It generated no core files
after half an hour. I
Eric Snow added the comment:
New changeset b75b1a3504a0cea6fac6ecba44c10b2629577025 by Eric Snow in branch
'master':
bpo-33608: Revert "Factor out a private, per-interpreter _Py_AddPendingCall()."
(gh-12806)
https://github.com/python/cpython/commit/b75b1a3504a0cea6fac6ecba44c10b
Change by Eric Snow :
--
pull_requests: +12733
___
Python tracker
<https://bugs.python.org/issue33608>
___
___
Python-bugs-list mailing list
Unsubscribe:
Eric Snow added the comment:
Thanks for checking, Victor. Don't feel bad about your results, nor about not
checking sooner. :) We'll get this sorted out.
For now I'll revert. This is not code that changes very often, so there isn't
much benefit to keeping it merged. Testing against
Eric Snow added the comment:
New changeset f13c5c8b9401a9dc19e95d8b420ee100ac022208 by Eric Snow in branch
'master':
bpo-33608: Factor out a private, per-interpreter _Py_AddPendingCall().
(gh-12360)
https://github.com/python/cpython/commit/f13c5c8b9401a9dc19e95d8b420ee100ac022208
Eric Snow added the comment:
Thanks for checking in, Cheryl!
Clearly no one picked up this banner. :) Furthermore, I'm not aware of any
reload-related complaints coming from the community. I know of only a couple
major use cases for reload(): refresh parts of a running app during
Change by Eric Snow :
--
versions: +Python 3.8 -Python 3.7
___
Python tracker
<https://bugs.python.org/issue32280>
___
___
Python-bugs-list mailing list
Unsub
Eric Snow added the comment:
I should have added something like this earlier, but here are key points to
consider covering:
* "main" interpreter is the original, created when the runtime initializes
* historically almost always the only Python interpreter in a process
* this i
Eric Snow added the comment:
Thanks for working on this, Joannah! :)
--
resolution: -> fixed
stage: patch review -> resolved
status: open -> closed
___
Python tracker
<https://bugs.python.or
Change by Eric Snow :
--
nosy: +eric.snow
___
Python tracker
<https://bugs.python.org/issue36487>
___
___
Python-bugs-list mailing list
Unsubscribe:
Eric Snow added the comment:
I'll take a look when I get a chance.
--
___
Python tracker
<https://bugs.python.org/issue33356>
___
___
Python-bugs-list mailin
Eric Snow added the comment:
> Currently PyEval_RestoreThread and its callers (mainly PyGILState_Ensure)
> can terminate the thread if the interpreter is finalizing:
s/interpreter/runtime/
Most likely (guaranteed?) it will be in the main interpreter, but it is
actually not tri
New submission from Eric Snow :
Currently when a thread acquires the GIL, it subsequently exits if the runtime
is finalizing. This helps with some cases like with stopping daemon threads.
This behavior should instead be triggered by the thread's interpreter
finalizing rather than
Eric Snow added the comment:
> > if (_Py_IsFinalizing() && !_Py_CURRENTLY_FINALIZING(tstate))
>
> _Py_IsFinalizing() check is redundant :-)
Not really:
* _Py_IsFinalizing() checks if the runtime is finalizing
* _Py_CURRENTLY_FINALIZING checks if the current threa
Eric Snow added the comment:
Related:
* #36475: "PyEval_AcquireLock() and PyEval_AcquireThread() do not handle
runtime finalization properly."
* #36476: "Runtime finalization assumes all other threads have exited."
--
___
Py
Eric Snow added the comment:
As I noted on the PR, this might be a good chance to make sure the C-API docs
are clear about what the "main" interpreter is.
--
___
Python tracker
<https://bugs.python.o
Change by Eric Snow :
--
resolution: -> fixed
stage: patch review -> resolved
status: open -> closed
___
Python tracker
<https://bugs.python.or
Eric Snow added the comment:
test__xxsubinterpreters is a great place for tests that exercise use of
subinterpreters, including most lifecycle operations. There are also one or
two subinterpreter-related tests in test_embed. However, for this issue the
interplay with runtime finalization
Eric Snow added the comment:
Interestingly, I noticed this independently today. :)
Here's what I wrote in #36477 (which I've closed as a duplicate):
When using subinterpreters, any that exist when Py_FinalizeEx() is called do
not appear to get cleaned up during runtime finalization. Maybe
Change by Eric Snow :
--
resolution: -> duplicate
stage: needs patch -> resolved
status: open -> closed
superseder: -> Lingering subinterpreters should be implicitly cleared on
shutdown
___
Python tracker
<https://bugs.python
Eric Snow added the comment:
@Remy, aside from the recommendations I've made, I'm not sure what else we can
do to help. Before we close the issue, I'd really like to ensure that one of
those threads is holding the GIL still. It would definitely be a problem if a
thread exited while still
Eric Snow added the comment:
FYI, I've opened issue36477 to deal with the subinterpreters case.
--
___
Python tracker
<https://bugs.python.org/issue36
New submission from Eric Snow :
When using subinterpreters, any that exist when Py_FinalizeEx() is called do
not appear to get cleaned up during runtime finalization. Maybe I've been
looking at the code too much and I'm missing something. :)
This really isn't a problem except for embedders
Eric Snow added the comment:
I've also opened issues #36476 and #36477.
--
___
Python tracker
<https://bugs.python.org/issue36469>
___
___
Python-bugs-list m
New submission from Eric Snow :
Among the first 3 things that happen in Py_FinalizeEx() are, in order:
1. wait for all non-daemon threads (of the main interpreter) to finish
2. call registered atexit funcs
3. mark the runtime as finalizing
At that point the only remaining Python threads
Change by Eric Snow :
--
components: +Interpreter Core
___
Python tracker
<https://bugs.python.org/issue36475>
___
___
Python-bugs-list mailing list
Unsubscribe:
Eric Snow added the comment:
Looking at the stack traces for all your threads (super helpful, BTW), I saw 4
groups:
* (1) main thread (blocked on GIL during runtime finalization)
+ Thread 1
* (12) blocked on GIL in call to PyEval_RestoreThread() (socket, select,
time.sleep(), file.read
Eric Snow added the comment:
I've opened issue36475 for the two C-API functions that do not cause daemon
threads to exit.
--
___
Python tracker
<https://bugs.python.org/issue36
New submission from Eric Snow :
Daemon threads keep running until they finish or until finalization starts.
For the latter, there is a check right after the thread acquires the GIL which
causes the thread to exit if runtime finalization has started. [1] However,
there are functions
Eric Snow added the comment:
Here are some things that would likely help:
* in the main thread stop your daemon threads if you hit SystemExit
* make sure daemon threads release/acquire the GIL frequently (so they notice
finalization)
* make sure daemon threads otherwise exit promptly
Eric Snow added the comment:
At this point I think it's likely that the problem relates to how daemon
threads are handled during runtime finalization.
What normally happens in the main thread of the "python3" executable is this:
1. Python runtime initializes
2. main interpreter i
Eric Snow added the comment:
* Are any signals (or signal handlers) involved?
* Are you monkeypatching any builtins, builtin/stdlib modules (or any parts of
those)?
Keep in mind that a number of bugs have been fixed in later releases related to
the various things I've asked about
Change by Eric Snow :
--
nosy: +eric.snow
___
Python tracker
<https://bugs.python.org/issue30703>
___
___
Python-bugs-list mailing list
Unsubscribe:
Eric Snow added the comment:
Also:
* are any of your threads daemon threads?
--
___
Python tracker
<https://bugs.python.org/issue36469>
___
___
Python-bug
Eric Snow added the comment:
As Inada-san indicated, the problem might be resolved already. So without the
option of reproducing the problem, it will be hard to resolve this.
Here's some information you could provide that might help narrow down the scope
a bit:
* what (stdlib/third-party
Eric Snow added the comment:
In Python, multiplication on a list does not make copies of the values in the
original list. So what you have done is equivalent to the following:
a = [0, 0]
b = [a, a]
M = [b, b]
Hence:
>>> M[0] is M[1]
True
>>> M[0][0] is M[0][1
Eric Snow added the comment:
> Neil Schemenauer added the comment:
> Regarding m_traverse, maybe the list of finalizers should be stored somewhere
> in the interpreter
> state, not in the atexit module state. That would make more sense to me.
> atexit could merely be
&g
Eric Snow added the comment:
@arigo, thanks for nudging us here. :) Let me know if there's anything else
that would help here.
--
resolution: -> fixed
stage: patch review -> resolved
status: open -> closed
___
Python tracke
Eric Snow added the comment:
New changeset d2fdd1fedf6b9dc785cf5025b548a989faed089a by Eric Snow in branch
'master':
bpo-36124: Add PyInterpreterState.dict. (gh-12132)
https://github.com/python/cpython/commit/d2fdd1fedf6b9dc785cf5025b548a989faed089a
Eric Snow added the comment:
New changeset c11183cdcff6af13c4339fdcce84ab63f7930ddc by Eric Snow in branch
'master':
bpo-36097: Use only public C-API in the_xxsubinterpreters module (adding as
necessary). (gh-12359)
https://github.com/python/cpython/commit
Change by Eric Snow :
--
pull_requests: +12327
___
Python tracker
<https://bugs.python.org/issue33608>
___
___
Python-bugs-list mailing list
Unsubscribe:
Change by Eric Snow :
--
pull_requests: +12326
stage: needs patch -> patch review
___
Python tracker
<https://bugs.python.org/issue36097>
___
___
Python-
Eric Snow added the comment:
New changeset 842a2f07f2f08a935ef470bfdaeef40f87490cfc by Eric Snow in branch
'master':
bpo-33608: Deal with pending calls relative to runtime shutdown. (gh-12246)
https://github.com/python/cpython/commit/842a2f07f2f08a935ef470bfdaeef40f87490cfc
Eric Snow added the comment:
New changeset 8479a3426eb7d1840473f7788e639954363ed37e by Eric Snow in branch
'master':
bpo-33608: Make sure locks in the runtime are properly re-created. (gh-12245)
https://github.com/python/cpython/commit/8479a3426eb7d1840473f7788e639954363ed37e
Eric Snow added the comment:
New changeset 5be45a6105d656c551adeee7770afdc3b806fbb5 by Eric Snow in branch
'master':
bpo-33608: Minor cleanup related to pending calls. (gh-12247)
https://github.com/python/cpython/commit/5be45a6105d656c551adeee7770afdc3b806fbb5
601 - 700 of 2619 matches
Mail list logo