Re: [Python-Dev] PEP 556 threaded garbage collection & linear recursion in gc

2019-03-28 Thread Neil Schemenauer
On 2019-03-28, Antoine Pitrou wrote: > On Wed, 27 Mar 2019 15:59:25 -0700 > "Gregory P. Smith" wrote: > > > > That had a C++ stack trace 1000+ levels deep repeating the pattern of > > > > ... > > @ 0x564d59bd21de 32 func_dealloc > > @ 0x564d59bce0c1 32 cell_deal

Re: [Python-Dev] PEP 556 threaded garbage collection & linear recursion in gc

2019-03-28 Thread Jeroen Demeyer
On 2019-03-28 01:38, Tim Peters wrote: The bad news is that the traschcan mechanism is excruciating, a long-time source of subtle bugs of its own :-( It just happens that I created a PR to fix some of the trashcan problems: see https://bugs.python.org/issue35983 and https://github.com/python/

Re: [Python-Dev] PEP 556 threaded garbage collection & linear recursion in gc

2019-03-28 Thread Antoine Pitrou
On Wed, 27 Mar 2019 15:59:25 -0700 "Gregory P. Smith" wrote: > > That had a C++ stack trace 1000+ levels deep repeating the pattern of > > ... > @ 0x564d59bd21de 32 func_dealloc > @ 0x564d59bce0c1 32 cell_dealloc > @ 0x564d5839db41 48 tupledeall

Re: [Python-Dev] PEP 556 threaded garbage collection & linear recursion in gc

2019-03-27 Thread Tim Peters
[Gregory P. Smith ] > Good point, I hadn't considered that it was regular common ref > count 0 dealloc chaining. It pretty much has to be whenever you see a chain of XXX_dealloc routines in a stack trace. gcmodule.c never even looks at a tp_dealloc slot directly, let alone directly invoke a deall

Re: [Python-Dev] PEP 556 threaded garbage collection & linear recursion in gc

2019-03-27 Thread Gregory P. Smith
Good point, I hadn't considered that it was regular common ref count 0 dealloc chaining. The processes unfortunately didn't have faulthandler enabled so there wasn't much info from where in the python code it happened (now fixed). I'll see if anything looks particularly unusual next time I hear o

Re: [Python-Dev] PEP 556 threaded garbage collection & linear recursion in gc

2019-03-27 Thread Tim Peters
[Gregory P. Smith ] > ... > A situation came up the other day where I believe this could've helped. > > Scenario (admittedly not one most environments run into): A Python process > with a C++ extension module implementing a threaded server (threads > spawned by C++) that could call back into CPytho

[Python-Dev] PEP 556 threaded garbage collection & linear recursion in gc

2019-03-27 Thread Gregory P. Smith
https://www.python.org/dev/peps/pep-0556/ This PEP is currently Deferred as nobody is actively working on a test implementation. A situation came up the other day where I *believe* this could've helped. Scenario (admittedly not one most environments run into): A Python process with a C++ extensi

Re: [Python-Dev] PEP 556: Threaded garbage collection

2017-09-09 Thread Nick Coghlan
On 8 September 2017 at 12:40, Nathaniel Smith wrote: > Would it make sense to also move signal handlers to run in this > thread? Those are the other major source of nasty re-entrancy > problems. Python level signal handlers are already only run in the main thread, so applications that want to ens

Re: [Python-Dev] PEP 556: Threaded garbage collection

2017-09-08 Thread Gregory P. Smith
On Fri, Sep 8, 2017 at 12:52 PM Antoine Pitrou wrote: > On Fri, 8 Sep 2017 12:40:34 -0700 > Nathaniel Smith wrote: > > > > PyPy just abandons everything when shutting down, instead of running > > finalizers. See the last paragraph of : > > > http://doc.pypy.org/en/latest/cpython_differences.html

Re: [Python-Dev] PEP 556: Threaded garbage collection

2017-09-08 Thread Antoine Pitrou
On Fri, 8 Sep 2017 12:40:34 -0700 Nathaniel Smith wrote: > > PyPy just abandons everything when shutting down, instead of running > finalizers. See the last paragraph of : > http://doc.pypy.org/en/latest/cpython_differences.html#differences-related-to-garbage-collection-strategies > > So that mi

Re: [Python-Dev] PEP 556: Threaded garbage collection

2017-09-08 Thread Larry Hastings
On 09/08/2017 12:30 PM, Benjamin Peterson wrote: On Fri, Sep 8, 2017, at 12:24, Larry Hastings wrote: On 09/08/2017 12:04 PM, Benjamin Peterson wrote: - Why not run all (Python) finalizers on the thread and not just ones from cycles? Two reasons: 1. Because some code relies on the final

Re: [Python-Dev] PEP 556: Threaded garbage collection

2017-09-08 Thread Nathaniel Smith
On Fri, Sep 8, 2017 at 12:13 PM, Antoine Pitrou wrote: > On Fri, 08 Sep 2017 12:04:10 -0700 > Benjamin Peterson wrote: >> I like it overall. >> >> - I was wondering what happens during interpreter shutdown. I see you >> have that listed as a open issue. How about simply shutting down the >> final

Re: [Python-Dev] PEP 556: Threaded garbage collection

2017-09-08 Thread Benjamin Peterson
On Fri, Sep 8, 2017, at 12:13, Antoine Pitrou wrote: > On Fri, 08 Sep 2017 12:04:10 -0700 > Benjamin Peterson wrote: > > I like it overall. > > > > - I was wondering what happens during interpreter shutdown. I see you > > have that listed as a open issue. How about simply shutting down the > >

Re: [Python-Dev] PEP 556: Threaded garbage collection

2017-09-08 Thread Benjamin Peterson
On Fri, Sep 8, 2017, at 12:24, Larry Hastings wrote: > > > On 09/08/2017 12:04 PM, Benjamin Peterson wrote: > > - Why not run all (Python) finalizers on the thread and not just ones > > from cycles? > > Two reasons: > > 1. Because some code relies on the finalizer being called on the thread

Re: [Python-Dev] PEP 556: Threaded garbage collection

2017-09-08 Thread Antoine Pitrou
On Fri, 08 Sep 2017 12:04:10 -0700 Benjamin Peterson wrote: > I like it overall. > > - I was wondering what happens during interpreter shutdown. I see you > have that listed as a open issue. How about simply shutting down the > finalization thread and not guaranteeing that finalizers are actually

Re: [Python-Dev] PEP 556: Threaded garbage collection

2017-09-08 Thread Larry Hastings
On 09/08/2017 12:04 PM, Benjamin Peterson wrote: - Why not run all (Python) finalizers on the thread and not just ones from cycles? Two reasons: 1. Because some code relies on the finalizer being called on the thread where the last reference is dropped. This is usually the same thread

Re: [Python-Dev] PEP 556: Threaded garbage collection

2017-09-08 Thread Benjamin Peterson
I like it overall. - I was wondering what happens during interpreter shutdown. I see you have that listed as a open issue. How about simply shutting down the finalization thread and not guaranteeing that finalizers are actually ever run à la Java? - Why not run all (Python) finalizers on the thre

Re: [Python-Dev] PEP 556: Threaded garbage collection

2017-09-08 Thread Antoine Pitrou
On Fri, 8 Sep 2017 10:03:33 -0700 Nick Coghlan wrote: > On 8 September 2017 at 08:05, Antoine Pitrou wrote: > > > > Hello, > > > > I've written a PEP by which you can tell the GC to run in a dedicated > > thread. The goal is to solve reentrancy issues with finalizers: > > https://www.python.org

Re: [Python-Dev] PEP 556: Threaded garbage collection

2017-09-08 Thread Nick Coghlan
On 8 September 2017 at 08:05, Antoine Pitrou wrote: > > Hello, > > I've written a PEP by which you can tell the GC to run in a dedicated > thread. The goal is to solve reentrancy issues with finalizers: > https://www.python.org/dev/peps/pep-0556/ +1 from me for the general concept. (Minor naming

[Python-Dev] PEP 556: Threaded garbage collection

2017-09-08 Thread Antoine Pitrou
Hello, I've written a PEP by which you can tell the GC to run in a dedicated thread. The goal is to solve reentrancy issues with finalizers: https://www.python.org/dev/peps/pep-0556/ Regards Antoine. PS: I did not come up with the idea for this PEP while other people were having nightmares.