[issue38379] finalizer resurrection in gc

[Eryk Sun]

Change by Eryk Sun :


--
Removed message: https://bugs.python.org/msg405696

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38379] finalizer resurrection in gc

[Eryk Sun]

Change by Eryk Sun :


--
components: +Interpreter Core -IDLE
nosy:  -ahmedsayeed1982, terry.reedy
versions: +Python 3.7, Python 3.8, Python 3.9 -Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38379] finalizer resurrection in gc

[Ahmed Sayeed]

Ahmed Sayeed  added the comment:

If you create a new TUI layout, don't include the status window, and then 
change from a layout with the status window to the new one, gdb crashes. 
http://www.compilatori.com/category/tech/

(gdb) layout src
(gdb) tui new-layout test src 2 cmd 1 
http://www.acpirateradio.co.uk/category/tech/
(gdb) layout test
http://www.logoarts.co.uk/category/tech/
On Windows I get a STATUS_HEAP_CORRUPTION exception (0xc374). 
http://www.mconstantine.co.uk/property/netherlands/
It happens because tui_apply_current_layout() deletes all windows that are no 
longer  http://www.go-mk-websites.co.uk/property/miyazaki/ needed, but the 
status (locator) window is never allocated dynamically.  
http://www.slipstone.co.uk/category/tech/
If you create a new TUI layout, don't include the status window, and then 
http://fishingnewsletters.co.uk/health/adrasan/ change from a layout with the 
status window to the new one, gdb crashes.
http://embermanchester.uk/category/tech/
(gdb) layout src
(gdb) tui new-layout test src 2 cmd 1 http://connstr.net/category/tech/
(gdb) layout test
 http://joerg.li/category/tech/
On Windows I get a STATUS_HEAP_CORRUPTION exception (0xc374).
It happens because tui_apply http://www.jopspeech.com/category/tech/ 
_current_layout() deletes all windows that are no longer needed, 
http://the-hunters.org/services/miui-12-5/ but the status (locator) window is 
never allocated dynamically.
http://www.wearelondonmade.com/category/tech/
If you create a new TUI layout, don't include the status window, and then 
change from a layout with the status window to the new one, gdb crashes. 
https://waytowhatsnext.com/category/property/

(gdb) layout src
(gdb) tui new-layout test src 2 cmd 1 
http://www.iu-bloomington.com/category/property/
(gdb) layout test
https://komiya-dental.com/category/property/
On Windows I get a STATUS_HEAP_CORRUPTION exception (0xc374).
It happens because tui 
http://www-look-4.com/category/tech/_apply_current_layout() deletes all windows 
that are no longer needed, but the status (locator) window is never allocated 
dynamically. https://www.webb-dev.co.uk/category/property/

--
assignee:  -> terry.reedy
components: +IDLE -Interpreter Core
nosy: +ahmedsayeed1982, terry.reedy -LewisGaul, miss-islington, 
nanjekyejoannah, nascheme, pablogsal, pitrou, tim.peters
versions: +Python 3.6 -Python 3.7, Python 3.8, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38379] finalizer resurrection in gc

[Lewis Gaul]

Lewis Gaul  added the comment:

You're right that's how I had interpreted it, thanks for clarifying.

I was wondering if this could be related to an issue I've hit with gc.collect() 
getting slower and slower in a test suite, but that now seems unlikely, so I 
won't go into that here.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38379] finalizer resurrection in gc

[Tim Peters]

Tim Peters  added the comment:

I suspect you're reading some specific technical meaning into the word "block" 
that the PR and release note didn't intend by their informal use of the word. 
But I'm unclear on what technical meaning you have in mind.

Before the change, gc "just gave up" after seeing a resurrection, ending the 
then-current cyclic gc run. It that sense, yes, resurrection "blocked" gc from 
making progress. It did not, e.g., "block" the interpreter in the sense of 
deadlock, or of waiting for some lock to be released, or of waiting for a 
network request to respond, or ...

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38379] finalizer resurrection in gc

[Lewis Gaul]

Lewis Gaul  added the comment:

I noticed this bug is mentioned in the 3.9 release notes with a note similar to 
the title of the 4th PR: "garbage collection does not block on resurrected 
objects".

I can't see any mention of a blocking issue here on the issue:

> The bug:  the stats keep claiming gc is collecting an enormous number of 
> objects, but in fact it's not collecting any.  Objects in the unreachable set 
> shouldn't add to the "collected" count unless they _are_ collected.

Would someone be able to elaborate on the blocking issue that was fixed as part 
of this BPO?

--
nosy: +LewisGaul

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38379] finalizer resurrection in gc

[Tim Peters]

Tim Peters  added the comment:

Everything here has been addressed, so closing this.  zleak.py can apparently 
run forever now without leaking a byte :-)

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38379] finalizer resurrection in gc

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:


New changeset 466326dcdf038b948d94302c315be407c73e60d1 by Pablo Galindo in 
branch 'master':
bpo-38379: Don't block collection of unreachable objects when some objects 
resurrect (GH-16687)
https://github.com/python/cpython/commit/466326dcdf038b948d94302c315be407c73e60d1


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38379] finalizer resurrection in gc

[Pablo Galindo Salgado]

Change by Pablo Galindo Salgado :


--
pull_requests: +16273
pull_request: https://github.com/python/cpython/pull/16687

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38379] finalizer resurrection in gc

[miss-islington]

miss-islington  added the comment:


New changeset a081e931505f190b5ccdff9a781e59b3f13fcc31 by Miss Islington (bot) 
(Pablo Galindo) in branch '3.7':
[3.7] bpo-38379:  don't claim objects are collected when they aren't (GH-16658) 
(GH-16685)
https://github.com/python/cpython/commit/a081e931505f190b5ccdff9a781e59b3f13fcc31


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38379] finalizer resurrection in gc

[miss-islington]

miss-islington  added the comment:


New changeset 0bd9fac7a866ec886ae8f93f9c24dcda6d436929 by Miss Islington (bot) 
(Pablo Galindo) in branch '3.8':
[3.8] bpo-38379:  don't claim objects are collected when they aren't  
(GH-16658) (GH-16683)
https://github.com/python/cpython/commit/0bd9fac7a866ec886ae8f93f9c24dcda6d436929


--
nosy: +miss-islington

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38379] finalizer resurrection in gc

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

Tim, I have created backports for 3.8 and 3.7 (PR 16683, PR 16685). In my case 
cherry_picker ecbf35f9335b0420cb8adfda6f299d6747a16515 3.7 and cherry_picker 
ecbf35f9335b0420cb8adfda6f299d6747a16515 3.8 works after fixing merge 
conflicts. Maybe there is something going on in your git repo (maybe you didn't 
fetch from master first and that is why git log fails?).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38379] finalizer resurrection in gc

[Pablo Galindo Salgado]

Change by Pablo Galindo Salgado :


--
pull_requests: +16271
pull_request: https://github.com/python/cpython/pull/16685

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38379] finalizer resurrection in gc

[Pablo Galindo Salgado]

Change by Pablo Galindo Salgado :


--
pull_requests: +16269
stage: backport needed -> patch review
pull_request: https://github.com/python/cpython/pull/16683

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38379] finalizer resurrection in gc

[Tim Peters]

Tim Peters  added the comment:

I checked the stat fix into master, but GH failed to backport to 3.7 or 3.8 and 
I'm clueless.  More info in the PR.  Does someone else here know how to get a 
backport done?

--
stage: patch review -> backport needed
versions: +Python 3.7, Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38379] finalizer resurrection in gc

[Tim Peters]

Tim Peters  added the comment:


New changeset ecbf35f9335b0420cb8adfda6f299d6747a16515 by Tim Peters in branch 
'master':
bpo-38379:  don't claim objects are collected when they aren't (#16658)
https://github.com/python/cpython/commit/ecbf35f9335b0420cb8adfda6f299d6747a16515


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38379] finalizer resurrection in gc

[Pablo Galindo Salgado]

Change by Pablo Galindo Salgado :


--
nosy: +pablogsal

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38379] finalizer resurrection in gc

[Tim Peters]

Tim Peters  added the comment:

PR 16658 aims to repair the stats reported.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38379] finalizer resurrection in gc

[Tim Peters]

Change by Tim Peters :


--
keywords: +patch
pull_requests: +16241
stage: needs patch -> patch review
pull_request: https://github.com/python/cpython/pull/16658

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38379] finalizer resurrection in gc

[Joannah Nanjekye]

Change by Joannah Nanjekye :


--
nosy: +nanjekyejoannah

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38379] finalizer resurrection in gc

[Tim Peters]

Tim Peters  added the comment:

Just noting that check_garbage() currently only determines which trash objects 
are now directly reachable from outside.  To be usable for the intended 
purpose, it would need to go on to compute which trash objects are reachable 
from those too.

Maybe a new function

static void
deduce_unreachable(PyGC_Head *base, PyGC_Head *unreachable)

that packaged the steps through move_unreachable().  The main function would 
use that on `young`, and check_garbage() on `unreachable` to find the _still_ 
unreachable objects.

I don't care about the expense.  Outside of zleak.py, the number of unreachable 
objects is usually small, so it's usually cheap to make another pass over just 
them.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38379] finalizer resurrection in gc

[Tim Peters]

New submission from Tim Peters :

While people are thinking about gc, zleak.py shows a small bug, and a possible 
opportunity for improvement, in the way gc treats finalizers that resurrect 
objects.

The bug:  the stats keep claiming gc is collecting an enormous number of 
objects, but in fact it's not collecting any.  Objects in the unreachable set 
shouldn't add to the "collected" count unless they _are_ collected.

Output:

resurrecting
collect 202
gen 2 stats {'collections': 2, 'collected': 202, 'uncollectable': 0}
resurrecting
collect 404
gen 2 stats {'collections': 3, 'collected': 606, 'uncollectable': 0}
resurrecting
collect 606
gen 2 stats {'collections': 4, 'collected': 1212, 'uncollectable': 0}
resurrecting
collect 808
gen 2 stats {'collections': 5, 'collected': 2020, 'uncollectable': 0}
resurrecting
collect 1010
gen 2 stats {'collections': 6, 'collected': 3030, 'uncollectable': 0}
...

Memory use grows without bound, and collections take ever longer.

The opportunity:  if any finalizer resurrects anything, gc gives up.  But the 
process of computing whether anything was resurrected also determines which 
initially-trash objects are reachable from the risen dead.  Offhand I don't see 
why we couldn't proceed collecting what remains trash.  Then zleak.py would 
reclaim everything instead of nothing.

Sketch:  rather than just set a flag, check_garbage() could move now-reachable 
objects to the old generation (and, for each one moved, decrement the count of 
collected objects).  Then delete_garbage() could proceed on what remains in the 
unreachable list.

--
components: Interpreter Core
files: zleak.py
messages: 354020
nosy: nascheme, pitrou, tim.peters
priority: normal
severity: normal
stage: needs patch
status: open
title: finalizer resurrection in gc
type: behavior
versions: Python 3.9
Added file: https://bugs.python.org/file48644/zleak.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com