Antoine Pitrou added the comment:
It appears that a few (1-3) module dicts are not being purged because they
have been orphaned. (i.e. the module object was garbaged collected before
we check the weakref, but the module dict survived.)
Module globals can be kept alive by any function
Changes by Christian Heimes li...@cheimes.de:
--
nosy: +christian.heimes
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18214
___
___
Antoine Pitrou added the comment:
As for the __main__ dict, perhaps we're missing a decref somewhere.
Actually, it's not surprising. Blob's methods hold a reference to the __main__
globals, and there's still a Blob object alive in encodings.
If you replace the end of your script with the
Richard Oudkerk added the comment:
On 01/08/2013 10:59am, Antoine Pitrou wrote:
If you replace the end of your script with the following:
for name, mod in sys.modules.items():
if name != 'encodings':
mod.__dict__[__blob__] = Blob(name)
del name, mod, Blob
then at the end
Antoine Pitrou added the comment:
You might want to open a prompt and look at gc.get_referrers() for
encodings.mbcs.__dict__ (or another of those modules).
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18214
Richard Oudkerk added the comment:
You might want to open a prompt and look at gc.get_referrers() for
encodings.mbcs.__dict__ (or another of those modules).
gc.get_referrers(sys.modules['encodings.mbcs'].__dict__)
[module 'encodings.mbcs' from
Richard Oudkerk added the comment:
I get different numbers from you. If I run ./python -v -c pass, most
modules in the wiping phase are C extension modules, which is expected.
Pretty much every pure Python module ends up garbage collected before
that.
The *module* gets gc'ed, sure. But
Antoine Pitrou added the comment:
Here (Linux) I get the following:
# purge/gc os.path 12
# purge/gc operator 50
# purge/gc io 49
# purge/gc _sysconfigdata 48
# purge/gc sysconfig 47
# purge/gc keyword 46
# purge/gc site 45
# purge/gc types 44
Also, do note that purge/gc after wiping can still
Richard Oudkerk added the comment:
Also, do note that purge/gc after wiping can still be a regular
gc pass unless the module has been wiped. The gc could be triggered
by another module being wiped.
For me, the modules which die naturally after purging begins are
# purge/gc encodings.aliases
Antoine Pitrou added the comment:
Also, do note that purge/gc after wiping can still be a regular gc
pass unless the module has been wiped. The gc could be triggered by
another module being wiped.
That said, I welcome any suggestions to improve things. The ultimate
reasons we need to purge
Richard Oudkerk added the comment:
Yes, I agree the patch is ok.
It would be would be much simpler to keep track of the module dicts if
they were weakrefable. Alternatively, at shutdown a weakrefable object
with a reference to the module dict could be inserted in to each module
dict. We
Antoine Pitrou added the comment:
Ok, let's attack the rest separately then :)
And thanks a lot for testing!
--
status: open - closed
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18214
___
Antoine Pitrou added the comment:
By the way, you may be interested to learn that the patch in issue10241 has
made things quite a bit better now: C extension modules can be collected much
earlier.
--
___
Python tracker rep...@bugs.python.org
Antoine Pitrou added the comment:
See issue10068 and issue7140.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18214
___
___
Python-bugs-list
Antoine Pitrou added the comment:
Updated patch has tests and also removes several cleanup hacks.
--
Added file: http://bugs.python.org/file31099/module_cleanup4.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18214
Changes by Arfrever Frehtes Taifersar Arahesis arfrever@gmail.com:
--
nosy: +Arfrever
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18214
___
Antoine Pitrou added the comment:
Updated patch with a hack in Lib/site to unpatch builtins early at shutdown.
--
Added file: http://bugs.python.org/file31101/module_cleanup5.patch
___
Python tracker rep...@bugs.python.org
Roundup Robot added the comment:
New changeset 79e2f5bbc30c by Antoine Pitrou in branch 'default':
Issue #18214: Improve finalization of Python modules to avoid setting their
globals to None, in most cases.
http://hg.python.org/cpython/rev/79e2f5bbc30c
--
nosy: +python-dev
Antoine Pitrou added the comment:
Let's wait for the buildbots on this one too.
--
resolution: - fixed
stage: patch review - committed/rejected
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18214
Richard Oudkerk added the comment:
I played a bit with the patch and -v -Xshowrefcount. The number of references
and blocks left at exit varies (and is higher than for unpatched python).
It appears that a few (1-3) module dicts are not being purged because they have
been orphaned. (i.e. the
Antoine Pitrou added the comment:
Now that PEP 442 is committed, here is the patch.
--
Added file: http://bugs.python.org/file31090/module_cleanup.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18214
Changes by Antoine Pitrou pit...@free.fr:
--
components: +Interpreter Core
nosy: +amaury.forgeotdarc, gregory.p.smith
stage: - patch review
type: - enhancement
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18214
Antoine Pitrou added the comment:
Slightly better patch.
Also, as I pointed out in python-dev
(http://mail.python.org/pipermail/python-dev/2013-July/127673.html), this is
still imperfect due to various ways modules can be kept alive from long-lived C
variables.
--
Added file:
Changes by Antoine Pitrou pit...@free.fr:
Added file: http://bugs.python.org/file31093/module_cleanup3.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18214
___
Changes by Richard Oudkerk shibt...@gmail.com:
--
nosy: +pitrou
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18214
___
___
Python-bugs-list
Antoine Pitrou added the comment:
Usually garbage collection will end up clearing the module's dict anyway.
This is not true, since global objects might have a __del__ and then hold the
whole module dict alive through a reference cycle. Happily though, PEP 442 is
going to make that concern
Richard Oudkerk added the comment:
On 15/06/2013 7:11pm, Antoine Pitrou wrote:
Usually garbage collection will end up clearing the module's dict anyway.
This is not true, since global objects might have a __del__ and then hold
the whole module dict alive through a reference cycle. Happily
Antoine Pitrou added the comment:
As for the interpreter shutdown itself, I have a pending patch (post-PEP
442)
to get rid of the globals cleanup as well. It may be better to merge the
two approaches.
So you would just depend on garbage collection?
No, I also clean up those modules
New submission from Richard Oudkerk:
Currently when a module is garbage collected its dict is purged by replacing
all values except __builtins__ by None. This helps clear things at shutdown.
But this can cause problems if it occurs *before* shutdown: if we use a
function defined in a module
29 matches
Mail list logo
