[issue812369] module shutdown procedure based on GC

[Antoine Pitrou]

Changes by Antoine Pitrou :


--
superseder: module shutdown procedure based on GC -> Stop purging modules which 
are garbage collected before shutdown

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue812369] module shutdown procedure based on GC

[Antoine Pitrou]

Antoine Pitrou added the comment:

Superceded by patch in issue 18214.

--
resolution:  -> duplicate
status: open -> closed
superseder:  -> module shutdown procedure based on GC

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue812369] module shutdown procedure based on GC

[Nick Coghlan]

Nick Coghlan added the comment:

In addition to the problem Neil noted with references in extension modules 
keeping module objects themselves alive, Antoine recently noted that the other 
major challenge is the reference cycles between module global dictionaries and 
their contents. As soon as a module global has both a __del__ method and a 
reference back to the module globals, the entire cycle becomes uncollectable. I 
suspect one of the reasons PyPy can cope without the explicit reference 
breaking step is that their GC is better able to cope with __del__ methods than 
ours.

I wonder if a useful interim step might be to make the current explicit 
reference breaking hack a bit smarter by looking at the reference counts. 
(Note: some aspects of this idea could be made simpler if modules supported 
weak references)

1. Call importlib.invalidate_caches()
2. Delete the first module from sys.modules that has a reference count of 
exactly one
3. Repeat 2 until sys.modules is empty or every remaining module has a 
reference count greater than 1 (meaning another module has a reference to it 
one way or another)
4. Pick the module in sys.modules with the lowest number of references to it, 
delete it from sys.modules and delete the reference from the module object to 
its dictionary
5. Repeat 4 until sys.modules is empty

Throughout the process, keep an eye on gc.garbage - if we see a module dict 
show up there, hit it with the "set all globals to None" hammer. (The new 
callback functionality in 3.3 makes that easier - for example, you could put a 
sentinel object in the globals of the module being cleared and watching for a 
dict containing that sentinel object showing up in 'uncollectable' during the 
stop phase)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue812369] module shutdown procedure based on GC

[Andrew Svetlov]

Changes by Andrew Svetlov :


--
nosy: +asvetlov

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue812369] module shutdown procedure based on GC

[Eric Snow]

Changes by Eric Snow :


--
nosy: +eric.snow

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue812369] module shutdown procedure based on GC

[Gregory P. Smith]

Changes by Gregory P. Smith :


--
assignee: gregory.p.smith -> 

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue812369] module shutdown procedure based on GC

[Neil Schemenauer]

Neil Schemenauer added the comment:

It's been quite a long time since I played with this patch so my memory might 
be a bit fuzzy.  As I recall, it sounds good in theory but in practice it 
doesn't really work.  One of the core problems is that many extension modules 
keep references to Python objects in global or static variables.  These 
references keep pretty much everything alive and prevent GC cleanup of modules.

So, a necessary condition to this working is to get rid of those references and 
use the new module struct facility introduced by Martin.  That would be a huge 
amount of work but I think should be the long term goal.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue812369] module shutdown procedure based on GC

[Martin v . Löwis]

Martin v. Löwis added the comment:

At the moment, it's like that the status of the patch needs to be 
reestablished. Does it apply? Does it work? Does the test suite still pass?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue812369] module shutdown procedure based on GC

[Stefan Friesel]

Stefan Friesel added the comment:

What is the status of this? Does the patch need more reviewing?

--
nosy: +Stefan.Friesel

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue812369] module shutdown procedure based on GC

[Armin Rigo]

Armin Rigo  added the comment:

Obviously we run atexit code too.  There is no point in having atexit if it's 
not guaranteed to run in a normal shutdown.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue812369] module shutdown procedure based on GC

[Martin v . Löwis]

Martin v. Löwis  added the comment:

> (This is just a report about PyPy's situation; I understand that the
> situation in CPython is a bit more delicate if CPython is embedded in
> a larger process.)

I think that would indeed be unacceptable for Python - there is a
long-standing expectation that we free all memory that we allocated,
as well as release any other resources that we hold. There are also
expectations wrt. running atexit code. So there clearly must be a
shutdown procedure.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue812369] module shutdown procedure based on GC

[Nick Coghlan]

Nick Coghlan  added the comment:

Also, since this issue was last updated, Antoine devised a scheme to test some 
of the embedding functionality (mainly to test subinterpreters, IIRC). Perhaps 
that could be harnessed to check GC-based shutdown is working correctly (it 
might even do it already, without any changes to the test suite).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue812369] module shutdown procedure based on GC

[Armin Rigo]

Armin Rigo  added the comment:

Fwiw, the behavior in PyPy is: don't do anything particular at shut-down, just 
shut down and quit the process.  No hacking at module globals to replace them 
with None, but also no guaranteeing that any __del__ method is ever called.  We 
didn't get a particular bug report about this so far, so it seems to work.

(This is just a report about PyPy's situation; I understand that the situation 
in CPython is a bit more delicate if CPython is embedded in a larger process.)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue812369] module shutdown procedure based on GC

[Florent Xicluna]

Changes by Florent Xicluna :


--
nosy: +flox

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue812369] module shutdown procedure based on GC

[Nick Coghlan]

Nick Coghlan  added the comment:

In #14035, Florent pointed out the current behaviour potentially causes 
problems for some uses of import_fresh_modules() in the test suite (with 
globals sometimes being set to None if there's no indepenent reference to the 
module).

GC based module cleanup would avoid that problem automatically.

--
nosy: +ncoghlan

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue812369] module shutdown procedure based on GC

[Antoine Pitrou]

Antoine Pitrou  added the comment:

The patch is obviously against 2.x (there are some PyString_Check's on module 
names, for example). It should be regenerated against 3.x.

Also, it would be nice if a test could be devised to check that the shutdown 
procedure works as expected (I'm not sure how such a test would look like).

--
versions:  -Python 2.7, Python 3.1

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue812369] module shutdown procedure based on GC

[Gregory P. Smith]

Gregory P. Smith  added the comment:

0001-update-GC-shutdown-patch.patch looks sane to me at first glance.  any 
other opinions?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue812369] module shutdown procedure based on GC

[Mark Lawrence]

Mark Lawrence  added the comment:

#1545463 has been reopened with comments about being used as a stop gap.  
Possibly review and implementation of the patch here would be a better option, 
sorry it's over my head.

--
versions: +Python 3.2

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue812369] module shutdown procedure based on GC

[Andrea Corbellini]

Changes by Andrea Corbellini :


--
nosy: +candrea

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue812369] module shutdown procedure based on GC

[Mark Lawrence]

Mark Lawrence  added the comment:

issue1545463 has been closed as "won't fix", so wouldn't implementing this 
patch kill two birds with one stone?

--
nosy: +BreamoreBoy

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue812369] module shutdown procedure based on GC

[cburroughs]

Changes by cburroughs :


--
nosy: +cburroughs

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue812369] module shutdown procedure based on GC

[Neil Schemenauer]

Neil Schemenauer  added the comment:

It should fix issue1545463 and running a quick test seems to show that
it does.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue812369] module shutdown procedure based on GC

[Gregory P. Smith]

Changes by Gregory P. Smith :


--
assignee:  -> gregory.p.smith
nosy: +gregory.p.smith

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue812369] module shutdown procedure based on GC

[Alexander Belopolsky]

Alexander Belopolsky  added the comment:

Does this patch fix issue1545463 by any chance?  I am away from a 
development box ATM and cannot test the patch myself.

--
nosy: +belopolsky

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue812369] module shutdown procedure based on GC

[Antoine Pitrou]

Antoine Pitrou  added the comment:

Retargetting, and I hope someone can take a look at the patch and give
it the green light :-)

--
stage:  -> patch review
type:  -> behavior
versions: +Python 2.7, Python 3.1 -Python 2.6, Python 3.0

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue812369] module shutdown procedure based on GC

[Neil Schemenauer]

Neil Schemenauer  added the comment:

This sounds like a nice idea.  The current cleanup procedure in
pythonrun.c is pretty lame since it can play havoc with __del__ methods
(e.g. if they run after globals have been cleared).

I've updated the patch to work with the current SVN head.  Probably this
should get tested with big applications based on Zope, Django, etc.

--
nosy: +nascheme
Added file: http://bugs.python.org/file13091/0001-update-GC-shutdown-patch.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue812369] module shutdown procedure based on GC

[Antoine Pitrou]

Antoine Pitrou  added the comment:

Looking at the patch, is there any reason it doesn't get rid of the
current _PyModule_Clear() implementation to replace it by a call to
PyDict_Clear() followed by PyGC_Collect()?
(the call to PyGC_Collect could be disabled while finalizing, because
there's no use calling it as many times as there are modules to be
disbanded)

The major annoyance with the current scheme is that, at interpreter
shutdown, some globals you want to rely on in your destructors suddenly
become None.

About what to do of gc.garbage at shutdown, there was another proposal
in #477863.

--
nosy: +pitrou

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue812369] module shutdown procedure based on GC

[Christian Heimes]

Christian Heimes added the comment:

Consider this patch for 2.6 and discuss it at the bug day.

--
nosy: +tiran
versions: +Python 2.6, Python 3.0 -Python 2.4


Tracker <[EMAIL PROTECTED]>


___
Python-bugs-list mailing list 
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com