[issue21351] refcounts not respected at process exit
STINNER Victor added the comment: Never rely on the GC. Avoid cycles by using the weakref module. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21351 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21351] refcounts not respected at process exit
Min RK added the comment: Thanks for clarifying that there is indeed a reference cycle by way of the module, I hadn't realized that. The gc blocking behavior is exactly why I brought up the issue. The real code where this causes a problem (rather than the toy example I attached) is in pyzmq, where destroying a Context object calls `zmq_term`, a GIL-less C call that will (and should) block until all associated sockets are closed. Deleting a socket closes it. Sockets hold a reference to the Context and not vice versa, which has ensured that the sockets are collected before the Context until Python 3.4. Does this mean it is no longer possible to express that one object should be cleaned up before another via references? I think I will switch to adding an atexit call to set a flag that prevents any cleanup logic during the atexit process, since it does not appear to be possible to ensure deletion of one object before another in 3.4. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21351 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21351] refcounts not respected at process exit
Tim Peters added the comment: There's no way to influence finalization order for objects in cycles, and there never was. So nothing actually changed in that respect ;-) What did change is that Python used to forcibly break many module-level cycles in a way that just happened to result in the finalization order you wanted. Outside of cycles it's still as predictable as before. For example, you could reliably get the finalization order you want by replacing the last two lines with p.child() p.child() Then the children aren't in cycles, and will be finalized first. But that's still due to other CPython implementation details (which haven't yet changed), not to language guarantees. But an atexit handler sounds like a saner way to proceed :-) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21351 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21351] refcounts not respected at process exit
Min RK added the comment: Thanks for your help and patience. Closing as slightly unfortunate, but not unintended behavior. -- resolution: - not a bug status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21351 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21351] refcounts not respected at process exit
Changes by STINNER Victor victor.stin...@gmail.com: -- nosy: +haypo ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21351 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21351] refcounts not respected at process exit
Tim Peters added the comment: After more thought, I don't think the user can do anything to influence finalization order in cases like this, short of adding del statements (or moral equivalents) to break cycles before the interpreter shuts down. Fine by me ;-) Something CPython could do, when collecting cyclic trash, is pick on objects with the smallest refcount first. That would most often mimic the finalization-order effects of the old bind-module-globals-to-None hack. But it would require more code and more expense to impose a partial order, and would still be an implementation detail specific to CPython (the implementation, as opposed to Python the language). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21351 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21351] refcounts not respected at process exit
New submission from Min RK: Reference counts appear to be ignored at process cleanup, which allows inter-dependent `__del__` methods to hang on exit. The problem does not seem to occur for garbage collection of any other context (functions, etc.). I have a case where one object must be cleaned up after some descendent objects. Those descendents hold a reference on the parent and not vice versa, which should guarantee that they are cleaned up before the parent. This guarantee is satisfied by Python 3.3 and below, but not 3.4. The attached test script hangs at exit on most (not all) runs on 3.4, but exits cleanly on earlier versions. -- components: Interpreter Core files: tstgc.py messages: 217168 nosy: minrk priority: normal severity: normal status: open title: refcounts not respected at process exit versions: Python 3.4 Added file: http://bugs.python.org/file35041/tstgc.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21351 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21351] refcounts not respected at process exit
Nathan Stocks added the comment: This affects me as well. I have to manually clean up objects in the correct order in script I am working on under 3.4.0. I have this problem under both OS X 10.9.2 Mavericks and under CentOS 6.5 -- nosy: +nathan.stocks ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21351 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21351] refcounts not respected at process exit
Tim Peters added the comment: Just noting that, for me, the problem goes away if del c, c2 is added as the last line of the test. This suggests the problem is due to changes in end-of-life module cleanup. Without that line, I see 3 kinds of output: 1. del child del child del parent parent deleted 2. del parent parent still has 2 children parent still has 2 children ... repeated forever ... 3. del child del parent parent still has 1 children parent still has 1 children ... repeated forever ... -- nosy: +tim.peters ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21351 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21351] refcounts not respected at process exit
Changes by Tim Peters t...@python.org: -- nosy: +pitrou ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21351 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21351] refcounts not respected at process exit
Antoine Pitrou added the comment: (the 3 kinds of output are probably due to hash randomization) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21351 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21351] refcounts not respected at process exit
Antoine Pitrou added the comment: Looking into it, it's normal for refcounts to be ignored: those objects belong to reference cycles: tstgc.__dict__ - p (or c, or c2) - p.__class__ (i.e. Parent, or Child respectivel)) - Parent.__dict__ - Parent.__del__ (or Parent.__init__, or Parent.child) - Parent.__del__.__globals__ (which is tstgc.__dict__) Since p, c, c2 belong to reference cycles, they get collected in an undefined order. Obviously, Parent.__del__ is buggy (it runs into an infinite loop when self.children != 0). Before Python 3.4, the module globals would have been set to None at shutdown, which would have broken those cycles, but caused other well-known problems. It's probably impossible to find a scheme that satisfies all constraints, so we'll see in the future if the new scheme brings more drawbacks than advantages (right now, my own evaluation is obviously that it's a step forward). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21351 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21351] refcounts not respected at process exit
Tim Peters added the comment: I think Antoine is right on all counts. The most surprising bit may be that p, c, and c2 are in reference cycles, but - surprising or not - that's always been true. The reason it worked before 3.4 is that CPython happened to break the cycles via the nasty hack of binding each module global to None at shutdown. minrk, note that gc in CPython does not (for example) run in a separate thread. That's why, when it triggers, the infinite loop in your Parent.__del__ will in fact run forever. gc runs in the same thread (the main thread) as Parent.__del__, so spinning in the Parent.__del__ loop prevents anything else (including more gc) from ever being done. Take out the infinite loop, and all three objects (p, c, c2) are collected. But the order in which they're collected isn't defined (because they're all in cyclic trash), and even changes from run to run because hash randomization changes the order in which they appear when traversing testgc.__dict__. An interesting question remaining is how you _could_ force a finalization order in this case, in a way that doesn't rely on implementation accidents. A clean way doesn't spring to my mind immediately. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21351 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
