Re: [Python-Dev] issue 9141, finalizers and gc module
-Original Message- From: python-dev-bounces+kristjan=ccpgames@python.org [mailto:python-dev-bounces+kristjan=ccpgames@python.org] On Behalf Of mar...@v.loewis.de Sent: 18. apríl 2012 07:11 To: python-dev@python.org Subject: Re: [Python-Dev] issue 9141, finalizers and gc module Invoking methods in tp_clear I find fairly harmless, in comparison. My only concern is that errors are silently ignored. However, I don't think this matters in practice, since io objects typically are not part of cycles, anyway. Why not allow it for all objects, then? It's *allowed* for all objects. Why do you think it is not? Oh, because dynamic classes with __del__ methods are deliberately not collected but put in gc.garbage. And the special case of the generator object, etc. etc. iobase.c probably documents its own needs well enough. The fact that I had to raise this question here, though, means that the source code for gcmodule.c doesn't have enough information to explain exactly the problem that it has with calling finalizers. It seems to me that it worries that __del__ methods may not run to completion because of attribute errors, and that it would have to silence such errors to not cause unexpected noise. That is the impression I get from this discussion. Correctness over memory conservation, or something like that. Btw, regarding object resurrection, I was working on a patch to get that to work better, particularly with subclasses. You may want to check out issue 8212, whence this discussion originates. K ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] issue 9141, finalizers and gc module
On Fri, Apr 20, 2012 at 3:33 PM, Kristján Valur Jónsson krist...@ccpgames.com wrote: Thanks. I wonder if these semantics might not belong in cPython too, us being consenting adults and all that J I would say it's saner, but it's just my opinion :) Cheers, fijal ** ** K ** ** *From:* python-dev-bounces+kristjan=ccpgames@python.org [mailto: python-dev-bounces+kristjan=ccpgames@python.org] *On Behalf Of *Maciej Fijalkowski *Sent:* 17. apríl 2012 21:29 *To:* Antoine Pitrou *Cc:* python-dev@python.org *Subject:* Re: [Python-Dev] issue 9141, finalizers and gc module ** ** ** ** PyPy breaks cycles randomly. I think a pretty comprehensive description of what happens is here: ** ** http://morepypy.blogspot.com/2008/02/python-finalizers-semantics-part-1.html http://morepypy.blogspot.com/2008/02/python-finalizers-semantics-part-2.html ** ** Cheers, fijal ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] issue 9141, finalizers and gc module
Thanks. I wonder if these semantics might not belong in cPython too, us being consenting adults and all that ☺ K From: python-dev-bounces+kristjan=ccpgames@python.org [mailto:python-dev-bounces+kristjan=ccpgames@python.org] On Behalf Of Maciej Fijalkowski Sent: 17. apríl 2012 21:29 To: Antoine Pitrou Cc: python-dev@python.org Subject: Re: [Python-Dev] issue 9141, finalizers and gc module PyPy breaks cycles randomly. I think a pretty comprehensive description of what happens is here: http://morepypy.blogspot.com/2008/02/python-finalizers-semantics-part-1.html http://morepypy.blogspot.com/2008/02/python-finalizers-semantics-part-2.html Cheers, fijal ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] issue 9141, finalizers and gc module
Well, we specifically decided that objects with __del__ methods that are part of a cycle cannot be run. The same reasoning was applied to generators, if they are in a certain state. What makes iobase so special that its 'close' method can be run even if it is part of a cycle? It's a hack, and I find it well-documented in iobase.c. It explains what tricks it has to go through to still invoke methods from tp_del. Invoking methods in tp_clear I find fairly harmless, in comparison. My only concern is that errors are silently ignored. However, I don't think this matters in practice, since io objects typically are not part of cycles, anyway. Why not allow it for all objects, then? It's *allowed* for all objects. Why do you think it is not? It must be opt-in, though. In the IO case, there are certain drawbacks; not being able to report errors is the most prominent one. Any other object implementation will have to evaluate whether to follow the iobase approach, or implement a regular __del__. I personally consider the resurrection in tp_del a much more serious problem, though, as this goes explicitly against the design of the release procedure. For iobase, it's ok since it can evolve along with the rest of the code base. Any third-party author would have to accept that such approach can break from one Python release to the next. I wonder why Python couldn't promise to always invoke tp_clear on GC objects; ISTM that this would remove the need for resurrection in tp_del. At the very least, I think this behaviour (this exception for iobase) merits being explicitly documented. I find all of this well-documented in iobase.c. If you think anything else needs to be said, please submit patches. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] issue 9141, finalizers and gc module
What I want to know is, why is this limitation in place? Here are two possibilities: 1) The order of calling finalizers in a cycle is undefined so it is not a solvable problem. But this would allow a single object, with only internal cycles to be collected. Currently this is not the case. It's similar to this, but not exactly: A finalizer in a cycle mail try to refer back to an object that was already cleared, and fail because of that; this may cause arbitrary failures changing from run to run. It's true that a cycle involving only a single object with __del__ could be safely collected. This special case was not implemented. 2) During collection, the interpreter is in a fragile state (linked lists of gc objects with refcount bookkeeping in place) and no unknown code can be allowed to run. This is the reason I personally think is the true reason. No, that's not the case at all. As Antoine explains in the issue, there are plenty of ways in which Python code can be run when breaking a cycle. Not only weakrefs, but also objects released as a consequence of tp_clear which weren't *in* the cycle (but hung from it). So, I ask you: What is allowed during tp_clear()? Is this a hard rule? What is the reason? We are all consenting adults. Everything is allowed - you just have to live with the consequences. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] issue 9141, finalizers and gc module
-Original Message- No, that's not the case at all. As Antoine explains in the issue, there are plenty of ways in which Python code can be run when breaking a cycle. Not only weakrefs, but also objects released as a consequence of tp_clear which weren't *in* the cycle (but hung from it). I see, that makes sense. The rule is, then that we cannot delete objects with finalalizer, that can reach other garbage, simply because doing so may find the objects in an unexpected (cleared) state and thus cause weird errors. (weakrefs are a special case, apparently dealt with separately. And the callback cannot refer back to the referent) . This reasoning belongs in the gcmodule.c, I think. So, I ask you: What is allowed during tp_clear()? Is this a hard rule? What is the reason? We are all consenting adults. Everything is allowed - you just have to live with the consequences. Well, we specifically decided that objects with __del__ methods that are part of a cycle cannot be run. The same reasoning was applied to generators, if they are in a certain state. What makes iobase so special that its 'close' method can be run even if it is part of a cycle? Why not allow it for all objects, then? At the very least, I think this behaviour (this exception for iobase) merits being explicitly documented. Kristján ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] issue 9141, finalizers and gc module
On Tue, 17 Apr 2012 17:22:57 + Kristján Valur Jónsson krist...@ccpgames.com wrote: We are all consenting adults. Everything is allowed - you just have to live with the consequences. Well, we specifically decided that objects with __del__ methods that are part of a cycle cannot be run. The same reasoning was applied to generators, if they are in a certain state. What makes iobase so special that its 'close' method can be run even if it is part of a cycle? The reason is that making file objects uncollectable when they are part of a reference cycle would be a PITA and a serious regression for many applications, I think. Why not allow it for all objects, then? I'm not the author of the original GC design. Perhaps it was deliberately conservative at the time? I think PyPy has a more tolerant solution for finalizers in reference cycles, perhaps they can explain it here. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] issue 9141, finalizers and gc module
On Tue, Apr 17, 2012 at 8:30 PM, Antoine Pitrou solip...@pitrou.net wrote: On Tue, 17 Apr 2012 17:22:57 + Kristján Valur Jónsson krist...@ccpgames.com wrote: We are all consenting adults. Everything is allowed - you just have to live with the consequences. Well, we specifically decided that objects with __del__ methods that are part of a cycle cannot be run. The same reasoning was applied to generators, if they are in a certain state. What makes iobase so special that its 'close' method can be run even if it is part of a cycle? The reason is that making file objects uncollectable when they are part of a reference cycle would be a PITA and a serious regression for many applications, I think. Why not allow it for all objects, then? I'm not the author of the original GC design. Perhaps it was deliberately conservative at the time? I think PyPy has a more tolerant solution for finalizers in reference cycles, perhaps they can explain it here. Regards Antoine. PyPy breaks cycles randomly. I think a pretty comprehensive description of what happens is here: http://morepypy.blogspot.com/2008/02/python-finalizers-semantics-part-1.html http://morepypy.blogspot.com/2008/02/python-finalizers-semantics-part-2.html Cheers, fijal ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
