Re: [Python-Dev] reference leaks, __del__, and annotations

2006-04-05 Thread Nick Coghlan
Tim Peters wrote: Note that it's very easy to do this with __del__. The trick is for your type not to have a __del__ method itself, but to point to a simple cleanup object with a __del__ method. Give that contained object references to the resources you want to close, and you're done.

Re: [Python-Dev] reference leaks, __del__, and annotations

2006-04-04 Thread Michael Hudson
Greg Ewing [EMAIL PROTECTED] writes: Michael Hudson wrote: And if we want to have a version of __del__ that can't reference 'self', we have it already: weakrefs with callbacks. Does that actually work at the moment? Last I heard, there was some issue with gc and weakref callbacks as well.

Re: [Python-Dev] reference leaks, __del__, and annotations

2006-04-04 Thread Michael Hudson
Tim Peters [EMAIL PROTECTED] writes: [Michael Hudson] ... What happened to the 'get rid of __del__ in py3k' idea? Apart from its initial mention, every now again someone asks what happened to it :-). Good enough for me :) Cheers, mwh (not subscribed to python-3000) -- You're going to

Re: [Python-Dev] reference leaks, __del__, and annotations

2006-04-04 Thread Neal Norwitz
On 4/3/06, Michael Hudson [EMAIL PROTECTED] wrote: Greg Ewing [EMAIL PROTECTED] writes: Michael Hudson wrote: And if we want to have a version of __del__ that can't reference 'self', we have it already: weakrefs with callbacks. Does that actually work at the moment? Last I heard,

Re: [Python-Dev] reference leaks, __del__, and annotations

2006-04-04 Thread Michael Hudson
Neal Norwitz [EMAIL PROTECTED] writes: On 4/3/06, Michael Hudson [EMAIL PROTECTED] wrote: Greg Ewing [EMAIL PROTECTED] writes: Michael Hudson wrote: And if we want to have a version of __del__ that can't reference 'self', we have it already: weakrefs with callbacks. Does that

Re: [Python-Dev] reference leaks, __del__, and annotations

2006-04-04 Thread Tim Peters
[Michael Hudson] And if we want to have a version of __del__ that can't reference 'self', we have it already: weakrefs with callbacks. [Greg Ewing] Does that actually work at the moment? Last I heard, there was some issue with gc and weakref callbacks as well. Has that been resolved?

Re: [Python-Dev] reference leaks, __del__, and annotations

2006-04-04 Thread Neil Schemenauer
Greg Ewing [EMAIL PROTECTED] wrote: Okay, so would it be possible for a generator that needs finalisation to set up a weakref callback, suitably rooted somewhere so that the callback is reachable, that references enough stuff to clean up after the generator, without referencing the generator

Re: [Python-Dev] reference leaks, __del__, and annotations

2006-04-04 Thread James Y Knight
On Apr 3, 2006, at 4:02 PM, Tim Peters wrote: Using which compiler? This varies across boxes. Most obviously, on a 64-bit box all these members are 8 bytes (note that ob_refcnt is Py_ssize_t in 2.5, not int anymore), but even on some 32-bit boxes the long double trick only forces 4-byte

Re: [Python-Dev] reference leaks, __del__, and annotations

2006-04-04 Thread Martin v. Löwis
James Y Knight wrote: All right, then, you could use the top bit of the ob_refcnt field. There's no way to possibly have 2**32 objects on a 32bit system anyhow. Of this kind, we have several spare bits: there are atleast two bits in the ob_type field, since the type pointer is atleast

Re: [Python-Dev] reference leaks, __del__, and annotations

2006-04-04 Thread Greg Ewing
Tim Peters wrote: A problem is that the variant semantics also seem pretty arbitrary ;-), and there's a dearth of compelling use cases to guide a decision. At the time I think I suggested that it would be reasonable if weakref callbacks were guaranteed to be called as long as they weren't trash

Re: [Python-Dev] reference leaks, __del__, and annotations

2006-04-04 Thread Greg Ewing
Neil Schemenauer wrote: I think so but it depends on what the finalizer needs to reference. That's really what I'm asking -- what *does* a generator finaliser need to reference? Or does it depend on what the generator's code does? -- Greg ___

Re: [Python-Dev] reference leaks, __del__, and annotations

2006-04-04 Thread Greg Ewing
Martin v. Löwis wrote: Of this kind, we have several spare bits: there are atleast two bits in the ob_type field, And that would require changing all code that dereferenced the ob_type field! This would be much worse -- at least Py_INCREF and Py_DECREF are macros... -- Greg

Re: [Python-Dev] reference leaks, __del__, and annotations

2006-04-04 Thread Tim Peters
[Greg Ewing] That's really what I'm asking -- what *does* a generator finaliser need to reference? Or does it depend on what the generator's code does? A generator finalizer can execute arbitrary Python code. Note that even if that was limited (which it isn't -- there are no limitations) to

Re: [Python-Dev] reference leaks, __del__, and annotations

2006-04-04 Thread Tim Peters
[various people debating how to steal a bit from an existing PyObject member] Let's stop this. It's a bike shed argument: even if we had 1000 spare bits, it wouldn't do any good. Having a bit to record whether an object has been finalized doesn't solve any problem that's been raised recently

Re: [Python-Dev] reference leaks, __del__, and annotations

2006-04-03 Thread Michael Hudson
Thomas Wouters [EMAIL PROTECTED] writes: While we're at it, I would like for the new __del__ (which would probably have to be a new method) to disallow reviving self, just because it makes it unnecessarily complicated and it's rarely needed. I'm not sure the problem is so much that anyone

Re: [Python-Dev] reference leaks, __del__, and annotations

2006-04-03 Thread Guido van Rossum
On 4/3/06, Michael Hudson [EMAIL PROTECTED] wrote: I'm not sure the problem is so much that anyone _wants_ to support resurrection in __del__, it's just that it can't be prevented. Well, Java has an answer to that (at least I believe Tim Peters told me so years ago): it allows resurrection, but

Re: [Python-Dev] reference leaks, __del__, and annotations

2006-04-03 Thread Neil Schemenauer
Guido van Rossum [EMAIL PROTECTED] wrote: This would require a bit __del__ already called on an object, but don't we have a whole word of GC-related flags? No. Neil ___ Python-Dev mailing list Python-Dev@python.org

Re: [Python-Dev] reference leaks, __del__, and annotations

2006-04-03 Thread Tim Peters
[Michael Hudson] I'm not sure the problem is so much that anyone _wants_ to support resurrection in __del__, it's just that it can't be prevented. [Guido] Well, Java has an answer to that (at least I believe Tim Peters told me so years ago): it allows resurrection, but will only call the

Re: [Python-Dev] reference leaks, __del__, and annotations

2006-04-03 Thread James Y Knight
On Apr 3, 2006, at 3:12 PM, Neil Schemenauer wrote: Guido van Rossum [EMAIL PROTECTED] wrote: This would require a bit __del__ already called on an object, but don't we have a whole word of GC-related flags? No. Actually there is. Kinda. Currently python's refcounting scheme uses 4 words

Re: [Python-Dev] reference leaks, __del__, and annotations

2006-04-03 Thread Tim Peters
[Guido] but don't we have a whole word of GC-related flags? [Neil S] No. [James Y Knight] Actually there is. Kinda. Currently python's refcounting scheme uses 4 words per object (gc_next, gc_prev, gc_refs, ob_refcnt), and has one spare word in the padding of PyGC_Head that's just sitting

Re: [Python-Dev] reference leaks, __del__, and annotations

2006-04-03 Thread Phillip J. Eby
At 08:14 AM 3/31/2006, Tim Peters wrote: [Phillip J. Eby] ... As Tim suggested, it'd be better to have the code be generator-specific, at least for now. That had actually been my original plan, to make it generator-specific, but I was afraid of breaking encapsulation in the garbage

Re: [Python-Dev] reference leaks, __del__, and annotations

2006-04-03 Thread Greg Ewing
Michael Hudson wrote: And if we want to have a version of __del__ that can't reference 'self', we have it already: weakrefs with callbacks. Does that actually work at the moment? Last I heard, there was some issue with gc and weakref callbacks as well. Has that been resolved? -- Greg Ewing,

Re: [Python-Dev] reference leaks, __del__, and annotations

2006-04-03 Thread Tim Peters
[Phillip J. Eby] I'm trying to figure out how to implement this now, and running into a bit of a snag. It's easy enough for gcmodule.c to check if an object is a generator, but I'm not sure how safe the dynamic check actually is, since it depends on the generator's state. In principle,

Re: [Python-Dev] reference leaks, __del__, and annotations

2006-04-03 Thread Tim Peters
[Michael Hudson] ... What happened to the 'get rid of __del__ in py3k' idea? Apart from its initial mention, every now again someone asks what happened to it :-). ___ Python-Dev mailing list Python-Dev@python.org

Re: [Python-Dev] reference leaks, __del__, and annotations

2006-04-03 Thread Greg Ewing
Tim Peters wrote: We already endure lots of pain to ensure that a weakref callback that gets executed (not all do) can't see anything that looks like trash. Okay, so would it be possible for a generator that needs finalisation to set up a weakref callback, suitably rooted somewhere so that the

Re: [Python-Dev] reference leaks, __del__, and annotations

2006-04-01 Thread Thomas Wouters
On 4/1/06, Nick Coghlan [EMAIL PROTECTED] wrote: Greg Ewing wrote: I find it rather worrying that there could be a few rare cases in which my generators cause memory leaks, through no fault of my own and without my being able to do anything about it. The GC changes PJE is looking at are to make

Re: [Python-Dev] reference leaks, __del__, and annotations

2006-04-01 Thread Thomas Wouters
On 4/1/06, Greg Ewing [EMAIL PROTECTED] wrote: I'm becoming more and more convinced that wedesperately need something better than __del__methods to do finalisation. A garbage collectorthat can't be relied upon to collect garbageis simply not acceptable. Sure. I don't believe it's too hard, it just

Re: [Python-Dev] reference leaks, __del__, and annotations

2006-04-01 Thread Thomas Wouters
On 4/1/06, Jim Jewett [EMAIL PROTECTED] wrote: Nick Coghlan A simple Boolean attribute (e.g. __finalized__) should be enough. ... If it's both present and true, the GC can ignore the finaliser on that instanceThat doesn't really take care of resource release, which needs to be called, and called

Re: [Python-Dev] reference leaks, __del__, and annotations

2006-04-01 Thread Thomas Wouters
On 4/1/06, Thomas Wouters [EMAIL PROTECTED] wrote: On 4/1/06, Jim Jewett [EMAIL PROTECTED] wrote: Nick Coghlan A simple Boolean attribute (e.g. __finalized__) should be enough. ... If it's both present and true, the GC can ignore the finaliser on that instanceThat doesn't really take care of

Re: [Python-Dev] reference leaks, __del__, and annotations

2006-03-31 Thread Duncan Booth
Jim Jewett [EMAIL PROTECTED] wrote in news:[EMAIL PROTECTED]: As a strawman proposal: deletes = [(obj.__del__.cycle, obj) for obj in cycle if hasattr(obj, __del__) and hasattr(obj.__del__, cycle)] deletes.sort() for (cycle, obj) in deletes:

Re: [Python-Dev] reference leaks, __del__, and annotations

2006-03-31 Thread Nick Coghlan
Duncan Booth wrote: Surely if you have a cycle what you want to do is to pick just *one* of the objects in the cycle and break the link which makes it participate in the cycle. That should be sufficient to cause the rest of the cycle to collapse with __del__ methods being called from the

Re: [Python-Dev] reference leaks, __del__, and annotations

2006-03-31 Thread Thomas Wouters
On 3/31/06, Jim Jewett [EMAIL PROTECTED] wrote: The checkins list has been struggling with generator reference leaks;the latest conclusion was that some are unavoidable because of __del__cycles. That was Tim's conclusion, but I wasn't quite done thinking about it ;) That sort of defeats the

Re: [Python-Dev] reference leaks, __del__, and annotations

2006-03-31 Thread Tim Peters
[Phillip J. Eby] ... As Tim suggested, it'd be better to have the code be generator-specific, at least for now. That had actually been my original plan, to make it generator-specific, but I was afraid of breaking encapsulation in the garbage collector by having it know about generators. It

Re: [Python-Dev] reference leaks, __del__, and annotations

2006-03-31 Thread Phillip J. Eby
At 12:14 PM 3/31/2006 -0500, Tim Peters wrote: [Phillip J. Eby] ... As Tim suggested, it'd be better to have the code be generator-specific, at least for now. That had actually been my original plan, to make it generator-specific, but I was afraid of breaking encapsulation in the

[Python-Dev] reference leaks, __del__, and annotations

2006-03-31 Thread Jim Jewett
Duncan Booth wrote: Surely if you have a cycle what you want to do is to pick just *one* of the objects in the cycle and break the link which makes it participate in the cycle. No, I really meant to do them all. I was trying to avoid creating an attractive nuisance. In

Re: [Python-Dev] reference leaks, __del__, and annotations

2006-03-31 Thread Greg Ewing
Nick Coghlan wrote: Generators are even more special, in that they only require finalisation in the first place if they're stopped on a yield statement inside a try-finally block. I find it rather worrying that there could be a few rare cases in which my generators cause memory leaks,

Re: [Python-Dev] reference leaks, __del__, and annotations

2006-03-31 Thread Nick Coghlan
Greg Ewing wrote: Nick Coghlan wrote: Generators are even more special, in that they only require finalisation in the first place if they're stopped on a yield statement inside a try-finally block. I find it rather worrying that there could be a few rare cases in which my generators

Re: [Python-Dev] reference leaks, __del__, and annotations

2006-03-31 Thread Greg Ewing
Nick Coghlan wrote: from contextlib import closing with closing(itr): # Use the iterator in here as you wish # secure in the knowledge it will be # cleaned up promptly when you are done # whether it is a file, a generator or # something with a database connection for

[Python-Dev] reference leaks, __del__, and annotations

2006-03-30 Thread Jim Jewett
The checkins list has been struggling with generator reference leaks; the latest conclusion was that some are unavoidable because of __del__ cycles. That sort of defeats the purpose of resource managers. (Yes, it can be worked around on a case-by-case basis.) As I see it, part of the problem is

Re: [Python-Dev] reference leaks, __del__, and annotations

2006-03-30 Thread Jim Jewett
(Apologies for the thinko -- corrected because it was in the example code.) The checkins list has been struggling with generator reference leaks; the latest conclusion was that some are unavoidable because of __del__ cycles. That sort of defeats the purpose of resource managers. (Yes, it can be

Re: [Python-Dev] reference leaks, __del__, and annotations

2006-03-30 Thread Greg Ewing
Jim Jewett wrote: The checkins list has been struggling with generator reference leaks; the latest conclusion was that some are unavoidable because of __del__ cycles. That sort of defeats the purpose of resource managers. Seems to me we need a whole new approach to finalization that's