[issue9141] Allow objects to decide if they can be collected by GC

[Kristján Valur Jónsson]

Kristján Valur Jónsson  added the comment:

Hi there!  By the time pep 442 was introduced, I wasn't very active in python 
core stuff anymore, and still am not.

The intent of this patch, which is explained (IMHO) quite clearly in the first 
few comments was to

- Formalize a way for custom objects to tell GC "No, please don't delete my 
references _at this time_ because if you do, I will have to run non-trivial 
code that may wreak havoc".  This is different from just having a __del__ 
method.  Sometimes deleting is okay.  Sometimes not. 
- Make this way available to all objects, not just Generator objects.  We 
already identified a separate such instance in stackless python and it seemed 
prudent to "give back" the generalization that we made there for the benefit of 
python at large.
- Not introduce new slots for this purpose.

Now, with pep 442, I have no idea how Generators can postpone being garbage 
collection since I'm honestly not familiar with how things work now.

I have no particular skin in this game anymore, I'm no longer actively working 
on Stackless or Python integrations and I stopped trying to push stuff thought 
the bugtracker to preserve my sanity.

So, lets just close this until the day in the future when needs arise once more 
:)

--
resolution:  -> out of date
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9141] Allow objects to decide if they can be collected by GC

[Antoine Pitrou]

Antoine Pitrou  added the comment:

It's hard to say as I never fully understood the underlying intent.

Perhaps Kristján or some other developer who better understood than me can 
chime in.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9141] Allow objects to decide if they can be collected by GC

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

Is this issue still actual after PEP 442?

--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9141] Allow objects to decide if they can be collected by GC

[Martin Morrison]

Changes by Martin Morrison m...@ensoft.co.uk:


--
nosy: +isoschiz, pconnell

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9141
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9141] Allow objects to decide if they can be collected by GC

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

 Surely, if these objects define non-trivial 'close' members, they must 
 not be called during garbage collection.

Define non-trivial. There are various tests for it in test_io.

Not ending up in gc.garbage is *by design*. Making file objects uncollectable 
as soon as they appear in a reference cycle would be a serious regression. 
That's why the cleanup is done in tp_dealloc instead of having a __del__.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9141
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9141] Allow objects to decide if they can be collected by GC

[Kristján Valur Jónsson]

Kristján Valur Jónsson krist...@ccpgames.com added the comment:

I _think_ the only python related things you can do from tp_clear() is 
Py_DECREF(), this is what I mean by trivial.  This is the reason, for example, 
that special care was done with generators.
An IO object could of course do non-python operations such as closing files and 
freeing buffers.

If file.close() can be an arbitrary python method, then it can no more be 
called from gc, than an object's __del__ method.  This would not be a 
regression, this would be a fact of life.

The point of _this_ defect (issue 9141) is to allow objects to be smarter about 
this, being able to tell gc if their finalizers are trivial or not.


I will start a discussion on python-dev to see if anyone knows exactly why 
these limitations are in place, and what they are.  They are not documented in 
the source code.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9141
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9141] Allow objects to decide if they can be collected by GC

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

 I _think_ the only python related things you can do from tp_clear() is
 Py_DECREF(), this is what I mean by trivial.

Well, Py_DECREF is not trivial at all, since it can invoke arbitrary
Python code (through e.g. weakref callbacks, or by releasing the GIL).
Therefore, I would say any code is allowed from tp_clear :-)

 If file.close() can be an arbitrary python method, then it can no more
 be called from gc, than an object's __del__ method.  This would not be
 a regression, this would be a fact of life.

I don't believe it. I don't see what's magical about being called by the
gc. Again, a Py_DECREF in tp_dealloc can invoke arbitrary Python code.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9141
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9141] Allow objects to decide if they can be collected by GC

[Kristján Valur Jónsson]

Kristján Valur Jónsson krist...@ccpgames.com added the comment:

I don't believe it. I don't see what's magical about being called by the
gc. Again, a Py_DECREF in tp_dealloc can invoke arbitrary Python code.

Look again.  gcmodule specifically takes any objects reachable from ob_clear 
and sees if any of them have side effects when Py_DECREF'd.  If any object has 
a finalizer, the entire cycle is put in gc.garbage.

gcmodule is trickier than you might think.  I've spent quite a time with it.

Anyway, I've put the issue to python-dev, let's see if they have some 
autorative insight on the matter.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9141
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9141] Allow objects to decide if they can be collected by GC

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

 Look again.  gcmodule specifically takes any objects reachable from
 ob_clear and sees if any of them have side effects when Py_DECREF'd.

has_finalizer() in gcmodule.c doesn't check for weakref callbacks, so a
weakref callback can still be invoked from tp_dealloc.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9141
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9141] Allow objects to decide if they can be collected by GC

[Daniel Stutzbach]

Daniel Stutzbach stutzb...@google.com added the comment:

On Tue, Apr 17, 2012 at 4:08 AM, Antoine Pitrou rep...@bugs.python.org wrote:
 has_finalizer() in gcmodule.c doesn't check for weakref callbacks, so a
 weakref callback can still be invoked from tp_dealloc.

Unless I'm mistaken, weakrefs will be handled and removed by
handle_weakrefs() before delete_garbage() is called.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9141
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9141] Allow objects to decide if they can be collected by GC

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

 On Tue, Apr 17, 2012 at 4:08 AM, Antoine Pitrou rep...@bugs.python.org 
 wrote:
  has_finalizer() in gcmodule.c doesn't check for weakref callbacks, so a
  weakref callback can still be invoked from tp_dealloc.
 
 Unless I'm mistaken, weakrefs will be handled and removed by
 handle_weakrefs() before delete_garbage() is called.

Perhaps, but what does that change?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9141
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9141] Allow objects to decide if they can be collected by GC

[Kristján Valur Jónsson]

Kristján Valur Jónsson krist...@ccpgames.com added the comment:

I've stumbled upon further cases of this problem, and a possible serious bug in 
python 3.
the _PyIOBase_finalize() will invoke a close method on the object if the 
object isn't closed.  This is the same as the object having a __del__ method.  
Yet, these objects are not treated as having finalizers by the garbage 
collector, and indeed, _PyIOBase_finalize() is called by the tp_clear() slot of 
several derived types. Surely, if these objects define non-trivial 'close' 
members, they must not be called during garbage collection.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9141
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9141] Allow objects to decide if they can be collected by GC

[Martin v . Löwis]

Martin v. Löwis mar...@v.loewis.de added the comment:

I'm still unclear about the rationale for this change. krisvale says in the 
patch and in msg109099 that this is to determine whether an object can be 
collected at this time. Is the intended usage that the result value may 
change over the lifetime of the object?

If so, I'm -1 on the patch. If an object cannot be collected at this time, it 
means that it is added to gc.garbage, which in turn means that it will never be 
collected (unless somebody explicitly clears gc.garbage).

Supporting the case of objects that can be collected despite living in a cycle 
is fine to me, but those objects must not change their mind.

Supporting the case of objects that are not collectable now, but may be 
collectable later, may have its use case (which one?), but this is not 
addressed by the patch (AFAICT). To support it, processing of the entire cycle 
must be postponed (to the next collection? to the next generation?).

I'm -0 on recycling the is_gc slot. Having a GC header and having a non-trivial 
tp_del are two unrelated issues. If this is done, I think it would be best to 
rename the slot to tp_gc_flags or something. There is also the slight risk of 
some type in the wild returning non-1 currently, which then would get 
misinterpreted.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9141
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9141] Allow objects to decide if they can be collected by GC

[Kristján Valur Jónsson]

Kristján Valur Jónsson krist...@ccpgames.com added the comment:

Jim: The edge case of collecting an object that is alone in a cycle is 
something that isn't handled.  I'm also not sure that it is worth doing or even 
safe or possible.  but that is beside the point and not the topic of this patch.

Martin: This patch is formalizing a known fact about cpython objects, albeit an 
uncommon one:  Some objects with finalizers can be safely collected if they are 
in a certain state.  The canonical example is the PyGeneratorObject.  
gccollect.c has special code for this type and there is a special evil API 
exposed to deal with the case.

_This patch is not changing any behaviour._  Generator objects that are in a 
special state of existance cannot be collected.  Rather, they are (and always 
have been) put in gc.garbage along with the rest of their cycle chain.  In 
other cases, the finalizer causes no side effects and simply clearing them is 
ok.  I couldn't tell you what those generator execution states are, but the 
fact is that they exist.

A similar problem exists in Stackless python with tasklets.  We solved it in 
this generic way there rather than add more exceptional code to gcmodule.c and 
this patch is the result of that work.  In Stackless, the inverse is true:  An 
object without an explicit finalizer can still be unsafe to collect, because 
the tp_dealloc can do evil things, but doesn't always, depending on object 
state.

So, objects who change their mind about whether they can be collected or not 
are a fact of life in python.  Yes, even cPython.  This patch aims to formalize 
that fact and give it an interface, rather than to have special code for 
generator objects in gcmodule.c and an inscrutable API exposed 
(PyGen_NeedsFinalizing())

About reusing the slot:  Slots are a precious commodity in python.  This 
particular slot is undocumented and used only for one known thing:  To 
distinguish PyTypeObjects from PyHeapTypeObjects.  In fact, creating a slot and 
not using special case code (like they did for PyGeneratorObjects) was forward 
thinking, and I'm trying to build on that.  Renaming the slot is a fine idea.

A brief search on google code (and google at large) showed no one using this 
slot.  It is one of those undocumented strange slots that one just initializes 
to 0.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9141
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9141] Allow objects to decide if they can be collected by GC

[Kristján Valur Jónsson]

Kristján Valur Jónsson krist...@ccpgames.com added the comment:

Btw. tangentially related to this discussion, issue 10576 aims to make the 
situation with uncollectable objects a little more bearable.  An application 
can listen for garbage collection, visit gc.garbage and deal with its 
problematic types in its own way.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9141
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9141] Allow objects to decide if they can be collected by GC

[Kristján Valur Jónsson]

Kristján Valur Jónsson krist...@ccpgames.com added the comment:

New patch with explicit mutually exclusive flags and better comments.

--
Added file: http://bugs.python.org/file25142/ob_is_gc.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9141
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9141] Allow objects to decide if they can be collected by GC

[Kristján Valur Jónsson]

Kristján Valur Jónsson krist...@ccpgames.com added the comment:

Thanks for your comments Jim.
They made me realize that the patch was incomplete.  What is needed are two 
mutually exclusive flags that override the presence of the tp_del slot.  This 
addresses your 1) point and is the case for Generators.

I don't think 2 is important.  Does the context of the call matter?  It is 
merely a question of whether a finalizer will or will not do something trivial.

Finalizers implemented in python can never run from garbage collection.  the 
potential to do harm is simply too great.  Which is why these extensions are 
purely the domain of specialized C extensions.  Hence I don't think adding 
another slot, or allowing self-declared idempotent python finalizers to run 
during cleanup is a good idea.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9141
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9141] Allow objects to decide if they can be collected by GC

[Jim Jewett]

Jim Jewett jimjjew...@gmail.com added the comment:

 I don't think 2 is important.  Does the context of the call matter?
 It is merely a question of whether a finalizer will or will not do
 something trivial.

It would affect how I would write such functions.  If I knew that it wouldn't 
be called until the object was already garbage, then I be inclined to move 
parts of tp_del there, and break the cycle.

 Finalizers implemented in python can never run from garbage 
 collection.  the potential to do harm is simply too great.

__del__ methods do run, even if an object was collected by the cycle detector.  
And they can't do any harm that couldn't also be done by a C finalizer.

The only change from today's situation with __del__ is that once an object is 
known to be cyclic garbage, it would get a chance to break the cycles itself 
(or, admittedly, to rescue itself) before the cycle-breaker began making 
arbitrary decisions or gave up and stuffed it in the uncollectable list.

Even in your case, instead of setting a timer to clean out garbage, you could 
have the garbage itself notify your cleaner that it needed attention.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9141
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9141] Allow objects to decide if they can be collected by GC

[Daniel Stutzbach]

Daniel Stutzbach stutzb...@google.com added the comment:

On Fri, Apr 6, 2012 at 12:51 PM, Jim Jewett rep...@bugs.python.org wrote:

 __del__ methods do run, even if an object was collected by the cycle
 detector.  And they can't do any harm that couldn't also be done by a C
 finalizer.


No, if an object with a __del__ method is part of a cycle, it is not
collected.  The objects get appended to gc.garbage instead.

See:  http://docs.python.org/py3k/library/gc.html#gc.garbage

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9141
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9141] Allow objects to decide if they can be collected by GC

[Kristján Valur Jónsson]

Kristján Valur Jónsson krist...@ccpgames.com added the comment:

Right Daniel, but currently an exception is made for Generator objects.  The 
generator can tell that the finalizer won't do anything and then allow 
collection regardless.  It knows the finalizer and that in its state it will at 
most do some Py_DECREF operations.

This patch aims to generalize this exception mechanism, and also its opposite, 
so that we don´t need to rely on the presence of the finalizer slot to decide.  
We can then remove a silly function from the public (but undocumented) API.

Of course, you can only allow collectino of an object with a finalizer if you 
know for sure that it will not do anything but Py_DECREF.  This is possible for 
generators because they are not an inheritable class.  But we could never do 
this in general for a python class.  Even running .py code from the GC 
collector cycle would crash everything instantly.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9141
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9141] Allow objects to decide if they can be collected by GC

[Daniel Stutzbach]

Daniel Stutzbach stutzb...@google.com added the comment:

Are __del__ and tp_del getting conflated in this conversation?  I don't see a 
__del__ method on generator objects:

filfre:$ python3.1 
Python 3.1.2 (r312:79147, Dec  9 2011, 20:47:34) 
[GCC 4.4.3] on linux2
 (i for i in range(5)).__del__
Traceback (most recent call last):
  File stdin, line 1, in module
AttributeError: 'generator' object has no attribute '__del__'

but I may be missing something.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9141
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9141] Allow objects to decide if they can be collected by GC

[Kristján Valur Jónsson]

Kristján Valur Jónsson krist...@ccpgames.com added the comment:

typeobject.c:
TPSLOT(__del__, tp_del, slot_tp_del, NULL, ),

I'm not super-familiar with how typeobjects and slots work, but I imagine that 
if a type is defined with a __del__ member, then the tp_del slot is 
automatically filled out.  The converse need not be true.

At any rate, the non-zero-ness of tp_del is what gcmodule.c uses to find out if 
an object has a finaliser.  There is a code rudiment present in gcmodule.c, 
that hints at an earlier time when '__del__' was looked up but that string is 
not actually used.  That can be removed to, eiter as part of this patch or 
separately since it should be quite uncontroversial.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9141
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9141] Allow objects to decide if they can be collected by GC

[Jim Jewett]

Jim Jewett jimjjew...@gmail.com added the comment:

On Fri, Apr 6, 2012 at 4:03 PM,
 Daniel Stutzbach stutzb...@google.com added the comment:

 __del__ methods do run, even if an object was collected by the cycle
 detector.  And they can't do any harm that couldn't also be done by a C
 finalizer.

 No, if an object with a __del__ method is part of a cycle, it is not
 collected.  The objects get appended to gc.garbage instead.

 See:  http://docs.python.org/py3k/library/gc.html#gc.garbage

They can still be collected if there is only one object with a __del__
method in the cycle.

(Whether the code actually does that, it appears not to at the moment,
and I won't swear by by own memory of 2.3 era code.)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9141
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9141] Allow objects to decide if they can be collected by GC

[Kristján Valur Jónsson]

Kristján Valur Jónsson krist...@ccpgames.com added the comment:

Here is a completely new patch.  This approach uses the already existing 
tp_is_gc enquiry slot to signal garbage collection.
The patch modifies the generator object to use this new mechanism.
The patch keeps the old PyGen_NeedsFinalizing() API, but this can now go away, 
unless people think it might be used in extension modules 

(why do we always expose all those internal apis from the dll? I wonder.)

--
Added file: http://bugs.python.org/file25131/ob_is_gc.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9141
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9141] Allow objects to decide if they can be collected by GC

[Andrew Svetlov]

Changes by Andrew Svetlov andrew.svet...@gmail.com:


--
nosy: +asvetlov

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9141
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9141] Allow objects to decide if they can be collected by GC

[Jim Jewett]

Jim Jewett jimjjew...@gmail.com added the comment:

I think the second patch is a fairly straightforward enhancement to the 
existing situation.  It still feels a bit like an awkward half-measure, though.

(1)  If a class does have a finalizer, instances should still be able to say 
Go ahead and collect me, I don't happen to be in the problematic state.  (In 
other words, there should be a return value that means not to bother checking 
for the existence of a tp_del / __del__ method.)

(2)  It should be explicit whether or not this is:
  (2a) just an inquiry (If you were in a cycle, could I collect you?),
  (2b) a statement plus inquiry (You are in a cycle.  Can I collect you?), or
  (2c) a request (You are in a cycle, and I would like to collect you.  Clean 
up if you can, then tell me whether you are still uncollectable.)

(3)  It may be worth adding an additional slot for safe idempotent finalizers.  
(Earlier suggestions called this __close__, but I suppose __finalize__ might be 
more general.)  Once a garbage cycle is detected, go ahead and call that slot's 
method before giving up and sticking the whole thing in garbage.  If python 
classes could also fill this slot, it would look a lot like (2c).

--
nosy: +Jim.Jewett

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9141
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9141] Allow objects to decide if they can be collected by GC

[Kristján Valur Jónsson]

Kristján Valur Jónsson krist...@ccpgames.com added the comment:

I just noticed that PyTypeObjects have a gc slot already:
inquiry tp_is_gc; /* For PyObject_IS_GC */

Now, this is used in only one place in 2.7, for type objects:
return type-tp_flags  Py_TPFLAGS_HEAPTYPE;

This is thus used to dynamically query if an object was allocated with a GC 
header or not, since static types cannot have this.

Now, it would be perfectly possible to change the semantics of this function to 
return a bit flag, with bit 0 being the has head and bit 1 meaning is 
collectable

This might simplify some stuff.  Any thoughts?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9141
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9141] Allow objects to decide if they can be collected by GC

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

 This might simplify some stuff.  Any thoughts?

I think you should float it on python-dev.

--
versions: +Python 3.3 -Python 3.2

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9141
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9141] Allow objects to decide if they can be collected by GC

[David Stanek]

Changes by David Stanek dsta...@dstanek.com:


--
nosy: +dstanek

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9141
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9141] Allow objects to decide if they can be collected by GC

[Kristján Valur Jónsson]

Kristján Valur Jónsson krist...@ccpgames.com added the comment:

I agree that this is not the _optimal_ solution, and that when you are 
collecting stuff, say a list of many items, it will cause X indirect calls to 
all of the things in the list.

My counterargument is, however, that all those items will, if collected, be 
handed off to the final Py_DECREF with all the baggage that that entails, 
ultimately resulting in PyObject_Free or PyMem_Free.  An extra indirect call, 
(only for unreachable objects, btw, which is only a small objects of all 
objects visited during a GC pass) should not play a significatn part in the 
process.

An alternative to this extra tp_traverse() pass, would be to flag objects that 
report that they are or are not collectable, with a special  value in gc_refs, 
probably a bitmask.  But this would be a much more intrusive change in an 
algorithm that is far from simple, and so need very rigorous review and testing.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9141
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9141] Allow objects to decide if they can be collected by GC

[Daniel Stutzbach]

Changes by Daniel Stutzbach dan...@stutzbachenterprises.com:


Removed file: http://bugs.python.org/file17925/unnamed

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9141
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9141] Allow objects to decide if they can be collected by GC

[Daniel Stutzbach]

Daniel Stutzbach dan...@stutzbachenterprises.com added the comment:

Yes, I see your point.  Even if more expensive than strictly necessary, the 
cost should be swamped by other existing costs.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9141
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9141] Allow objects to decide if they can be collected by GC

[Kristján Valur Jónsson]

Kristján Valur Jónsson krist...@ccpgames.com added the comment:

Hi Antoine.
I am not adding another traversal pass.  I am modifying the function that gets 
called for every object about to be collected, to call the object's traversal 
function.  This is hardly an extra pass and is done only for objects that are 
about to be collected (i.e. only in the rare case).

The particular case is this:  I have objects (tasklets) that depending on their 
runtime state, need to run a finalizer or not.  If not, they can be collected.  
If they do, then they end up on gc.garbage.

This is exactly the same case as with generators.  But instead of adding yet 
another special case into stackless python, I realized that it may be up to 
objects' runtime state whether they need to run a finalizer or not.  So, I 
wrote a more generic way and decided to contribute it to C Python, so that a 
future developer, having a C extension that only sometimes needs to run a 
finalizer, could have a way to cleanly deal with garbage collection.  This is 
twice now that this problem has been seen in python.  The first time, a special 
case was coded into gcmodule.  This second time, I'm offering a more generic 
solution.

Now, I have a number of useful improvements and additions to C python sitting 
in our repository, the result of over 7 years work with Python at CCP games.  I 
have another one sitting and waiting that adds useful functionality to 
gcmodule.  Useful in my opinion anyway, and worth at least being kept for 
posterity as a patch in the tracker.  But since these patches of mine seem to 
be met with rather limited enthusiasm,  I wonder if I should be bothering.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9141
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9141] Allow objects to decide if they can be collected by GC

[Kristján Valur Jónsson]

Kristján Valur Jónsson krist...@ccpgames.com added the comment:

Hi Daniel.
Your message was classified as spam, I have no idea why, but this is why I only 
noticed it now.

Your suggestion is interesting, I hadn't thought of that.  Yes, it is possible 
to use the track/untrack functions (I think), but that would mean that you 
would have to monitor your object for every state change and reliably detect 
the transition from one state to another.  Being able to query the current 
state and let gc know:  No, I cannot be collected as I am now is a much more 
robust solution from the programmers perspective.

A further difference is this:  If an object isn't tracked, it won't be 
collected if it is part of a cycle, but it will not be put in gc.garbage 
either.  In effect, it will just remain in memory, unreachable, with no chance 
of it ever being released.

In contrast, a generator (or in my case, a tasklet) that needs to have a 
finalizer run when it goes away, will end up in gc.garbage, which a dilligent 
application will visit once in a while in order to clean it out.  I'm not sure 
how you would clear out a generator like that, but in Stackless python, you 
could do something like this once in a while:
for g in gc.garbage:
  if isinstance(g, stackless.tasklet):
g.kill()
del gc.garbage[:]

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9141
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9141] Allow objects to decide if they can be collected by GC

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

 The particular case is this:  I have objects (tasklets) that depending 
 on their runtime state, need to run a finalizer or not.  If not, they
 can be collected.  If they do, then they end up on gc.garbage.

But why don't you try using tp_dealloc as I suggested instead? It will allow 
you to run a finalizer in all cases, without anything ending up in gc.garbage. 
It's what we do in the new io lib.

Also, I'm still worried that the mechanism you're proposing is a bit quirky. I 
hope other developers can give opinions or even suggestions.

 But since these patches of mine seem to be met with rather limited
 enthusiasm,  I wonder if I should be bothering.

Well, the GC is a rather delicate affair and moreover it doesn't have a 
dedicated maintainer. That doesn't mean your patches are not interesting, but I 
think everyone tries to be very careful with that area of the code.

--
nosy: +amaury.forgeotdarc, loewis

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9141
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9141] Allow objects to decide if they can be collected by GC

[Daniel Stutzbach]

Daniel Stutzbach dan...@stutzbachenterprises.com added the comment:

I posted some of this earlier in the week, but for some reason the bug tracker 
marked my message as spam.  Hopefully this message will survive. ;-)

Can you compare and contrast your approach with calling PyObject_GC_UnTrack and 
PyObject_GC_Track to mark the object as uncollectable/collectable?

I can't speak for anyone else, but speaking as someone who has ungracefully 
poked into the innards of garbage collector in extension modules, I'm 
interested in seeing your patches.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9141
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9141] Allow objects to decide if they can be collected by GC

[Daniel Stutzbach]

Changes by Daniel Stutzbach dan...@stutzbachenterprises.com:


--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9141
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9141] Allow objects to decide if they can be collected by GC

[Raymond Hettinger]

Raymond Hettinger rhettin...@users.sourceforge.net added the comment:

 I'm merely providing a way for objects to signal
 that they _have_ (or don't have) non-trivial finalizers.

+1 This is useful and non-hackish way to communicate
with GC and solves a problem that doesn't otherwise
have a clean solution.

--
nosy: +rhettinger

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9141
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9141] Allow objects to decide if they can be collected by GC

[Daniel Stutzbach]

Daniel Stutzbach dan...@stutzbachenterprises.com added the comment:

2010/7/9 Kristján Valur Jónsson rep...@bugs.python.org

 Your message was classified as spam, I have no idea why, but this is why I
 only noticed it now.


Yes, I just noticed that tonight as well.  I filed a bug on the meta-tracker
in the hopes that someone can dig into the underlying cause.

 Your suggestion is interesting, I hadn't thought of that.  Yes, it is
 possible to use the track/untrack functions (I think), but that would mean
 that you would have to monitor your object for every state change and
 reliably detect the transition from one state to another.  Being able to
 query the current state and let gc know:  No, I cannot be collected as I am
 now is a much more robust solution from the programmers perspective.

 A further difference is this:  If an object isn't tracked, it won't be
 collected if it is part of a cycle, but it will not be put in gc.garbage
 either.  In effect, it will just remain in memory, unreachable, with no
 chance of it ever being released.


Yes, I see.  I have used the track/untrack approach, but it was in a very
different situation.  I have a long-running C function which keeps alive a
large number of objects.  At the start of the function, I untrack them all
as a performance optimization, so the garbage collector does not have to
spend time traversing them.  Before the function returns, I track them
again.

I see now why that wouldn't work for your use-case.  Thank you.  I like the
idea of your patch to give opportunities to tell the GC that they are
uncollectable on the fly.

I'm concerned about the performance impact of making tp_traverse do
double-duty, though.  Calling tp_traverse for every object in a cycle will
have the effect of making an extra pass on every reference from every object
participating in the cycle.  For example, consider a large list that's part
of a cycle.  If we call the list's tp_traverse to establish if
it's collectible, list's tp_traverse will call visit() on every item in the
list.  Even though you've made visit() a do-nothing function, that's still a
function call per reference.  It seems a shame to do all of that work
unnecessarily.

--
Added file: http://bugs.python.org/file17925/unnamed

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9141
___div class=gmail_quote2010/7/9 Kristján Valur Jónsson span 
dir=ltrlt;a href=mailto:rep...@bugs.python.org; 
target=_blankrep...@bugs.python.org/agt;/spanbrblockquote 
class=gmail_quote style=margin:0 0 0 .8ex;border-left:1px #ccc 
solid;padding-left:1ex


divYour message was classified as spam, I have no idea why, but this is why I 
only noticed it now./div/blockquotedivbr/divdivYes, I just noticed 
that tonight as well.  I filed a bug on the meta-tracker in the hopes that 
someone can dig into the underlying cause./div


div /divblockquote class=gmail_quote style=margin:0 0 0 
.8ex;border-left:1px #ccc solid;padding-left:1exYour suggestion is 
interesting, I hadn#39;t thought of that.  Yes, it is possible to use the 
track/untrack functions (I think), but that would mean that you would have to 
monitor your object for every state change and reliably detect the transition 
from one state to another.  Being able to query the current state and let gc 
know:  quot;No, I cannot be collected as I am nowquot; is a much more robust 
solution from the programmers perspective.br



br
A further difference is this:  If an object isn#39;t tracked, it won#39;t be 
collected if it is part of a cycle, but it will not be put in gc.garbage 
either.  In effect, it will just remain in memory, unreachable, with no chance 
of it ever being released.br


/blockquotedivbr/divdivYes, I see.  I have used the track/untrack 
approach, but it was in a very different situation.  I have a long-running C 
function which keeps alive a large number of objects.  At the start of the 
function, I untrack them all as a performance optimization, so the garbage 
collector does not have to spend time traversing them.  Before the function 
returns, I track them again./div


divbr/divdivI see now why that wouldn#39;t work for your use-case. 
 Thank you.  I like the idea of your patch to give opportunities to tell the 
GC that they are uncollectable on the fly.  /divdivbr/divdiv
I#39;m concerned about the performance impact of making tp_traverse do 
double-duty, though.  Calling tp_traverse for every object in a cycle will 
have the effect of making an extra pass on every reference from every object 
participating in the cycle.  For example, consider a large list that#39;s 
part of a cycle.  If we call the list#39;s tp_traverse to establish if 
it#39;s collectible, list#39;s tp_traverse will call visit() on every item 
in the list.  Even though you#39;ve made visit() a do-nothing function, 
that#39;s still a function call per reference.  It seems 

[issue9141] Allow objects to decide if they can be collected by GC

[Kristján Valur Jónsson]

Kristján Valur Jónsson krist...@ccpgames.com added the comment:

I don't understand the tp_del / tp_dealloc argument here.  This is really not 
the concern of gc.  Gc only goes and finds unreachable objects, and if they are 
deemend collectable, makes them go away by clearing their references.
GC only cares that during the final release of any objects in a reference 
cycle, no non-trivial code may run (for various technical reasons).  So, 
objects deemed sensitive in this way, are left alone (and put in gc.garbage 
instead).
GC has no other knowledge of how objects behave, if or when any finalizers are 
called.  The only thing gc does is calling Py_DECREF (through the tp_clear 
slot).  Everything else is up to the objects.

The intent of this patch is for objects themselves to be able to provide better 
information to GC as to whether they are too sensitive to be cleared by GC, 
rather than GC relying solely on the presence of a __del__ method, or using the 
builtin special case knowledbe for PyGen objects.

Whether finalizers are called from tp_del or tp_dealloc is beyond the scope of 
the gc module or, indeed, this patch.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9141
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9141] Allow objects to decide if they can be collected by GC

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

 GC only cares that during the final release of any objects in a
 reference cycle, no non-trivial code may run (for various technical
 reasons).  So, objects deemed sensitive in this way, are left alone
 (and put in gc.garbage instead).

Which is really the problem, and why you should prefer tp_dealloc over
tp_del.
(unless you want gc.garbage to fill up too easily)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9141
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9141] Allow objects to decide if they can be collected by GC

[Kristján Valur Jónsson]

Kristján Valur Jónsson krist...@ccpgames.com added the comment:

I'm confused, Antoine. _I_ am not preferring anything.  I am not modifying the 
way finalizers are called.  I'm merely providing a way for objects to signal 
that they _have_ (or don't have) non-trivial finalizers.

A finalizer _may_ be tp_del, but it may be any code that is called as part of 
tp_dealloc().  This patch allows an object to say:  Calling tp_dealloc() is 
currently unsafe from a gc.collect() run.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9141
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9141] Allow objects to decide if they can be collected by GC

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

 A finalizer _may_ be tp_del, but it may be any code that is called as
 part of tp_dealloc().  This patch allows an object to say:  Calling
 tp_dealloc() is currently unsafe from a gc.collect() run.

Well, I don't know what you mean. tp_dealloc should always be safe and
is never checked for by the GC.

By the way, tp_del isn't even documented, which reinforces my point that
it's not really for use by third-party extension types.

I think we are reaching the point where, if you want this to happen, you
should ask for opinions on python-dev, because I really don't understand
which problem you are trying to solve.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9141
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9141] Allow objects to decide if they can be collected by GC

[Kristján Valur Jónsson]

Kristján Valur Jónsson krist...@ccpgames.com added the comment:

tp_del() is generally invoked by tp_dealloc().  See for example 
typeobject.c:849.  gc.collect() never invokes tp_del() directy, that is left 
for the objects themselves (as part of tp_dealloc())

Now, gc already knows that if tp_del is present, it cannot cause break the 
cycle, (for various technical reasons, one of them being that tp_del would be 
invoked with various dependency objects already nerfed by tp_clear()).

But this is not always enough:
1) it may be too pessimistic.  Sometimes the tp_del method doesn't do anything 
significant, or isn't even called, depending on runtime state, so it is safe 
for gc to clear the object (see for example genobject.c:30).
2) Sometimes finalization behaviour is not defined by a tp_del method at all, 
but rather some custom code in tp_dealloc.

All this patch does, is to generalize the mechanism already provided for 
genobject.c (by PyGen_NeedsFinalizing()), to any object: Any object can signal 
to gc that: a) it is ok to collect a cycle with me in it, or b) no, it is 
unsafe to do so.  With this patch in place, PyGen_NeedsFinalizing() no longer 
needs to be a special case in gcmodule.c.

Yes, I'll send a message to python-dev.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9141
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9141] Allow objects to decide if they can be collected by GC

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

 All this patch does, is to generalize the mechanism already provided
 for genobject.c (by PyGen_NeedsFinalizing()), to any object: Any
 object can signal to gc that: a) it is ok to collect a cycle with me
 in it, or b) no, it is unsafe to do so.  With this patch in place,
 PyGen_NeedsFinalizing() no longer needs to be a special case in
 gcmodule.c.

Adding an API function and an additional traversal pass to the GC just
for the sake of removing a special case doesn't seem reasonable to me.
That's why I'm asking what specific problem you are trying to solve.
(rather than simply trying to make things nicer)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9141
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9141] Allow objects to decide if they can be collected by GC

[Kristján Valur Jónsson]

New submission from Kristján Valur Jónsson krist...@ccpgames.com:

The GC module currently relies only on the presence of a __del__ method to 
decide if an object can be safely collected at all.  Also, there is a special 
case for generator objects.

This patch allows any object to call an api during its traversal,
PyObject_GC_Collectable(), to indicate whether it is fit to be collected at 
this time, overriding any presence of a __del__ method or not.

This mechanism is being put in place in stackless python 2.7 because tasklets 
cannot always be collected depending on their runtime state, and I thought this 
might be a useful addition for regular python, especially since there already 
is such a dynamic special case for generator objects.

--
components: Interpreter Core
files: gc_collectable.patch
keywords: needs review, patch, patch
messages: 109099
nosy: krisvale
priority: normal
severity: normal
status: open
title: Allow objects to decide if they can be collected by GC
type: feature request
versions: Python 3.2
Added file: http://bugs.python.org/file17838/gc_collectable.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9141
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9141] Allow objects to decide if they can be collected by GC

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

First, what's the use case? Just backporting an internal Stackless API isn't a 
reasonable request.
Second, this tells that there *is* a finalization function, but not what the 
function is. How is the GC supposed to find it?
Third, the implementation looks quite suboptimal. Better define a new slot 
method, such as tp_needs_finalizing, or tp_get_finalizer.

--
nosy: +pitrou, tim_one

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9141
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9141] Allow objects to decide if they can be collected by GC

[Kristján Valur Jónsson]

Kristján Valur Jónsson krist...@ccpgames.com added the comment:

1. The use case is any C extension that may need to run non-trivial code when 
being deleted, but where this can not be statically known.  An example in the 
code today is generators, and the PyGen_NeedsFinalizing() function, doing 
exactly this.  This is an attempt to make this functionality available to any C 
extension.

2. No, this gives a C extension the oppertunity to say: It's okay to delete me 
from GC, or it's not okay to delete me from GC.  There is no need to specify 
any particular finalizing function, it will be invoked when the refcount goes 
to 0, just as it happens with __del__(), or as it happens with PyGen objects.

3. This code is only invoked for garbage deemed collectable.  As such it is not 
on any critical path, most gc collections don't actually find any garbage. The 
cost of a few extra indirect function calls is likely to drown in the memory 
allocator overhead when the objects are released. 

Also, this is designed to be a minimum impact patch.  We really shouldn't be 
creating new slot methods when it can be avoided.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9141
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9141] Allow objects to decide if they can be collected by GC

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

 1. The use case is any C extension that may need to run non-trivial
 code when being deleted, but where this can not be statically known.

tp_del is IMO a bad place to do it. I'd recommend tp_dealloc instead,
precisely so that you don't end up with uncollectable objects tied to
internal OS structures or other non-trivial resources.

(also, tp_del seems to have problems with subclassing. I don't remember
the specifics)

 3. This code is only invoked for garbage deemed collectable.  As such
 it is not on any critical path, most gc collections don't actually
 find any garbage. The cost of a few extra indirect function calls is
 likely to drown in the memory allocator overhead when the objects are
 released.

Ok.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9141
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com