Re: [Python-Dev] Weak references: dereference notification
Gustavo J. A. M. Carneiro wrote: > The object isn't really destroyed. Simply ob_refcnt drops to zero, > then tp_dealloc is called, which is supposed to destroy it. But since I > wrote tp_dealloc, I choose not to destroy it, Be aware that a C subclass of your wrapper that overrides tp_dealloc is going to have its tp_dealloc called before yours, and will therefore be partly destroyed before you get control. Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Weak references: dereference notification
Qui, 2005-11-10 às 13:57 +1300, Greg Ewing escreveu: > Gustavo J. A. M. Carneiro wrote: > > > OK, but what if it is a subclass of a builtin type, with instance > > variables? What if the PyObject is GC'ed but the ObjC object remains > > alive, and later you get a new reference to it? Do you create a new > > PyObject wrapper for it? What happened to the instance variables? > > Your proposed scheme appears to involve destroying and > then re-initialising the Python wrapper. Isn't that > going to wipe out any instance variables it may > have had? The object isn't really destroyed. Simply ob_refcnt drops to zero, then tp_dealloc is called, which is supposed to destroy it. But since I wrote tp_dealloc, I choose not to destroy it, and revive it by calling PyObject_Init(), which makes ob_refcnt == 1 again, among other things. > > Also, it seems to me that as soon as the refcount on > the wrapper drops to zero, any weak references to it > will be broken. Or does your resurrection code > intervene before that happens? Yes, I intervene before that happens. Regards. -- Gustavo J. A. M. Carneiro <[EMAIL PROTECTED]> <[EMAIL PROTECTED]> The universe is always one step beyond logic. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Weak references: dereference notification
Qui, 2005-11-10 às 08:15 +0100, "Martin v. Löwis" escreveu: > Gustavo J. A. M. Carneiro wrote: > > OK, but what if it is a subclass of a builtin type, with instance > > variables? What if the PyObject is GC'ed but the ObjC object remains > > alive, and later you get a new reference to it? Do you create a new > > PyObject wrapper for it? What happened to the instance variables? > > Normally, wrappers don't have state. But if you do have state, this > is how it could work: > > 1. Make two Python objects, PyState and PyWrapper (actually, > PyState doesn't need to be a Python object) > PyState holds the instance variables, and PyWrapper just > holds a pointer to a GObject. > 2. When a Python reference to a GObject is created for the > first time, create both a PyState and a PyWrapper. Have > the GObject point to the PyState, and the PyWrapper to > the GObject. Have the PyState weakly reference the > PyWrapper. > 3. When the refcount to the PyWrapper drops to zero, discard it. > 4. When somebody asks for the data in the PyWrapper, > go to the GObject, then to the PyState, and return the > data from there. > 5. When somebody wants a reference to a GObject which already > has a PyState, check the weak reference to find out > whether there is a PyWrapper already. If yes, return it; > if not, create a new one (and weakly reference it). > 6. When the GObject is discarded, drop the PyState as well. > > This has the following properties: > 1. There are no cyclic references for wrapping GObjects. > 2. Weakly-referencing wrappers is supported; if there > are no strong Python references to the wrapper, > the wrapper goes away, and, potentially, the GObject > as well. > 3. The PyState object lives as long as the GObject. > 4. Using "is" for GObjects/PyWrappers "works": there is > at most one PyWrapper per GObject at any time. > 5. id() of a GObject may change over time, if the wrapper > becomes unreferenced and then recreated. This was my first approach, actually, in patch 4.1 in [1]. Only your property 2 above drove me to try a different approach -- the weakrefs may become invalid while the GObject may still be alive. That's a bit "surprising". Of course, if I could override weakref.ref() for GObject wrapper types, even that could be worked around... ;-) Thanks, [1] http://bugzilla.gnome.org/show_bug.cgi?id=320428 -- Gustavo J. A. M. Carneiro <[EMAIL PROTECTED]> <[EMAIL PROTECTED]> The universe is always one step beyond logic. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Weak references: dereference notification
Gustavo J. A. M. Carneiro wrote: > OK, but what if it is a subclass of a builtin type, with instance > variables? What if the PyObject is GC'ed but the ObjC object remains > alive, and later you get a new reference to it? Do you create a new > PyObject wrapper for it? What happened to the instance variables? Normally, wrappers don't have state. But if you do have state, this is how it could work: 1. Make two Python objects, PyState and PyWrapper (actually, PyState doesn't need to be a Python object) PyState holds the instance variables, and PyWrapper just holds a pointer to a GObject. 2. When a Python reference to a GObject is created for the first time, create both a PyState and a PyWrapper. Have the GObject point to the PyState, and the PyWrapper to the GObject. Have the PyState weakly reference the PyWrapper. 3. When the refcount to the PyWrapper drops to zero, discard it. 4. When somebody asks for the data in the PyWrapper, go to the GObject, then to the PyState, and return the data from there. 5. When somebody wants a reference to a GObject which already has a PyState, check the weak reference to find out whether there is a PyWrapper already. If yes, return it; if not, create a new one (and weakly reference it). 6. When the GObject is discarded, drop the PyState as well. This has the following properties: 1. There are no cyclic references for wrapping GObjects. 2. Weakly-referencing wrappers is supported; if there are no strong Python references to the wrapper, the wrapper goes away, and, potentially, the GObject as well. 3. The PyState object lives as long as the GObject. 4. Using "is" for GObjects/PyWrappers "works": there is at most one PyWrapper per GObject at any time. 5. id() of a GObject may change over time, if the wrapper becomes unreferenced and then recreated. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Weak references: dereference notification
On 9-nov-2005, at 23:44, Gustavo J. A. M. Carneiro wrote: > On Wed, 2005-11-09 at 20:40 +0100, Ronald Oussoren wrote: >> On 9-nov-2005, at 18:52, Gustavo J. A. M. Carneiro wrote: >> >>> Qua, 2005-11-09 às 09:23 -0800, Guido van Rossum escreveu: >> Gustavo J. A. M. Carneiro wrote: >>> I have come across a situation where I find the current weak >>> references interface for extension types insufficient. >>> >>> Currently you only have a tp_weaklistoffset slot, pointing >>> to a >>> PyObject with weak references. However, in my case[1] I >>> _really_ need >>> to be notified when a weak reference is dereferenced. I find reading through the bug discussion a bit difficult to understand your use case. Could you explain it here? If you can't explain it you certainly won't get your problem solved! :-) >>> >>> This is a typical PyObject wrapping C object (GObject) problem. >>> Both >>> PyObject and GObject have independent reference counts. For each >>> GObject there is at most one PyObject wrapper. >>> >>> When the refcount on the wrapper drops to zero, tp_dealloc is >>> called. >>> In tp_dealloc, and if the GObject refcount is > 1, I do something >>> slightly evil: I 'resurect' the PyObject (calling PyObject_Init), >>> create >>> a weak reference to the GObject, and drop the "strong" reference. I >>> call this a 'hibernation state'. >> >> Why do you do that? The only reasons I can think of are that you hope >> to gain >> some speed from this or that you want to support weak references to >> the GObject. > > We want to support weak references to GObjects. Mainly because that > support has always been there and we don't want/can't break API. > And it > does have some uses... > >> >> For what its worth, in PyObjC we don't support weak references to the >> underlying >> Objective-C object and delete the proxy object when it is garbage >> collected. >> Objective-C also has reference counts, we increase that in the >> constructor for >> the proxy object and decrease it again in the destroctor. > > OK, but what if it is a subclass of a builtin type, with instance > variables? What if the PyObject is GC'ed but the ObjC object remains > alive, and later you get a new reference to it? Do you create a new > PyObject wrapper for it? What happened to the instance variables? Our main goal is that there is at most one wrapper for a python object alive at any one time. And likewise there is at most one Objective-C wrapper for a python object. If a PyObject is GC'ed and the ObjC object remains alive you will get a new PyObject when a reference to the ObjC object passes into python space again. That is no problem because the proxy object contains no state other than the pointer to the ObjC object. ObjC's runtime might be more flexible than that of GObject. If you create a subclass of an ObjC class the PyObjC runtime will create a real ObjC class for you and all object state, including Python instance variables, are stored on the ObjC side. > > Our goal in wrapping GObject is that, once a Python wrapper for a > GObject instance is created, it never dies until the GObject dies too. > At the same time, once the python wrapper loses all references, it > should not stop keeping the GObject alive. I tried that too, but ran into some very ugly issues and decided that weak references are not important enough for that. There's also the problem that this will keep the python proxy alive even when it is not needed anymore, which gives significant overhead of you traverse a large datastructure. What I don't quite understand is how you know that your python wrapper is the last reference to the GObject and your wrapper should not be forcefully kept alive. > > What happens currently, which is what I'm trying to change, is that > there is a reference loop between PyObject and GObject, so that > deallocation only happens with the help of the cyclic GC. But relying > on the GC for _everything_ causes annoying problems: At one time I used Python's reference counts as the reference count of the Objective-C object (ObjC's reference count management is done through method calls and can therefore be overridden in subclasses). That did work, but getting the semantics completely correct turned the code into a mess. Our current solution is much more satisfying. > > 1- The GC runs only once in a while, not soon enough if eg. you > have an > image object with several megabytes; > > 2- It makes it hard to debug reference counting bugs, as the symptom > only appears when the GC runs, far away from the code that cause the > problem in the first place; > > 3- Generally the GC has a lot more work, since every PyGTK object > needs > it, and a GUI app can have lots of PyGTK objects. > > Regards. > > -- > Gustavo J. A. M. Carneiro > <[EMAIL PROTECTED]> <[EMAIL PROTECTED]> > The universe is always one step beyond logic > > __
Re: [Python-Dev] Weak references: dereference notification
Gustavo J. A. M. Carneiro wrote: > OK, but what if it is a subclass of a builtin type, with instance > variables? What if the PyObject is GC'ed but the ObjC object remains > alive, and later you get a new reference to it? Do you create a new > PyObject wrapper for it? What happened to the instance variables? Your proposed scheme appears to involve destroying and then re-initialising the Python wrapper. Isn't that going to wipe out any instance variables it may have had? Also, it seems to me that as soon as the refcount on the wrapper drops to zero, any weak references to it will be broken. Or does your resurrection code intervene before that happens? -- Greg Ewing, Computer Science Dept, +--+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | [EMAIL PROTECTED] +--+ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Weak references: dereference notification
On Wed, 2005-11-09 at 20:40 +0100, Ronald Oussoren wrote: > On 9-nov-2005, at 18:52, Gustavo J. A. M. Carneiro wrote: > > > Qua, 2005-11-09 às 09:23 -0800, Guido van Rossum escreveu: > Gustavo J. A. M. Carneiro wrote: > > I have come across a situation where I find the current weak > > references interface for extension types insufficient. > > > > Currently you only have a tp_weaklistoffset slot, pointing to a > > PyObject with weak references. However, in my case[1] I > > _really_ need > > to be notified when a weak reference is dereferenced. > >> > >> I find reading through the bug discussion a bit difficult to > >> understand your use case. Could you explain it here? If you can't > >> explain it you certainly won't get your problem solved! :-) > > > > This is a typical PyObject wrapping C object (GObject) problem. > > Both > > PyObject and GObject have independent reference counts. For each > > GObject there is at most one PyObject wrapper. > > > > When the refcount on the wrapper drops to zero, tp_dealloc is > > called. > > In tp_dealloc, and if the GObject refcount is > 1, I do something > > slightly evil: I 'resurect' the PyObject (calling PyObject_Init), > > create > > a weak reference to the GObject, and drop the "strong" reference. I > > call this a 'hibernation state'. > > Why do you do that? The only reasons I can think of are that you hope > to gain > some speed from this or that you want to support weak references to > the GObject. We want to support weak references to GObjects. Mainly because that support has always been there and we don't want/can't break API. And it does have some uses... > > For what its worth, in PyObjC we don't support weak references to the > underlying > Objective-C object and delete the proxy object when it is garbage > collected. > Objective-C also has reference counts, we increase that in the > constructor for > the proxy object and decrease it again in the destroctor. OK, but what if it is a subclass of a builtin type, with instance variables? What if the PyObject is GC'ed but the ObjC object remains alive, and later you get a new reference to it? Do you create a new PyObject wrapper for it? What happened to the instance variables? Our goal in wrapping GObject is that, once a Python wrapper for a GObject instance is created, it never dies until the GObject dies too. At the same time, once the python wrapper loses all references, it should not stop keeping the GObject alive. What happens currently, which is what I'm trying to change, is that there is a reference loop between PyObject and GObject, so that deallocation only happens with the help of the cyclic GC. But relying on the GC for _everything_ causes annoying problems: 1- The GC runs only once in a while, not soon enough if eg. you have an image object with several megabytes; 2- It makes it hard to debug reference counting bugs, as the symptom only appears when the GC runs, far away from the code that cause the problem in the first place; 3- Generally the GC has a lot more work, since every PyGTK object needs it, and a GUI app can have lots of PyGTK objects. Regards. -- Gustavo J. A. M. Carneiro <[EMAIL PROTECTED]> <[EMAIL PROTECTED]> The universe is always one step beyond logic ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Weak references: dereference notification
On 9-nov-2005, at 18:52, Gustavo J. A. M. Carneiro wrote: > Qua, 2005-11-09 às 09:23 -0800, Guido van Rossum escreveu: Gustavo J. A. M. Carneiro wrote: > I have come across a situation where I find the current weak > references interface for extension types insufficient. > > Currently you only have a tp_weaklistoffset slot, pointing to a > PyObject with weak references. However, in my case[1] I > _really_ need > to be notified when a weak reference is dereferenced. >> >> I find reading through the bug discussion a bit difficult to >> understand your use case. Could you explain it here? If you can't >> explain it you certainly won't get your problem solved! :-) > > This is a typical PyObject wrapping C object (GObject) problem. > Both > PyObject and GObject have independent reference counts. For each > GObject there is at most one PyObject wrapper. > > When the refcount on the wrapper drops to zero, tp_dealloc is > called. > In tp_dealloc, and if the GObject refcount is > 1, I do something > slightly evil: I 'resurect' the PyObject (calling PyObject_Init), > create > a weak reference to the GObject, and drop the "strong" reference. I > call this a 'hibernation state'. Why do you do that? The only reasons I can think of are that you hope to gain some speed from this or that you want to support weak references to the GObject. For what its worth, in PyObjC we don't support weak references to the underlying Objective-C object and delete the proxy object when it is garbage collected. Objective-C also has reference counts, we increase that in the constructor for the proxy object and decrease it again in the destroctor. Ronald ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Weak references: dereference notification
Qua, 2005-11-09 às 09:23 -0800, Guido van Rossum escreveu: > > > Gustavo J. A. M. Carneiro wrote: > > > > I have come across a situation where I find the current weak > > > > references interface for extension types insufficient. > > > > > > > > Currently you only have a tp_weaklistoffset slot, pointing to a > > > > PyObject with weak references. However, in my case[1] I _really_ need > > > > to be notified when a weak reference is dereferenced. > > I find reading through the bug discussion a bit difficult to > understand your use case. Could you explain it here? If you can't > explain it you certainly won't get your problem solved! :-) This is a typical PyObject wrapping C object (GObject) problem. Both PyObject and GObject have independent reference counts. For each GObject there is at most one PyObject wrapper. When the refcount on the wrapper drops to zero, tp_dealloc is called. In tp_dealloc, and if the GObject refcount is > 1, I do something slightly evil: I 'resurect' the PyObject (calling PyObject_Init), create a weak reference to the GObject, and drop the "strong" reference. I call this a 'hibernation state'. Now the problem. Suppose the user had a weak ref to the PyObject: 1- At certain point in time, when the wrapper is in hibernation state, the user calls the weak ref 2- It gets a PyObject that contains a weak reference to the GObject; 3- Now suppose whatever was holding the GObject ref drops its reference, which was the last one, and the GObject dies; 4- Now the user does something with the PyObject obtained through the weakref -> invalid memory access. The cause for the problem is that between steps 2 and 3 the wrapper needs to change the weak reference to the GObject to a strong one. Unfortunately, I don't get any notification that 2 happened. BTW, I fixed this problem in the mean time with a bit more of slightly evil code. I override tp_call of the standard weakref type :-P [...] > > and weakref.ref was smart enough to lookup this type and use it, only > > _then_ it could work. > > Looks what you're looking for is a customizable factory fuction. Sure, if weakref.ref could be such a factory, and could take "advice" on what type of weakref to use for each class. Regards. -- Gustavo J. A. M. Carneiro <[EMAIL PROTECTED]> <[EMAIL PROTECTED]> The universe is always one step beyond logic. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Weak references: dereference notification
> > Gustavo J. A. M. Carneiro wrote: > > > I have come across a situation where I find the current weak > > > references interface for extension types insufficient. > > > > > > Currently you only have a tp_weaklistoffset slot, pointing to a > > > PyObject with weak references. However, in my case[1] I _really_ need > > > to be notified when a weak reference is dereferenced. I find reading through the bug discussion a bit difficult to understand your use case. Could you explain it here? If you can't explain it you certainly won't get your problem solved! :-) > > > What happens now > > > is that, when you call a weakref object, a simple Py_INCREF is done on > > > the referenced object. It would be easy to implement a new slot to > > > contain a function that should be called when a weak reference is > > > dereferenced. Or, alternatively, a slot or class attribute that > > > indicates an alternative type that should be used to create weak > > > references: instead of the builtin weakref object, a subtype of it, so > > > you can override tp_call. > > > > > > Does this sounds acceptable? [Jim Fulton] > > Since you can now (as of 2.4) subclass the weakref.ref class, you should be > > able to > > do this yourself in Python. See for example, weakref.KeyedRef. > > I know I can subclass it, but it doesn't change anything. If people > keep writing code like weakref.ref(myobj) instead of myweakref(myobj), > it still won't work. > > I wouldn't want to have to teach users of the library that they need > to use an alternative type; that seldom doesn't work. > > Now, if there was a place in the type that contained information like > > "for creating weak references of instances of this type, use this > weakref class" > > and weakref.ref was smart enough to lookup this type and use it, only > _then_ it could work. Looks what you're looking for is a customizable factory fuction. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Weak references: dereference notification
Qua, 2005-11-09 às 11:50 -0500, Jim Fulton escreveu: > Gustavo J. A. M. Carneiro wrote: > > Hello, > > > > I have come across a situation where I find the current weak > > references interface for extension types insufficient. > > > > Currently you only have a tp_weaklistoffset slot, pointing to a > > PyObject with weak references. However, in my case[1] I _really_ need > > to be notified when a weak reference is dereferenced. What happens now > > is that, when you call a weakref object, a simple Py_INCREF is done on > > the referenced object. It would be easy to implement a new slot to > > contain a function that should be called when a weak reference is > > dereferenced. Or, alternatively, a slot or class attribute that > > indicates an alternative type that should be used to create weak > > references: instead of the builtin weakref object, a subtype of it, so > > you can override tp_call. > > > > Does this sounds acceptable? > > Since you can now (as of 2.4) subclass the weakref.ref class, you should be > able to > do this yourself in Python. See for example, weakref.KeyedRef. I know I can subclass it, but it doesn't change anything. If people keep writing code like weakref.ref(myobj) instead of myweakref(myobj), it still won't work. I wouldn't want to have to teach users of the library that they need to use an alternative type; that seldom doesn't work. Now, if there was a place in the type that contained information like "for creating weak references of instances of this type, use this weakref class" and weakref.ref was smart enough to lookup this type and use it, only _then_ it could work. Thanks, -- Gustavo J. A. M. Carneiro <[EMAIL PROTECTED]> <[EMAIL PROTECTED]> The universe is always one step beyond logic. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Weak references: dereference notification
Gustavo J. A. M. Carneiro wrote: > Hello, > > I have come across a situation where I find the current weak > references interface for extension types insufficient. > > Currently you only have a tp_weaklistoffset slot, pointing to a > PyObject with weak references. However, in my case[1] I _really_ need > to be notified when a weak reference is dereferenced. What happens now > is that, when you call a weakref object, a simple Py_INCREF is done on > the referenced object. It would be easy to implement a new slot to > contain a function that should be called when a weak reference is > dereferenced. Or, alternatively, a slot or class attribute that > indicates an alternative type that should be used to create weak > references: instead of the builtin weakref object, a subtype of it, so > you can override tp_call. > > Does this sounds acceptable? Since you can now (as of 2.4) subclass the weakref.ref class, you should be able to do this yourself in Python. See for example, weakref.KeyedRef. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
