Re: [Zope-dev] cPickleCache endless loop...
Am 26. Jan 2004, um 12:22:43 schrieb Tim Peters: It's actually that the number of __del__-resurrecting objects *plus* the number of non-ghostifiable objects in cache is larger than the cache target size, right? Yes, but when you push the minimize button, the cache target size is 0. Nonwithstanding our code (it was mostly debugging/tracing functionality anyway, so it was easy to fix), it is just that with Zope 2.5, it seemed to work (at least I never got a bug report back then) so it took us a while to track this down. Especially since we had just moved to RHEL3, so we were expecting more like a threading issue... Given that this property is not that widely published (in the various tutorials etc.), I wonder if it might be a good idea to improve that loop check code, and walk through the ring not more than once, using a counter, and not the comparison vs. the ring start, which, as we have seen, may be a target that moves too quickly. Similarly wink, except that if there's a large number of non-ghostifiable objects (more than the cache target size), then only one __del__-resurrected object is enough to provoke an infinite loop. Yes, this is exactly what happened here. Mario -- Mario Lorenz EMail: [EMAIL PROTECTED] Tel: 03774 6625-78 Technik Netze Handy: 0160 3151600 km3 teledienst GmbH Fax: 03774 6625-79 ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] cPickleCache endless loop...
On Tuesday 27 January 2004 19:08, Tim Peters wrote: Maybe Toby remembers which release(s) of ZODB the current cache implementation first appeared in Zope 2.6 -- Toby Dickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] cPickleCache endless loop...
On Tuesday 27 January 2004 09:39, Mario Lorenz wrote:
Given that this property is not that widely published (in the various
tutorials etc.), I wonder if it might be a good idea to improve that loop
check code, and walk through the ring not more than once, using a counter,
and not the comparison vs. the ring start, which, as we have seen, may be a
target that moves too quickly.
You got me curious
The attached patch to 2.6 uses an extra temporary ring node to mark where
scanning should stop. Normally this will be right next to the home node, and
behaviour is unchanged. Any of these troublesome objects that move themselves
to the top of the LRU list *during* the scan will end up *after* the scan
terminator node, and therefore do not get rescanned.
This works for me on 2.6, which is the only branch Im set up to use at the
moment.
--
Toby Dickenson
? diff
? diff2
Index: cPickleCache.c
===
RCS file: /cvs-repository/Zope/lib/python/ZODB/Attic/cPickleCache.c,v
retrieving revision 1.68.10.3
diff -c -4 -r1.68.10.3 cPickleCache.c
*** cPickleCache.c 30 Apr 2003 17:03:34 - 1.68.10.3
--- cPickleCache.c 27 Jan 2004 23:02:06 -
***
*** 216,224
#endif
static int
! scan_gc_items(ccobject *self,int target)
{
/* This function must only be called with the ring lock held,
because it places a non-object placeholder in the ring.
*/
--- 216,224
#endif
static int
! scan_gc_items(ccobject *self,int target,CPersistentRing *terminal)
{
/* This function must only be called with the ring lock held,
because it places a non-object placeholder in the ring.
*/
***
*** 265,281
return -1;
}
#endif
! /* back to the home position. stop looking */
! if (here == self-ring_home)
! return 0;
/* At this point we know that the ring only contains nodes
! from persistent objects, plus our own home node. We know
! this because the ring lock is held. We can safely assume
! the current ring node is a persistent object now we know it
! is not the home */
object = OBJECT_FROM_RING(self, here, scan_gc_items);
if (!object)
return -1;
--- 265,288
return -1;
}
#endif
! /* reached the end position. stop looking */
! if (here == terminal)
! return 0;
!
! /* back to the home position. should not happen */
! if (here == self-ring_home) {
! PyErr_SetString(PyExc_RuntimeError,
! reached home before terminal, in scan_gc_items);
! return -1;
! }
/* At this point we know that the ring only contains nodes
! from persistent objects, plus our own home node, plus the
!terminal node. We know this because the ring lock is held.
!We can safely assume the current ring node is a persistent
!object now. */
object = OBJECT_FROM_RING(self, here, scan_gc_items);
if (!object)
return -1;
***
*** 325,332
--- 332,343
static PyObject *
lockgc(ccobject *self, int target_size)
{
+ int error;
+ CPersistentRing terminal;
+
+
/* This is thread-safe because of the GIL, and there's nothing
* in between checking the ring_lock and acquiring it that calls back
* into Python.
*/
***
*** 338,350
if (IS_RING_CORRUPT(self, pre-gc))
return NULL;
ENGINE_NOISE();
self-ring_lock = 1;
! if (scan_gc_items(self, target_size)) {
! self-ring_lock = 0;
! return NULL;
! }
self-ring_lock = 0;
ENGINE_NOISE(\n);
if (IS_RING_CORRUPT(self, post-gc))
return NULL;
--- 349,376
if (IS_RING_CORRUPT(self, pre-gc))
return NULL;
ENGINE_NOISE();
self-ring_lock = 1;
!
! /* insert a placeholder just before the head node. This will be used to
! * terminate the scan_gc_items loop. Items touched during the
! * deactivation run will get inserted between the head and this terminal
! * node, and will not get re-scanned. */
! terminal.next = self-ring_home;
! terminal.prev = self-ring_home.prev;
! self-ring_home.prev-next = terminal;
! self-ring_home.prev = terminal;
!
! error = scan_gc_items(self, target_size, terminal);
!
! /* unlink the placeholder from the ring */
! terminal.next-prev = terminal.prev;
! terminal.prev-next = terminal.next;
!
self-ring_lock = 0;
+ if (error)
+ return NULL;
+
ENGINE_NOISE(\n);
if (IS_RING_CORRUPT(self, post-gc))
return NULL;
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
Re: [Zope-dev] cPickleCache endless loop...
On Friday 23 January 2004 17:08, Tim Peters wrote: It looks like ghostifying your self triggers self.__del__(). Then the __del__ method unghostifies self, which has the side effect of moving self to the MRU end of the ring, which in turn means the list traversal will visit self *again*. When it does, same thing happens all over again, ad infinitum. Not necessaralily ad infinitum. It will only run forever if the number of __del__-resurrecting objects in the cache is larger than the cache target size. Does that fit with your scenario? 2. If you need a __del__ method (it's hard to imagine why, since it will get called whenever the object is ghostified, and has nothing to do with the object's actual lifetime), don't reference any persistent objects (and esp. not self) within it. or 2b as jeremy suggested, put your __del__ on a non-persistent sub object. -- Toby Dickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] cPickleCache endless loop...
On Mon, Jan 26, 2004 at 08:59:36AM +, Chris Withers wrote: Paul Winkler wrote: On Fri, Jan 23, 2004 at 12:08:27PM -0500, Tim Peters wrote: def __del__(self): print About to destroy: , self.id I don't know what your intention is there, but fwiw, if what you're *really* interested in is the object being marked for deletion in the ZODB, you can use: I'm pretty sure that only works in Zope, so? the O.P. was about zope, and this IS the zope-dev list... and even then, when the ObjectManager stuff comes into play ;-) true enough. -- Paul Winkler http://www.slinkp.com Look! Up in the sky! It's THE FASCIST! (random hero from isometric.spaceninja.com) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] cPickleCache endless loop...
[Tim Peters] It looks like ghostifying your self triggers self.__del__(). Then the __del__ method unghostifies self, which has the side effect of moving self to the MRU end of the ring, which in turn means the list traversal will visit self *again*. When it does, same thing happens all over again, ad infinitum. [Toby Dickenson] Not necessaralily ad infinitum. It will only run forever if the number of __del__-resurrecting objects in the cache is larger than the cache target size. It's actually that the number of __del__-resurrecting objects *plus* the number of non-ghostifiable objects in cache is larger than the cache target size, right? Does that fit with your scenario? Similarly wink, except that if there's a large number of non-ghostifiable objects (more than the cache target size), then only one __del__-resurrected object is enough to provoke an infinite loop. 2. If you need a __del__ method (it's hard to imagine why, since it will get called whenever the object is ghostified, and has nothing to do with the object's actual lifetime), don't reference any persistent objects (and esp. not self) within it. or 2b as jeremy suggested, put your __del__ on a non-persistent sub object. Yup. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] cPickleCache endless loop...
On Monday 26 January 2004 17:22, Tim Peters wrote: It's actually that the number of __del__-resurrecting objects *plus* the number of non-ghostifiable objects in cache is larger than the cache target size, right? Yes, Right. That is more achievable than I thought. -- Toby Dickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] cPickleCache endless loop...
Jeremy Hylton wrote at 2004-1-23 12:12 -0500: ... The only useful answer is to avoid using __del__ on persistent objects. If there are resources that really need to be finalized whenever the object is ghosted, you can put them in a non-persistent sub-object that does have an __del__. A minor note (not essential for the current discussion): __del__ is not called when the object is ghostified but when the last (non cache) reference to it is deleted (i.e. when the object is flushed from memory). By the way, there seem to be some objects around which trigger this infinite loop behaviour. I noticed that a minimize in our production environment often leads to a no longer responding Zope instance. I think now that this may be caused by the above problem. We use mainly stock Zope and CMF objects, Postgres and MySQL database adapters and LocalFS. -- Dieter ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] cPickleCache endless loop...
Defining __del__ on a persistent object has unknown effects, FWIW. A persistent object's __del__ method may be called many times during its lifetime. See http://zope.org/Wikis/ZODB/FrontPage/guide/node3.html#SECTION00036 for more info. - C On Fri, 2004-01-23 at 09:55, Mario Lorenz wrote: Hello, we have spent most of the day tracking down obscure hangs of Zope (2.6.4rc1) under python2.1.3 on a RHEL3 machine. The problem seems to be a logic flaw somewhere related to the cPickleCache, when using a destructor in a Zope object that accesses itself. In our case(shortened to the offending line): def __del__(self): print About to destroy: , self.id What seems to happen is that the self.id access causes the object to be cached again, causing scan_gc_items() to run in circles. Any ideas on how to best fix this? Mario ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] cPickleCache endless loop...
[Mario Lorenz] we have spent most of the day tracking down obscure hangs of Zope (2.6.4rc1) under python2.1.3 on a RHEL3 machine. Based on what you say next, it sure sounds like this isn't unique to 2.6.4rc1. Did the same code work under some previous release? The infinite loop appears to be an inherent property of this iteration of the cPickleCache design, and that's not new in 2.6.4rc1. The problem seems to be a logic flaw somewhere related to the cPickleCache, when using a destructor in a Zope object that accesses itself. In our case(shortened to the offending line): def __del__(self): print About to destroy: , self.id What seems to happen is that the self.id access causes the object to be cached again, causing scan_gc_items() to run in circles. Based on eyeballing the C code, the ring is a list of objects in cache, ordered from least recently used to most recently used. scan_gc_items traverses this list once, from LRU to MRU, ghosting ghostifiable objects until the target number of non-ghost objects remaining is reached, or the entire list has been traversed. It looks like ghostifying your self triggers self.__del__(). Then the __del__ method unghostifies self, which has the side effect of moving self to the MRU end of the ring, which in turn means the list traversal will visit self *again*. When it does, same thing happens all over again, ad infinitum. Any ideas on how to best fix this? As the docs Chris pointed you at say, persistent objects shouldn't have __del__ methods. If the by-eyeball analysis above is correct, if a persistent object does have a __del__ method referencing an attribute of self, an infinite loop in scan_gc_items() is inevitable. So I only see 3 workarounds short of rewriting the C code: 1. Lose the __del__ method (recommended). 2. If you need a __del__ method (it's hard to imagine why, since it will get called whenever the object is ghostified, and has nothing to do with the object's actual lifetime), don't reference any persistent objects (and esp. not self) within it. 3. Recompile with MUCH_RING_CHECKING defined. Then scan_gc_items will give up after max(cache_size * 10, 1) iterations instead of running forever. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] cPickleCache endless loop...
On Fri, Jan 23, 2004 at 12:08:27PM -0500, Tim Peters wrote: def __del__(self): print About to destroy: , self.id I don't know what your intention is there, but fwiw, if what you're *really* interested in is the object being marked for deletion in the ZODB, you can use: def manage_beforeDelete(self): print About to destroy:, self.id -- Paul Winkler http://www.slinkp.com Look! Up in the sky! It's REPUBLICAN DOC! (random hero from isometric.spaceninja.com) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
