Re: [ZODB-Dev] [Persistent] STICKY mechanism unsafe
Hi, just for my understanding: we don't get bitten by this because Zope uses the ThreadTransactionManager which keeps objects within one thread (at one time)? Christian ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] [Persistent] STICKY mechanism unsafe
On Aug 18, 2007, at 5:01 AM, Dieter Maurer wrote: Excellent analysis snipped 1. and 3. (but obviously not 2.) could be handled by implementing STICKY not by a bit but by a counter. This has been planned for some. :/ Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] [Persistent] STICKY mechanism unsafe
On 8/20/07, Jim Fulton [EMAIL PROTECTED] wrote: On Aug 18, 2007, at 5:01 AM, Dieter Maurer wrote: Excellent analysis snipped 1. and 3. (but obviously not 2.) could be handled by implementing STICKY not by a bit but by a counter. This has been planned for some. :/ I think Jim mentioned this problem to me when I started working on ZODB :-). Jeremy Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] [Persistent] STICKY mechanism unsafe
On Aug 20, 2007, at 2:25 AM, Christian Theune wrote: Hi, just for my understanding: we don't get bitten by this because Zope uses the ThreadTransactionManager which keeps objects within one thread (at one time)? Dieter appears to have been bitten by this and he is one of we. :) We, and I presume he, can be bitten by a Python function called from BTree code calling back into the code on the same object. This is possible, for example, in a __cmp__ or related method. I assume that this is what happened to Dieter. Obviously, this would be a fairly special comparison method. In general though, you are right that the ZODB is designed to support a threading model in which multiple threads run single-threaded code in multiple isolated threads. It's worth noting that there are indirect ways that objects can be called from multiple threads: - Object deactivation. Until recently (3.8), objects could be deactivated from other threads while they were in use by n application threads. - __del__ methods or weakref callbacks can cause calls to persistent objects from other (non-application) threads. Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] [Persistent] STICKY mechanism unsafe
Am Montag, den 20.08.2007, 10:45 -0400 schrieb Jim Fulton: On Aug 20, 2007, at 2:25 AM, Christian Theune wrote: Hi, just for my understanding: we don't get bitten by this because Zope uses the ThreadTransactionManager which keeps objects within one thread (at one time)? Dieter appears to have been bitten by this and he is one of we. :) I didn't mean to separate him from us, thanks for pointing this out. I just tried to understand why the error doesn't occur more frequently. Thanks (also to Tres) for the explanations! ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] [Persistent] STICKY mechanism unsafe
Jim Fulton wrote at 2007-8-20 10:15 -0400: Excellent analysis snipped 1. and 3. (but obviously not 2.) could be handled by implementing STICKY not by a bit but by a counter. This has been planned for some. :/ I have (reread) this in your Different Cache Interaction proposal. Thanks to the GIL, it will also work for concurrent access from different threads -- if Used and Unused are notified while the GIL is held. -- Dieter ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] [Persistent] STICKY mechanism unsafe
Jim Fulton wrote at 2007-8-20 10:45 -0400: ... Dieter appears to have been bitten by this and he is one of we. :) We, and I presume he, can be bitten by a Python function called from BTree code calling back into the code on the same object. This is possible, for example, in a __cmp__ or related method. I assume that this is what happened to Dieter. Obviously, this would be a fairly special comparison method. I am not yet sure what really has bitten us -- I am not even sure whether the object was really deactivated or some memory corruption caused the object's tail to be overwritten by 0. When the SIGSEGV had hit, usually a bucket in a TextIndexNG3.lexicon was affected. This lexicon uses BTrees in a very innocent way. Its keys are integers and strings -- no fancy __cmp__ method is involved. Moreover, we need two things for the deactivation to happen: the STICKY mechanism must fail *AND* a deactivation must be called for. In our Zope/ZODB version, deactivation is done only at transaction boundaries (it is an early ZODB 3.4 version where snapshops did not yet call incrgc). Therefore, some commit would need to be done during the BUCKET_SEARCH call. The only conceivable cause appears to me that a different thread modified the bucket and called abort. This would mean a persistency bug (concurrent use of a persistent object by several threads). I tried to find such a bug in TextIndexNG3, but failed. The problem appears only very rarely -- about 1 to 2 times in about 1 to 2 month. When I analysed the problem in the past, I failed to look at the object's persistent state (it would have told me whether the object has been deactivated or overwritten). I just noticed that the object's head was apparently intact while the object's true data was 0. Only a few days ago, I recognized that this could have been the effect of a deactivation. -- Dieter ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] [Persistent] STICKY mechanism unsafe
We currently see occational SIGSEGVs in BTrees/BucketTemplate.c:_bucket_get. I am not yet sure but it looks as if the object had been deactivated during the BUCKET_SEARCH. Trying to analyse the problem, I had a close look at the STICKY mechanism of persistent.Persistent which should prevent accidental deactivation -- and found it unsafe. STICKY is one of the states on persistent objects -- beside GHOST, UPTODATE and CHANGED. It is mostly equivalent to UPTODATE but prevents deactivation (but not invalidation). It is used by C extensions that may release the GIL or call back to Python (which may indirectly release the GIL). Their typical usage pattern is if (obj-state == GHOST) obj-unghostify(); if (obj-state == UPTODATE) obj-state = STICKY; ... do whatever needs to be done with obj ... if (obj-state == STICKY) obj-state = UPTODATE This usage pattern obviously breaks when a similar code sequence is executed for obj while ... do whatever .. is executed as it resets STICKY too early (in the nested code sequence rather than the original one). This may happen in several ways: 1. ... do whatever ... does it explicitly 2. obj is accessed from a different thread 3. obj is accessed from a Python callback 1. might be considered a bug in ... do whatever ... -- although one that is not easily avoidable. 2. is a general problem. Not only STICKY is unsafe against concurrent use -- the complete state model of Persistent is. We might explicitly state that the concurrent use of persistent objects is unsafe and check against it. With respect to STICKY, all three cases can be detected by prepending if (obj-state == STICKY) ERROR;. 1. and 3. (but obviously not 2.) could be handled by implementing STICKY not by a bit but by a counter. -- Dieter ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
