Re: [ZODB-Dev] [Persistent] STICKY mechanism unsafe
Hi, just for my understanding: we don't get bitten by this because Zope uses the ThreadTransactionManager which keeps objects within one thread (at one time)? Christian ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Re: [Persistent] STICKY mechanism unsafe
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Christian Theune wrote: just for my understanding: we don't get bitten by this because Zope uses the ThreadTransactionManager which keeps objects within one thread (at one time)? Zope works for this case because each application thread uses a per-request connection, to which it has exclusive access while the connection is checked out from the pool (i.e., for the duration of the request). Tres. - -- === Tres Seaver +1 540-429-0999 [EMAIL PROTECTED] Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFGyZ6D+gerLs4ltQ4RAj2CAJ4p8Bejji45cnXHSKvnFRmRpMXviwCfS1tb 2aekSB3XuR6S7y0dOxm4lEU= =+MXl -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] [Persistent] STICKY mechanism unsafe
On Aug 18, 2007, at 5:01 AM, Dieter Maurer wrote: Excellent analysis snipped 1. and 3. (but obviously not 2.) could be handled by implementing STICKY not by a bit but by a counter. This has been planned for some. :/ Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Serializability
On Aug 19, 2007, at 9:34 AM, Dieter Maurer wrote: Analysing the STICKY behaviour of 'Persistent', I recognized that 'Persistent' does not customize the '__getattr__' but in fact the '__getattribute__' method. Therefore, 'Persistent' is informed about any attribute access and not only attribute access on a ghosted instance. Thogether with the 'accessed' call in Jim's proposal http://wiki.zope.org/ZODB/DecouplePersistenceDatabaseAndCache;, this could be used for a very crude check of potential serializability conflicts along the following lines. The DataManager maintains a set of objects accessed during a transaction. At transaction start, this set is empty and all cached objects are in state 'Ghost' or 'Saved'. Whenever an object is accessed for the first time, the DataManager's 'accessed' or 'register' method is called. In both cases, the manager adds the object to its accessed set. At transaction end, the manager can check whether the state of any of its accessed objects has changed in the meantime. If not, no serializability conflict happened. Otherwise, a conflict would be possible (provided the transaction changed any objects). Yup. I'd like someday to provide this sort of check as an option. The test is very crude, as it does not track whether the tracked transaction's change really depends on one of the objects changed by different transactions. We must expect lots of ConflictErrors. Perhaps. I would expect more, but whether there are lots will depend on the application. Application specific conflict resolution would become a really difficult task. I'm sure you realize that application specific conflict resolution violates serializability. I have a hard time believing that someone would use this higher-level serializability mode together with conflict resolution. At least not in general. Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] [Persistent] STICKY mechanism unsafe
On 8/20/07, Jim Fulton [EMAIL PROTECTED] wrote: On Aug 18, 2007, at 5:01 AM, Dieter Maurer wrote: Excellent analysis snipped 1. and 3. (but obviously not 2.) could be handled by implementing STICKY not by a bit but by a counter. This has been planned for some. :/ I think Jim mentioned this problem to me when I started working on ZODB :-). Jeremy Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] [Persistent] STICKY mechanism unsafe
On Aug 20, 2007, at 2:25 AM, Christian Theune wrote: Hi, just for my understanding: we don't get bitten by this because Zope uses the ThreadTransactionManager which keeps objects within one thread (at one time)? Dieter appears to have been bitten by this and he is one of we. :) We, and I presume he, can be bitten by a Python function called from BTree code calling back into the code on the same object. This is possible, for example, in a __cmp__ or related method. I assume that this is what happened to Dieter. Obviously, this would be a fairly special comparison method. In general though, you are right that the ZODB is designed to support a threading model in which multiple threads run single-threaded code in multiple isolated threads. It's worth noting that there are indirect ways that objects can be called from multiple threads: - Object deactivation. Until recently (3.8), objects could be deactivated from other threads while they were in use by n application threads. - __del__ methods or weakref callbacks can cause calls to persistent objects from other (non-application) threads. Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] [Persistent] STICKY mechanism unsafe
Am Montag, den 20.08.2007, 10:45 -0400 schrieb Jim Fulton: On Aug 20, 2007, at 2:25 AM, Christian Theune wrote: Hi, just for my understanding: we don't get bitten by this because Zope uses the ThreadTransactionManager which keeps objects within one thread (at one time)? Dieter appears to have been bitten by this and he is one of we. :) I didn't mean to separate him from us, thanks for pointing this out. I just tried to understand why the error doesn't occur more frequently. Thanks (also to Tres) for the explanations! ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Proposal for better object cache
Hi, Jim pointed out an idea for a better object cache when he was in Potsdam (at least IIRC). I've picked that idea up and converted it into a proposal at http://wiki.zope.org/ZODB/ClassifyingObjectCache I'd be happy to work on that when I have some rd time. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Re: [Persistent] STICKY mechanism unsafe
Tres Seaver wrote at 2007-8-20 10:00 -0400: ... Zope works for this case because each application thread uses a per-request connection, to which it has exclusive access while the connection is checked out from the pool (i.e., for the duration of the request). At least unless one make persistency errors, such as storing persistent objects outside the connection (e.g. on class level or in a global cache). -- Dieter ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] [Persistent] STICKY mechanism unsafe
Jim Fulton wrote at 2007-8-20 10:15 -0400: Excellent analysis snipped 1. and 3. (but obviously not 2.) could be handled by implementing STICKY not by a bit but by a counter. This has been planned for some. :/ I have (reread) this in your Different Cache Interaction proposal. Thanks to the GIL, it will also work for concurrent access from different threads -- if Used and Unused are notified while the GIL is held. -- Dieter ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] [Persistent] STICKY mechanism unsafe
Jim Fulton wrote at 2007-8-20 10:45 -0400: ... Dieter appears to have been bitten by this and he is one of we. :) We, and I presume he, can be bitten by a Python function called from BTree code calling back into the code on the same object. This is possible, for example, in a __cmp__ or related method. I assume that this is what happened to Dieter. Obviously, this would be a fairly special comparison method. I am not yet sure what really has bitten us -- I am not even sure whether the object was really deactivated or some memory corruption caused the object's tail to be overwritten by 0. When the SIGSEGV had hit, usually a bucket in a TextIndexNG3.lexicon was affected. This lexicon uses BTrees in a very innocent way. Its keys are integers and strings -- no fancy __cmp__ method is involved. Moreover, we need two things for the deactivation to happen: the STICKY mechanism must fail *AND* a deactivation must be called for. In our Zope/ZODB version, deactivation is done only at transaction boundaries (it is an early ZODB 3.4 version where snapshops did not yet call incrgc). Therefore, some commit would need to be done during the BUCKET_SEARCH call. The only conceivable cause appears to me that a different thread modified the bucket and called abort. This would mean a persistency bug (concurrent use of a persistent object by several threads). I tried to find such a bug in TextIndexNG3, but failed. The problem appears only very rarely -- about 1 to 2 times in about 1 to 2 month. When I analysed the problem in the past, I failed to look at the object's persistent state (it would have told me whether the object has been deactivated or overwritten). I just noticed that the object's head was apparently intact while the object's true data was 0. Only a few days ago, I recognized that this could have been the effect of a deactivation. -- Dieter ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
