[ZODB-Dev] Re: [Zope-dev] Re: sessions in the presence of conflicts
Tim Peters wrote: [Dennis Allison] I have yet to figure out how to map a TransientObject state back to the object it represents, but it clearly is possible. I didn't see a response to that bit yet, so: the state of an object P is whatever P.__getstate__() returns. Given such a return value `state`, and some object Q of the same type as P, Q.__setstate__(state) gives Q the same state P had. What state means is entirely up to the type's __setstate__() and __getstate__() implementations (if any). Objects deriving from Persistent inherit (by default) implementations that retrieve and update an instance's __dict__. BTrees.Length is a good example of a class that overrides these methods, using an integer as the state. Plagiarism! ;) http://mail.zope.org/pipermail/zodb-dev/2005-December/009560.html Florent -- Florent Guillaume, Nuxeo (Paris, France) CTO, Director of RD +33 1 40 33 71 59 http://nuxeo.com [EMAIL PROTECTED] ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] RE: [Zope-dev] Re: sessions in the presence of conflicts
[Florent Guillaume] Plagiarism! ;) http://mail.zope.org/pipermail/zodb-dev/2005-December/009560.html Fascinating! I didn't know plagiarism was French for open source ;-) ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
RE: [ZODB-Dev] Re: [Zope-dev] Re: sessions in the presence of conflicts
[Chris McDonough] Note that changing the transientobject conflict resolution algorithm won't get rid of all write conflict errors, because the BTree-based indexes in the transient object container will still conflict during a bucket split and other situations that I can't exactly recall (they're documented in the BTrees source code). A more readable account is here: http://www.zope.org/Wikis/ZODB/BTreeConflictResolution BTrees are mappings too, and looks like Dennis is trying to apply similar conflict-resolution rules to session mapping objects. Conflict resolution algorithms are difficult and any algorithm will have DWIM-y tradeoffs, so it's useful to keep it as simple as possible. Or no more complex as is actually helpful ;-) ... [Dennis Allison] I have yet to figure out how to map a TransientObject state back to the object it represents, but it clearly is possible. I didn't see a response to that bit yet, so: the state of an object P is whatever P.__getstate__() returns. Given such a return value `state`, and some object Q of the same type as P, Q.__setstate__(state) gives Q the same state P had. What state means is entirely up to the type's __setstate__() and __getstate__() implementations (if any). Objects deriving from Persistent inherit (by default) implementations that retrieve and update an instance's __dict__. BTrees.Length is a good example of a class that overrides these methods, using an integer as the state. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Re: [Zope-dev] Re: sessions in the presence of conflicts
Zope 2.8.4, ZODB 3.4.2 Chris, I'm pretty sure that I mentioned having done that in one of my postings. I have followed your recommendations, but the problem remains. (um... persists grin) The systems are running a Zope/ZEO combination with a store configuration of: # zodb_db temporary # Temporary storage database (for sessions) temporarystorage name temporary storage for sessioning /temporarystorage mount-point /temp_folder container-class Products.TemporaryFolder.TemporaryContainer /zodb_db # # ZEO client storage: # zodb_db main mount-point / # ZODB cache, in number of objects cache-size 5000 zeoclient server 192.168.0.92:8301 storage 1 var $INSTANCE/var # ZEO client cache, in bytes cache-size 20MB # Uncomment to have a persistent disk cache client group1-zeo /zeoclient /zodb_db # Although the connection to ZEO is via a network port, it runs on the same physical hardware. TemporaryStorage is not transactional. Does it need to be under MVCC? TemporaryStorage does provide a conflict cache to do rudimentary conflict resolution. There are several timing and scaling parameters that need to be considered: CONFLICT_CACHE_MAXAGE = 60 (seconds) CONFLICT_CACHE_GCEVERY = 60 (seconds) RECENTLY_GC_OIDS_LEN = 200 Entries in the recently gc's oids list are those which may be resolvable by a retry. These numbers may be too small given the loads we see and the number of accesses made to the session variables. I plan to incrase them to see if there is any impact. MAYBE CONFLICTS AND THEIR RESOLUTION ARE NOT THE ROOT CAUSE OF THE SESSION VARIABLE PROBLEM. The observed problem is that session variables suddenly disappear. At the point of failure due to a KeyError, inspecting the SESSION object shows two failure modes: either all the session variables are gone and only the container remains or most of the session variables are gone and a few remain. 74769573A2H7SIH2AKo=id: 11343269231636975299, token: 74769573A2H7SIH2AKo, contents keys: ['currentTab', 'calendarPage', 'currentCourse', 'currentTextbook'] and 77307574A2HTTdXCYYg=id: 11343267811075063138, token: 77307574A2HTTdXCYYg, contents keys: [] Access to the session variables are almost alwsys through a pair of Scripts(Python). Occasionally a session variable is read with an expression of the form REQUEST['SESSION']['key']. ## Script (Python) getSessionVariable ##bind container=container ##bind context=context ##bind namespace= ##bind script=script ##bind subpath=traverse_subpath ##parameters=varname ##title= ## request=container.REQUEST session=request['SESSION'] return session[varname] # Script (Python) setSessionVariable ##bind container=container ##bind context=context ##bind namespace= ##bind script=script ##bind subpath=traverse_subpath ##parameters=var, val ##title= ## request = container.REQUEST RESPONSE = request.RESPONSE session=request['SESSION'] session[var]=val request.set( 'SESSION', session ) This all seems right to me. Any suggestions as to how to localized when the session variables get lost? That might help localize the root cause. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Re: [Zope-dev] Re: sessions in the presence of conflicts
On 12/16/05, Dennis Allison [EMAIL PROTECTED] wrote: MAYBE CONFLICTS AND THEIR RESOLUTION ARE NOT THE ROOT CAUSE OF THE SESSION VARIABLE PROBLEM. The observed problem is that session variables suddenly disappear. Perhaps your app is tripping over some bug in conflict handling. But I'd say it is worth entertain other ideas too. For now, just comment out all of TransientObject._p_resolveConflict and if you still get errors then you know you have to look elsewhere. (Sure, some of that elsewhere may well include the conflict resolution code above _p_resolveConflict.) Your application and sessions should cope just fine in the face of any ConflictError. ConflictError's are an essential part of the machinery that keeps data state consistent. As Chris mentions, look at how your using sessions and some of the assumptions you might be making. Might be useful to try with sessions that don't timeout, set session-timeout-minutes to 0. And try maximum-number-of-session-objects of 0. Also trying the turning those knobs the other way, session-timeout-minutes of 1 and maximum-number-of-session-objects of 2. For now, stay focused on making sure you maintain a consistent state. Only once you have a consistent state then is it worth trying to improve the rate of ConflictErrors. (Which in your case of sessions lasting for many hours I think the numbers you quote elsewhere are too small. And, yeah, an alternative implementation for _p_resolveConflict might help there. Personally I prefer the simple approach of just commenting that out completely and living with a slightly higher conflict count.) Might be worth trying without ZEO in the mix. Definitely worth the effort, if you have not already, to recreate the whole system on a separate host that you can feel comfortable making changes to. Then you can happily tune the various knobs downwards which may help with trying to observe the problem. For example session-timeout-minutes of 1. cheers michael ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
