RE: [ZODB-Dev] More exciting ZODB errors ;-)
Tim Peters wrote at 2005-12-14 12:05 -0500:
...
return self.rpc.call('tpc_begin', id, user, descr, ext, tid, status)
File lib/python/ZEO/zrpc/connection.py, line 536, in call
raise inst # error raised by server
StorageTransactionError: Multiple simultaneous tpc_begin requests from one
client.
We see this occationally in ZODB 3.2 in the following situation:
From time to time our ZEO (running on Solaris) fails
to create a TemporaryFile (because Solaris raises
an OSError: error 11 -- resource temporarily unavailabe from fdopen).
Apparently, ZEO does not expect this kind of exception and
forgets to remove the lock set for this client.
As a consequence, any further commit trial by this client
results in the Multiple simulataneous tpc_begin... exception.
--
Dieter
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] sessions in the presence of conflicts
Dennis Allison wrote at 2005-12-14 12:58 -0800: ... _p_resolveConflict(self, oldState, savedState, newState) returns the state of the object after resolving different changes. The arguments are: oldState -- state of the object at the beginning of the current transaction (mutable) savedState -- state currently stored in the database. This state was written after oldState and reflect changes made by a transaction that committed before the current transaction (immutable) newState -- state after changes made by the current transaction (immutable) ... It seems to me that we can do much better for sessions because we know a bit about the semantics of sessions. A session object is a dictionary-like object mapping key-value pairs. Adding or deleting keys or changing the value associated with a key are independent operations and do not conflict unless the keys are duplicated in both the transactions. Any conflict resolution mechanism needs to be able to manage multiple keys independently since the session object is modified as a unit. In addition, new keys may be added and old keys deleted; any conflict resolution mechanism at the key level needs to be comprehend those operations. A more session-friendly conflict resolution might use: 1. if any of the states are invalid (that is, has a key '_invalid') return the invalid state. 2. if any any of the states attributes ['token','id','_created'] differ then there is a conflict, raise the conflict exception. 3. order the newState and savedState by modification time (or if that cannot be computed, by access time). newState is always after savedState (saveState represents a formerly committed state, newState the current state the commit of which failed). 4. any key appearing in oldState's dictionary but not appearing in both savedState and newState should be removed from all. This corresponds to a key-value pair being deleted in one of the transactions. Insertions will be managed automatically by the updates. That looks doubtful: It means, if any of the transactions deletes the key, then it should be removed. You favour deletion a bit. 5. beginning with the oldest, update oldState dictionary of key-value pairs using the dictionary part of newState and savedState. Return oldState. ... -- Dieter ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
RE: [ZODB-Dev] Re: [Zope-dev] Re: sessions in the presence of conflicts
[Chris McDonough] Note that changing the transientobject conflict resolution algorithm won't get rid of all write conflict errors, because the BTree-based indexes in the transient object container will still conflict during a bucket split and other situations that I can't exactly recall (they're documented in the BTrees source code). A more readable account is here: http://www.zope.org/Wikis/ZODB/BTreeConflictResolution BTrees are mappings too, and looks like Dennis is trying to apply similar conflict-resolution rules to session mapping objects. Conflict resolution algorithms are difficult and any algorithm will have DWIM-y tradeoffs, so it's useful to keep it as simple as possible. Or no more complex as is actually helpful ;-) ... [Dennis Allison] I have yet to figure out how to map a TransientObject state back to the object it represents, but it clearly is possible. I didn't see a response to that bit yet, so: the state of an object P is whatever P.__getstate__() returns. Given such a return value `state`, and some object Q of the same type as P, Q.__setstate__(state) gives Q the same state P had. What state means is entirely up to the type's __setstate__() and __getstate__() implementations (if any). Objects deriving from Persistent inherit (by default) implementations that retrieve and update an instance's __dict__. BTrees.Length is a good example of a class that overrides these methods, using an integer as the state. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Re: [Zope-dev] Re: sessions in the presence of conflicts
Zope 2.8.4, ZODB 3.4.2 Chris, I'm pretty sure that I mentioned having done that in one of my postings. I have followed your recommendations, but the problem remains. (um... persists grin) The systems are running a Zope/ZEO combination with a store configuration of: # zodb_db temporary # Temporary storage database (for sessions) temporarystorage name temporary storage for sessioning /temporarystorage mount-point /temp_folder container-class Products.TemporaryFolder.TemporaryContainer /zodb_db # # ZEO client storage: # zodb_db main mount-point / # ZODB cache, in number of objects cache-size 5000 zeoclient server 192.168.0.92:8301 storage 1 var $INSTANCE/var # ZEO client cache, in bytes cache-size 20MB # Uncomment to have a persistent disk cache client group1-zeo /zeoclient /zodb_db # Although the connection to ZEO is via a network port, it runs on the same physical hardware. TemporaryStorage is not transactional. Does it need to be under MVCC? TemporaryStorage does provide a conflict cache to do rudimentary conflict resolution. There are several timing and scaling parameters that need to be considered: CONFLICT_CACHE_MAXAGE = 60 (seconds) CONFLICT_CACHE_GCEVERY = 60 (seconds) RECENTLY_GC_OIDS_LEN = 200 Entries in the recently gc's oids list are those which may be resolvable by a retry. These numbers may be too small given the loads we see and the number of accesses made to the session variables. I plan to incrase them to see if there is any impact. MAYBE CONFLICTS AND THEIR RESOLUTION ARE NOT THE ROOT CAUSE OF THE SESSION VARIABLE PROBLEM. The observed problem is that session variables suddenly disappear. At the point of failure due to a KeyError, inspecting the SESSION object shows two failure modes: either all the session variables are gone and only the container remains or most of the session variables are gone and a few remain. 74769573A2H7SIH2AKo=id: 11343269231636975299, token: 74769573A2H7SIH2AKo, contents keys: ['currentTab', 'calendarPage', 'currentCourse', 'currentTextbook'] and 77307574A2HTTdXCYYg=id: 11343267811075063138, token: 77307574A2HTTdXCYYg, contents keys: [] Access to the session variables are almost alwsys through a pair of Scripts(Python). Occasionally a session variable is read with an expression of the form REQUEST['SESSION']['key']. ## Script (Python) getSessionVariable ##bind container=container ##bind context=context ##bind namespace= ##bind script=script ##bind subpath=traverse_subpath ##parameters=varname ##title= ## request=container.REQUEST session=request['SESSION'] return session[varname] # Script (Python) setSessionVariable ##bind container=container ##bind context=context ##bind namespace= ##bind script=script ##bind subpath=traverse_subpath ##parameters=var, val ##title= ## request = container.REQUEST RESPONSE = request.RESPONSE session=request['SESSION'] session[var]=val request.set( 'SESSION', session ) This all seems right to me. Any suggestions as to how to localized when the session variables get lost? That might help localize the root cause. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
FW: [ZODB-Dev] More exciting ZODB errors ;-)
FYI, I also added this info to Chris's other bug report, at http://www.zope.org/Collectors/Zope/1971 -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Dieter Maurer Sent: Thursday, December 15, 2005 11:52 AM To: Chris Withers Cc: zodb-dev@zope.org Subject: Re: [ZODB-Dev] More exciting ZODB errors ;-) Chris Withers wrote at 2005-12-14 16:23 +: ... File lib/python/ZODB/Connection.py, line 788, in _setstate_noncurrent assert end is not None AssertionError This means that the latest modification for this object lies before the respective transaction. In this case, we should not have an invalidation for the object, such that we would not call _setstate_current. I expect a missing flush_invalidations during Connection._setDB. I had to add such a call in our ZODB version: def _setDB(self, odb, mvcc=None, txn_mgr=DEPRECATED_ARGUMENT, transaction_manager=None, synch=None): self.transaction_manager = transaction_manager or transaction.manager # DM 2005-08-22: always call '_flush_invalidations' as it does # more than cache handling only self._flush_invalidations() if self._reset_counter != global_reset_counter: # New code is in place. Start a new cache. self._resetCache() # DM 2005-08-22: always call '_flush_invalidations' ##else: ##self._flush_invalidations() -- Dieter ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
FW: [ZODB-Dev] More exciting ZODB errors ;-)
FYI, I added this info to Chris's bug report, at
http://www.zope.org/Collectors/Zope/1970
Thanks!
-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Dieter Maurer
Sent: Thursday, December 15, 2005 11:02 AM
To: Tim Peters
Cc: 'Chris Withers'; zodb-dev@zope.org
Subject: RE: [ZODB-Dev] More exciting ZODB errors ;-)
Tim Peters wrote at 2005-12-14 12:05 -0500:
...
return self.rpc.call('tpc_begin', id, user, descr, ext, tid, status)
File lib/python/ZEO/zrpc/connection.py, line 536, in call
raise inst # error raised by server
StorageTransactionError: Multiple simultaneous tpc_begin requests from
one
client.
We see this occationally in ZODB 3.2 in the following situation:
From time to time our ZEO (running on Solaris) fails
to create a TemporaryFile (because Solaris raises
an OSError: error 11 -- resource temporarily unavailabe from fdopen).
Apparently, ZEO does not expect this kind of exception and
forgets to remove the lock set for this client.
As a consequence, any further commit trial by this client
results in the Multiple simulataneous tpc_begin... exception.
--
Dieter
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Dynamic Wrapper?
Instead of requiring all persistable objects to inherit a special class, wouldn't it be possible to dynamically wrap a class's __setattr__ and/or __setitem__ methods to determine when an object's been modified? Something like: class Monitor(object): class Proxy(object): def __init__(self, obj): self.obj = obj def __hash__(self): return id(self.obj) def __cmp__(self, p): return cmp(id(self.obj),id(p.obj)) def __init__(self): self.modified = set() def add(self, obj): def newSetter(func): def registerChanges(theirself, *args, **kwargs): self.modified.add(self.Proxy(theirself)) return func(theirself, *args, **kwargs) return registerChanges if hasattr(obj, '__setattr__'): obj.__class__.__setattr__ = newSetter(obj.__class__.__setattr__) if hasattr(obj, '__setitem__'): obj.__class__.__setitem__ = newSetter(obj.__class__.__setitem__) m = Monitor() from UserDict import UserDict d = UserDict() m.add(d) d[123] = 'abc' print m.modified set([__main__.Proxy object at 0xb7f28acc]) I understand this has some drawbacks. Namely, it will only work for new-style classes, but for a large code base this might be easier than manually writing _p_changed = 1 everywhere. Chris ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Re: [Zope-dev] Re: sessions in the presence of conflicts
On 12/16/05, Dennis Allison [EMAIL PROTECTED] wrote: MAYBE CONFLICTS AND THEIR RESOLUTION ARE NOT THE ROOT CAUSE OF THE SESSION VARIABLE PROBLEM. The observed problem is that session variables suddenly disappear. Perhaps your app is tripping over some bug in conflict handling. But I'd say it is worth entertain other ideas too. For now, just comment out all of TransientObject._p_resolveConflict and if you still get errors then you know you have to look elsewhere. (Sure, some of that elsewhere may well include the conflict resolution code above _p_resolveConflict.) Your application and sessions should cope just fine in the face of any ConflictError. ConflictError's are an essential part of the machinery that keeps data state consistent. As Chris mentions, look at how your using sessions and some of the assumptions you might be making. Might be useful to try with sessions that don't timeout, set session-timeout-minutes to 0. And try maximum-number-of-session-objects of 0. Also trying the turning those knobs the other way, session-timeout-minutes of 1 and maximum-number-of-session-objects of 2. For now, stay focused on making sure you maintain a consistent state. Only once you have a consistent state then is it worth trying to improve the rate of ConflictErrors. (Which in your case of sessions lasting for many hours I think the numbers you quote elsewhere are too small. And, yeah, an alternative implementation for _p_resolveConflict might help there. Personally I prefer the simple approach of just commenting that out completely and living with a slightly higher conflict count.) Might be worth trying without ZEO in the mix. Definitely worth the effort, if you have not already, to recreate the whole system on a separate host that you can feel comfortable making changes to. Then you can happily tune the various knobs downwards which may help with trying to observe the problem. For example session-timeout-minutes of 1. cheers michael ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
