Rather than chucking this in the collector as-is, it would be useful if you could get me the output of the ZEO log rather than the output of the client log. Right now, the actual traceback is in the ZEO log; the client log only shows that a conflict error happened; not where it happened.

FWIW, I've not tested ZEO+tempstorage, as in my own deployments I always use "session affinity" which always sends subsequent requests to the same appserver that received a request containing a particular session cookie.

Thanks!

- C


On Jan 24, 2006, at 8:41 AM, Chris Withers wrote:

Okay, now that we have 2.8.4 in place, we get proper reporting of ConflictErrors and today we started seeing one happening over and over again which looked roughly as follows:

Traceback (most recent call last):
File "lib/python/Products/Transience/Transience.py", line 844, in new_or_existing
    item = self.get(key, _marker)
File "lib/python/Products/Transience/Transience.py", line 491, in get
    item = self._move_item(k, current_ts, default)
File "lib/python/Products/Transience/Transience.py", line 346, in _move_item
    if getattr(self._data[current_ts][k], 'setLastAccessed', None):
  File "lib/python/ZODB/Connection.py", line 704, in setstate
    self._setstate(obj)
  File "lib/python/ZODB/Connection.py", line 760, in _setstate
    self._reader.setGhostState(obj, p)
  File "lib/python/ZODB/serialize.py", line 495, in setGhostState
    state = self.getState(pickle)
  File "lib/python/ZODB/serialize.py", line 488, in getState
    return unpickler.load()
  File "lib/python/ZODB/serialize.py", line 436, in _persistent_load
    return self._conn.get(oid)
  File "lib/python/ZODB/Connection.py", line 207, in get
    p, serial = self._storage.load(oid, self._version)
  File "lib/python/ZEO/ClientStorage.py", line 746, in load
    return self.loadEx(oid, version)[:2]
  File "lib/python/ZEO/ClientStorage.py", line 769, in loadEx
    data, tid, ver = self._server.loadEx(oid, version)
  File "lib/python/ZEO/ServerStub.py", line 192, in loadEx
    return self.rpc.call("loadEx", oid, version)
  File "lib/python/ZEO/zrpc/connection.py", line 536, in call
    raise inst # error raised by server
ConflictError: database conflict error (oid 0x010f0d)

The things which got me worried is that it was always the same oid that was conflicting, and that's not usually something we see. We were also getting it very frequently, and on both reads and writes. That in itself is enough to get me to mail here. I had a dig on the ZEO server and found the matching traceback for one of them there:

------
2006-01-24T12:42:57 INFO ZEO.zrpc.Connection(S) (x.x.x.x:y)
loadEx()  raised exception: database conflict error (oid 0x010f0d)
Traceback (most recent call last):
  File "lib/python/ZEO/zrpc/connection.py",line 421, in handle_request
    ret = meth(*args)
  File "lib/python/ZEO/StorageServer.py",line 248, in loadEx
    return self.storage.loadEx(oid, version)
File"lib/python/tempstorage/TemporaryStorage.py", line 132, in loadEx
    data = self.load(oid, version)
  File"lib/python/tempstorage/TemporaryStorage.py", line 117, in load
    raise POSException.ConflictError(oid=oid)
ConflictError: database conflict error (oid 0x010f0d)

Now, I wanted to find out what the troublesome object behind 0x010f0d was, so I fired up and debug zope client, and was distrubed by the result:

>>> app.temp_folder._p_jar[p64(0x010f0d)]
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "lib/python/ZODB/Connection.py", line 207, in get
    p, serial = self._storage.load(oid, self._version)
  File "lib/python/ZEO/ClientStorage.py", line 746, in load
    return self.loadEx(oid, version)[:2]
  File "lib/python/ZEO/ClientStorage.py", line 769, in loadEx
    data, tid, ver = self._server.loadEx(oid, version)
  File "lib/python/ZEO/ServerStub.py", line 192, in loadEx
    return self.rpc.call("loadEx", oid, version)
  File "lib/python/ZEO/zrpc/connection.py", line 536, in call
    raise inst # error raised by server
ZODB.POSException.ConflictError: database conflict error (oid 0x010f0d)

We managed to stop the problem causing furthe conflict errors by fiddling with the timeout resolution on the Transient Object Container at /temp_folder/session_data through the web. I guess this re-created the data structures used by the temp_folder or session_data objects and so took the object with the dodgy oid out of the equation.

However, even now, an hour or so later, if I fire up a debug client and try and look up that oid, I get a ConflictError. Any idea what's going on?

cheers,

Chris
_______________________________________________
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


_______________________________________________
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev

Reply via email to