On Wed, Aug 26, 2009 at 8:08 AM, Chris Withersch...@simplistix.co.uk wrote:
Hi All,
I'm wondering if there's any way to get a ZEO server to log about how
it's performing?
(eg: how many connections, how many clients are waiting, where they're
waiting, etc)
I'm trying to debug a problem where app servers are being dropped out of
a load balancer pool for being unresponsive, and I think they're waiting
on the ZEO server so I'm trying to get some stats on the ZEO server side
as to what's going on...
As far as the clients go, DeadlockDebugger is providing some insights:
File /opt/Zope-2.9/lib/python/ZEO/ClientStorage.py, line 749, in loadEx
self._lock.acquire() # for atomic processing of invalidations
I assume this is the client waiting for the storage server's lock?
Is this the ZEO server's single threadedness or is this a lock between
threads on the app server?
File /opt/Zope-2.9/lib/python/ZEO/ClientStorage.py, line 769, in loadEx
data, tid, ver = self._server.loadEx(oid, version)
File /opt/Zope-2.9/lib/python/ZEO/ServerStub.py, line 192, in loadEx
return self.rpc.call(loadEx, oid, version)
File /opt/Zope-2.9/lib/python/ZEO/zrpc/connection.py, line 531, in call
r_flags, r_args = self.wait(msgid)
File /opt/Zope-2.9/lib/python/ZEO/zrpc/connection.py, line 638, in wait
asyncore.poll(delay, self._singleton)
File /usr/local/lib/python2.4/asyncore.py, line 122, in poll
r, w, e = select.select(r, w, e, timeout)
What does this mean? I'm guessing it's the actual transfer of data from
the storage server to the app server, right?
What version of ZODB are you using? If you have zeo on in debugging
what you can see is if there are outstanding clients waiting to write
while a transaction is already writing. It says something like # clients
blocked while transaction in progress in zeo server log files.
How many connections:
iirc zrpc is connection oriented so you should have # of application clients
connected to ZEO server. Or it may be # of clients * # of ZODB threads.
How many clients are waiting:
You should be able to see this in zeo log files
Where they're waiting:
deadlockdebugger or products.signalstack will show what the clients are doing.
may want to strace the client and server to see if there is long
pauses of inactivity.
maybe your running up against a real ZODB bug that was fixed in later releases?
look at ZODB 3.8 branch NEWS.txt
good luck
alan
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev
