Hi Laurence,
Thanks for your reply and sorry for the delayed response (been fighting fires).
On 01/24/2012 09:18 PM, Laurence Rowe wrote:
On 24 January 2012 13:50, steve st...@lonetwin.net wrote:
Hi All,
...snip...
Any pointers/comments would be appreciated.
(Following up only on zodb-dev as I'm not subscribed to the other lists.)
I'm guessing, but I suspect your load tests may only be reading from
the ZODB so you rarely see any cache misses.
Yes, that's correct, we do test (mostly) read operations.
The most important tuning paramaters for ZODB in respect to memory
usage are the number of threads and the connection_cache_size. The
connection_cache_size controls the number of persistent objects kept
live in the interpreter at a time. It's a per-connection setting and
as each thread needs its own connection. Memory usage increases
proportionally to connection_cache_size * number of threads. Most
people use either one or two threads per process with the ZODB. I know
plone.recipe.zope2instance defaults to two threads per process, though
I think this is only to avoid locking up in the case of Plone being
configured to load an RSS feed from itself.
The Python Global Interpreter Lock prevents threads from running
concurrently, so with ZEO running so many threads per process is
likely to be counter-productive. Try with one or two threads and
perhaps up the connection_cache_size (though loading from the zeo
cache is very quick you must ensure your working set fits in the
connection cache or else you'll be loading the same objects every
request). If your CPU usage goes down a lot it means your spending
time waiting for objects to be loaded in the connection and you might
want to run a couple more processes than you have cpus if you are
running one thread per process.
Thanks for that information. That is helpful. Yes, I shall try and experiment
with the connection_cache_size.
I also suggest you check that your mod_wsgi config is correctly
specifying WSGIProcessGroup, see:
http://www.martinaspeli.net/articles/update-repoze-under-mod-wsgi-is-not-slow
and
http://code.google.com/p/modwsgi/wiki/ConfigurationDirectives#WSGIDaemonProcess.
Yes, it is.
To get more information about what is going on, try porting
https://github.com/collective/collective.stats to work as WSGI
middleware instead of hooking into Zope2's ZPublisher.
Hmm, this is interesting too. I'll try to see if I can port and fit it into our
app.
Thanks for taking the time.
cheers,
- steve
--
random spiel: http://lonetwin.net/
what i'm stumbling into: http://lonetwin.stumbleupon.com/
___
For more information about ZODB, see http://zodb.org/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
