Re: [Zope] large installations and conflict errors
On Aug 9, 2005, at 7:01 AM, Sascha Ottolski wrote: What we've seen so far: we used to set the cache-size to 250 MB; and every know and than we have to observe that after a restart of a client a cache verification is necessary (while not always obvious, why this happened), which often takes "forever" (that is, 10 to 20 minutes are possible). And that during such a verification, the whole system seems to be down (in the sense that delivery of pages is slow as hell :-(). I think that in such a case simply deleting the cache file and starting with a fresh one is way faster, but have no hard numbers to prove that feeling. In Zope 2.6, we wound up turning off persistent cache files. Having things be a little bit slow on one client while the cache filled up was preferable to the systemwide torture the re-verification of a cache file. Zope 2.7 and higher have a shortcut to cache verification, which can be adjusted with the "invalidation-queue- size" parameter. Unfortunately, there are times where the ZEO client will decide to do a re-verification of a transient cache file (I think due to unexpected runtime exceptions) The best we've found to do is to monitor the health of the clients and the servers and when they all get simultaneously busy, find the client re-verifying and restart it. ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] large installations and conflict errors
Am Dienstag, 9. August 2005 02:03 schrieb Andrew Langmead: > (on these > machines, the ZEO client cache is set to 2GB and a cache flip occurs > maybe twice a week.) I think this part to be quite interesting. We too are experimenting with the optimal cache sizes, where our (daily) packed Data.fs is "only" about 7 GB. Its only since friday that we updated to the latest 2.8-svn codebase, to get the latest fixes of the ZODB/ZEO cache tracing parts. Our setup is one ZEO server, and two clients on seperate machines. A third client is on the same machine as the server, which is only used for admin/maintenance tasks. What we've seen so far: we used to set the cache-size to 250 MB; and every know and than we have to observe that after a restart of a client a cache verification is necessary (while not always obvious, why this happened), which often takes "forever" (that is, 10 to 20 minutes are possible). And that during such a verification, the whole system seems to be down (in the sense that delivery of pages is slow as hell :-(). I think that in such a case simply deleting the cache file and starting with a fresh one is way faster, but have no hard numbers to prove that feeling. Now we look what happens at different cache sizes. While it might be too early to really make conclusions, at least it might give some hints: We have set the cache on one machine to be 200 MB, and on the other to 100 MB. Up to now, even the 100 MB cache gives a hitrate of better than 98 % (!) most of the time. What I hope is, that the smaller the cache size, the faster the verification, if necessary. Again, no real hard numbers to check this (seems to be hard to get some, sometimes the log says things like 2005-08-08T00:10:55 INFO ZEO.ClientStorage (27891) No verification necessary (last_inval_tid up-to-date) -- 2005-08-08T00:10:55 INFO ZEO.ClientStorage (27891) Waiting for cache verification to finish -- 2005-08-08T00:10:55 INFO ZEO.ClientStorage (27891) Verifying cache -- 2005-08-08T00:10:55 INFO ZEO.ClientStorage (27891) endVerify finishing -- 2005-08-08T00:10:55 INFO ZEO.ClientStorage (27891) endVerify finished huh? no verifictation necesarry, but waiting for verification to finish? sometimes, the "endVerify" messages never appear; and you never know which of our three storages the messages are about...). We increased the servers invalidation-queue-size from 100 to 1000 along the way, don't know yet if this has any effect. Having said all this, our scenario might not be common, as a major part of the traffic concentrates on a small part of the site, which might be the reason that even a small cache seems to be sufficient here. I'm wondering if anybody else has seen long lasting cache verifications, which seems to block the whole system. Or if someone knows how to prevent this from happening at all... Cheers, Sascha ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] large installations and conflict errors
On Aug 8, 2005, at 10:01 AM, M. Krainer wrote: So far our story, but what I really wonder is, if there's anyone out there who has a similar large installation. Please let me know how large your zope instance is and what you have done to increase your (write) performance. Also any ideas that may help us are welcome. We have a ZODB that packs down to about 30 gigs. The unpacked size grows about 10 gigs a week, which shows that there is a lot of write activity in our environment too. We have three Zope instances as ZEO clients (1.4Gig PIII, with about two gigs of RAM) A load balancer in front of those machines is set to favor certain URL prefixes towards the same machine. This someone unbalanced set up for the load balancer improves the chance that the ZEO client cache will have the appropriate object and avoid accessing the ZODB for it. (on these machines, the ZEO client cache is set to 2GB and a cache flip occurs maybe twice a week.) Three other machines are handling purely non-interactive tasks (either through wget or through the Scheduler product) If possible, these machines are set up with a single zope thread and a large memory cache. (instead of the standard setup with four threads of xMB each, it is one thread of x*4MB.) Not only does this help with the speed of a request, but prevents each threads private object cache from having duplicate copies of the same object. (these machines also have a 2GB ZEO client cache, but flip daily) A ZCatalog has a index that is single large Zope object, loosing it from cache will cause a lot of pain when you need it again. Although we don't use QueueCatalog, I can see the advantage of having it concentrate a lot of catalog work in a single thread and transaction. Zope's opportunistic transactions are assuming that a request will complete relatively quickly, and that the likelihood of two entirely separate requests accessing the same object is slim. I like to think of it as the assumption that it is hard for two lightning bolts to hit the same place at the same time. The two ways you can run afoul to this assumption is to either have one object whose modification is greatly favored over others, or requests taking much longer than average. I've had to investigate object hotspots before, and what I've found useful is fsdump.py on an unpacked version of the database. fsdump.py var/storage/var/Data.fs|sed -n 's/.*data #[0-9]*//p'|sort| uniq -c|sort -n then finding particular oids that occur in the fsdump log much more frequently than the rest. Once you've found the hot objects, you can looj back through the fsdump.py log to find the transactions that they belong to and the URL associated with them. Once you've found the code paths that are all modifying the same object, then the changes that need to be done to make the object less hot are application specific. For requests that are taking so long that they are starting to interfere with other requests, they might be able to be found with requestprofiler.py and the ZopeProfiler product. Once they are found, standard code optimization techniques are needed to reduce them. That's about all I can think of writing at the moment, but if you have anything you want to ask me, give me a yell. ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] large installations and conflict errors
--On 8. August 2005 16:01:13 +0200 "M. Krainer" <[EMAIL PROTECTED]> wrote: Hi! We run a self developed zope product originally based on cmf but highly customized and extended. Our Data.fs now has grown to a (packed) size of > 25GB and we are running more and more into performance troubles (mostly due to a lot of conflict errors). Our users make a lot of updates, i.e. we have lots of write operations. Currently we think that our bottle neck is the 'portal_catalog' and I'm working on splitting it up into multiple ZCatalogs. So far our story, but what I really wonder is, if there's anyone out there who has a similar large installation. Please let me know how large your zope instance is and what you have done to increase your (write) performance. Also any ideas that may help us are welcome. Additional info: 4 ZEO clients, 1 ZEO server (all machines have dual P4 and 3-4GB ram), Zope-2.8.0 The CMS that we have built recently on top of the ZODB and CMF consists of several ZEO storages with local catalogs + catalog queue. I don't the total size of all storages in mind but it might be similar the size as your system. However we are using MVCC (implementation by Dieter Maurer) on top of Zope 2.7.something. Since you are using Zope 2.8 should not see much read conflict error for normales storages except for the TemporaryStorage (ChrisM added MVCC support for sessions recently for Zope 2.8.1). -aj pgpFD0lxPXc3X.pgp Description: PGP signature ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] large installations and conflict errors
Do you mount your catalog in a separate database? If not, you may want to try this to split up the conflict domains. Multiple catalogs may or may not help. Multiple databases will help more likely, IMO. Stefan On 8. Aug 2005, at 16:01, M. Krainer wrote: Currently we think that our bottle neck is the 'portal_catalog' and I'm working on splitting it up into multiple ZCatalogs. -- Anything that happens, happens. --Douglas Adams ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] large installations and conflict errors
I can't provide any specific advice but you may want to read the "Scaling Zope" proposal that this points to: http://www.plope.com/Members/chrism/scalingzope/view Note that sessions are a frequent source of conflict errors. If you use them, you may want to upgrade to Zope 2.8.1, which has MVCC for the (eliminates read conflicts) for the storage that backs sessions by default. Write operations are pretty hard to optimize away. One possible optimization if your bottleneck really is the catalog is to use a-QueueCatalog-based portal_catalog instead of a straight-ZCatalog-based one. See http://cvs.zope.org/Products/QueueCatalog/ for more info. - C On Mon, 2005-08-08 at 16:01 +0200, M. Krainer wrote: > Hi! > > We run a self developed zope product originally based on cmf but > highly customized and extended. Our Data.fs now has grown to a > (packed) size of > 25GB and we are running more and more into > performance troubles (mostly due to a lot of conflict errors). Our > users make a lot of updates, i.e. we have lots of write operations. > Currently we think that our bottle neck is the 'portal_catalog' and > I'm working on splitting it up into multiple ZCatalogs. > > So far our story, but what I really wonder is, if there's anyone out > there who has a similar large installation. Please let me know how > large your zope instance is and what you have done to increase your > (write) performance. Also any ideas that may help us are welcome. > > Additional info: > 4 ZEO clients, 1 ZEO server (all machines have dual P4 and 3-4GB ram), > Zope-2.8.0 > > > Thanks, > Markus > ___ > Zope maillist - Zope@zope.org > http://mail.zope.org/mailman/listinfo/zope > ** No cross posts or HTML encoding! ** > (Related lists - > http://mail.zope.org/mailman/listinfo/zope-announce > http://mail.zope.org/mailman/listinfo/zope-dev ) > ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
