Re: [Zope-dev] ZCatalog caching with memcached
The usual Plone catalogs (portal_catalog, uid_catalog, reference_catalog and membrane_tool) all run above 90% hit rate if the server is up to it. portal_catalog is invalidated the most so it fluctuates the most. If the server is severely underpowered then catalogcache is much less effective. portal_catalog hit rates will degrade over time. This is the situation I'm currently facing with on one site, but more servers will fix that. It's quite easy to benchmark / load test with funkload. What I've found is that memcached is very light on CPU, but if the Zope processes are constantly using all CPU it is starved and runs into trouble. As long as you avoid that case (which would be fatal without catalogcache in any case) then everything works perfectly. Run a few tests and let me know please. Hedley ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
On 25.10.2008 8:48 Uhr, Hedley Roos wrote: The usual Plone catalogs (portal_catalog, uid_catalog, reference_catalog and membrane_tool) all run above 90% hit rate if the server is up to it. portal_catalog is invalidated the most so it fluctuates the most. If the server is severely underpowered then catalogcache is much less effective. portal_catalog hit rates will degrade over time. This is the situation I'm currently facing with on one site, but more servers will fix that. It's quite easy to benchmark / load test with funkload. What I've found is that memcached is very light on CPU, but if the Zope processes are constantly using all CPU it is starved and runs into trouble. As long as you avoid that case (which would be fatal without catalogcache in any case) then everything works perfectly. Have you measures the time needs for some standard ZCatalog queries used with a Plone site with the communication overhead with memcached? Generally spoken: I think the ZCatalog is in general fast. Queries using a fulltext index are known to be more expensive or if you have to deal with large resultsets or complex queries. Andreas begin:vcard fn:Andreas Jung n:Jung;Andreas org:ZOPYX Ltd. Co. KG adr;quoted-printable:;;Charlottenstr. 37/1;T=C3=BCbingen;;72070;Germany email;internet:[EMAIL PROTECTED] title:CEO tel;work:+49-7071-793376 tel;fax:+49-7071-7936840 tel;home:+49-7071-793257 x-mozilla-html:FALSE url:www.zopyx.com version:2.1 end:vcard ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
Have you measures the time needs for some standard ZCatalog queries used with a Plone site with the communication overhead with memcached? Generally spoken: I think the ZCatalog is in general fast. Queries using a fulltext index are known to be more expensive or if you have to deal with large resultsets or complex queries. No I haven't. Roche Compaan has done extensive benchmarking using funkload testing plain catalog vs module level cache vs memcached, but the tests are more about page serving than catalog query time. I'll ask him to comment more on that. As for standard queries on a Plone site the typical folder contents query is a good example. The query will be fast unless it sorts on sortable_title (a ZCTextIndex) right? Not sure right now. Since memcached is distributed only a single Zope client needs to perform that query and the result is available to all other Zope clients. And the cache is persistent as long as memcached runs, so you can merrily restart Zope instances and have a warm cache. I didn't even realise this until Roche pointed it out to me. To answer the question: I believe catalogcache will win every time since the return time of a cached query is not dependent on the complexity of the query. We should get a few benchmarks running at query level. I'll have a bit of time next week. Hedley ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Zope Tests: 5 OK
Summary of messages to the zope-tests list. Period Fri Oct 24 11:00:00 2008 UTC to Sat Oct 25 11:00:00 2008 UTC. There were 5 messages: 5 from Zope Tests. Tests passed OK --- Subject: OK : Zope-2.8 Python-2.3.6 : Linux From: Zope Tests Date: Fri Oct 24 20:56:52 EDT 2008 URL: http://mail.zope.org/pipermail/zope-tests/2008-October/010361.html Subject: OK : Zope-2.9 Python-2.4.4 : Linux From: Zope Tests Date: Fri Oct 24 20:58:23 EDT 2008 URL: http://mail.zope.org/pipermail/zope-tests/2008-October/010362.html Subject: OK : Zope-2.10 Python-2.4.4 : Linux From: Zope Tests Date: Fri Oct 24 20:59:53 EDT 2008 URL: http://mail.zope.org/pipermail/zope-tests/2008-October/010363.html Subject: OK : Zope-2.11 Python-2.4.4 : Linux From: Zope Tests Date: Fri Oct 24 21:01:23 EDT 2008 URL: http://mail.zope.org/pipermail/zope-tests/2008-October/010364.html Subject: OK : Zope-trunk Python-2.4.4 : Linux From: Zope Tests Date: Fri Oct 24 21:02:53 EDT 2008 URL: http://mail.zope.org/pipermail/zope-tests/2008-October/010365.html ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
Hedley Roos wrote: As for standard queries on a Plone site the typical folder contents query is a good example. The query will be fast unless it sorts on sortable_title (a ZCTextIndex) right? Not sure right now. sortable_title is a field index and shouldn't be slower than any other index. This all sounds very cool, by the way. :) Martin -- Author of `Professional Plone Development`, a book for developers who want to work with Plone. See http://martinaspeli.net/plone-book ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
Hi, On Fri, 2008-10-24 at 15:41 +0200, Hedley Roos wrote: The product is a monkey patch to Catalog.py. I'd love some feedback and suggestions. I'd love if this wouldn't be a monkey patch. Also, there is nothing that makes this integrate correctly with transactions. Your cache will happily deliver never-committed data and also it will not isolate transactions from each other. Christian -- Christian Theune · [EMAIL PROTECTED] gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development signature.asc Description: This is a digitally signed message part ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
I'd love if this wouldn't be a monkey patch. So would I, but I couldn't find another way in this case. Also, there is nothing that makes this integrate correctly with transactions. Your cache will happily deliver never-committed data and also it will not isolate transactions from each other. I patched 4 methods - clear, search, catalogObject, uncatalogObject. Method clear is the simplest one - I simply flush the cache. Methods catalogObject and uncatalogObject both invalidate the cache. Should the transaction fail later the only drawback is that you threw a few things out of the cache. They'll soon be re-entered by subsequent searches. Method search just inspects queries and stores results to memcache. Can you give me an example where the cache would deliver non-committed data? ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
In addition, you need to include a serial in your cache keys to avoid dirty reads. The cache invalidation code actively removes items from the cache. Am I understanding you correctly? H ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
On 25.10.2008 14:53 Uhr, Hedley Roos wrote: I'd love if this wouldn't be a monkey patch. So would I, but I couldn't find another way in this case. Also, there is nothing that makes this integrate correctly with transactions. Your cache will happily deliver never-committed data and also it will not isolate transactions from each other. I patched 4 methods - clear, search, catalogObject, uncatalogObject. Method clear is the simplest one - I simply flush the cache. Methods catalogObject and uncatalogObject both invalidate the cache. Should the transaction fail later the only drawback is that you threw a few things out of the cache. They'll soon be re-entered by subsequent searches. Using a DataManager is likely the better and more safe choice. Andreas begin:vcard fn:Andreas Jung n:Jung;Andreas org:ZOPYX Ltd. Co. KG adr;quoted-printable:;;Charlottenstr. 37/1;T=C3=BCbingen;;72070;Germany email;internet:[EMAIL PROTECTED] title:CEO tel;work:+49-7071-793376 tel;fax:+49-7071-7936840 tel;home:+49-7071-793257 x-mozilla-html:FALSE url:www.zopyx.com version:2.1 end:vcard ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
On Sat, 2008-10-25 at 14:53 +0200, Hedley Roos wrote: I'd love if this wouldn't be a monkey patch. So would I, but I couldn't find another way in this case. Also, there is nothing that makes this integrate correctly with transactions. Your cache will happily deliver never-committed data and also it will not isolate transactions from each other. I patched 4 methods - clear, search, catalogObject, uncatalogObject. Method clear is the simplest one - I simply flush the cache. This is probably harmless but will cause unnecessary cache flushes for other clients. Methods catalogObject and uncatalogObject both invalidate the cache. Should the transaction fail later the only drawback is that you threw a few things out of the cache. They'll soon be re-entered by subsequent searches. Right. This is the same as clear. Method search just inspects queries and stores results to memcache. That's the issue. If you catalog an object, then search for it and then abort the transaction, your cache will have data in it that isn't committed. Additionally when another transaction is already running in parallel, it will see cache inserts from other transactions. Christian -- Christian Theune · [EMAIL PROTECTED] gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development signature.asc Description: This is a digitally signed message part ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
On Sat, Oct 25, 2008 at 2:57 PM, Andreas Jung [EMAIL PROTECTED] wrote: On 25.10.2008 14:53 Uhr, Hedley Roos wrote: I'd love if this wouldn't be a monkey patch. So would I, but I couldn't find another way in this case. Also, there is nothing that makes this integrate correctly with transactions. Your cache will happily deliver never-committed data and also it will not isolate transactions from each other. I patched 4 methods - clear, search, catalogObject, uncatalogObject. Method clear is the simplest one - I simply flush the cache. Methods catalogObject and uncatalogObject both invalidate the cache. Should the transaction fail later the only drawback is that you threw a few things out of the cache. They'll soon be re-entered by subsequent searches. Using a DataManager is likely the better and more safe choice. Andreas Thanks Andreas. I'll have a look at your code when available. Christian, I do have a mistake in my reasoning. If an object is added to the catalog in a transaction and I cache that object as result of a query in that same transaction, and then the transaction fails I'll have a bad cache. H ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
If you catalog an object, then search for it and then abort the transaction, your cache will have data in it that isn't committed. Kind of like how I came to the same conclusion in parallel to you and stuffed up this thread :) Additionally when another transaction is already running in parallel, it will see cache inserts from other transactions. So this is the area I have to focus on right now. H ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
Additionally when another transaction is already running in parallel, it will see cache inserts from other transactions. A possible solution is to keep a module level cache which can be committed to the memcache on transaction boundaries. That way I'll incur no performance penalty. H ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
On Sat, 2008-10-25 at 14:55 +0200, Hedley Roos wrote: In addition, you need to include a serial in your cache keys to avoid dirty reads. The cache invalidation code actively removes items from the cache. Am I understanding you correctly? I wasn't even talking about invalidation as your cache wouldn't see 'invalidations' anyways. It's memcached's task to forget stuff: it's a cache anyway. -- Christian Theune · [EMAIL PROTECTED] gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development signature.asc Description: This is a digitally signed message part ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
