[Zope-dev] Alternative to reload()
Sorry for the crosspost, i think it is of interest to both zope-dev and zope3-dev. Guido van Rossum on Edu-sig mailing list just posted some code to reload a module in place, updating classes, methods and functions. [Edu-sig] Reloading code (was Re: OLPC: first thoughts) http://mail.python.org/pipermail/edu-sig/2007-February/007787.html Alternative to reload(). This works by executing the module in a scratch namespace, and then patching classes, methods and functions. This avoids the need to patch instances. New objects are copied into the target namespace. Michael -- http://zope.org/Members/d2m http://planetzope.org ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: Caching ZCatalog results
On Sat, 2007-02-24 at 09:48 +0100, Dieter Maurer wrote: Roché Compaan wrote at 2007-2-23 22:00 +0200: ... Thanks for that pointer. It's good that way, it should make invalidation easier. It could be as simple as invalidating any cached result that contains the documentId being indexed. Do you see any problem with the following invalidation strategy: If the 'documentId' exists (cataloging existing object), invalidate all cached result sets that contain the documentId. If the 'documentId' doesn't exist (cataloging new object), invalidate all result sets where the ids of indexes applied, are contained in the cache key for that result set. I see several problems: * the RAMCacheManager does not provide an API to implement this policy * a cache manager would need a special data structure to efficiently implement the policy (given a documentId, find all cached results containing the documentId). Can you elaborate. Would and IISet be efficient? * Apparently, your cache key contains the indexes involved in producing the result. This is coincidental. I'm building a cache key from all arguments passed in as keyword arguments and on the REQUEST. The problem with this is that these indexes are known only after the query has been performed: The catalog API allows indexes to respond to subqueries, that do not contain their own name. I use this feature to allow a Managable RangeIndex to transparently replace effective, expires queries. But otherwise, the feature is probably not used intensively. If these parameters are on the request or in keywords they will form part of the cache key. Of course, you can add the information *after* the query has been performed and use it for invalidation -- in a specialized cache manager. On the other hand, new objects are usually indexed with all available (and not only a few) indexes. While some of the indexes may not be able to determine a senseful value for the document, the standard indexes have problems to handle this properly (ManagableIndexes can) and the API does not propagate the information. I think it will not be trivial to implement invalidation that doesn't bite you. I thought of checking for document ids because invalidating results when a whole index changes might cause to many invalidations. For example, querying for the same UID of an object should yield a cached result most of the times. Indexing a new object's UID shouldn't invalidate the cached results for existing UID queries. Let's assume we have a specialised cache manager and a cache that copes with the subtleties of sub queries, do think that the invaliding the cache according to the logic I suggested would work? Can you think of cases where it can lead to stale results that one should guard against. -- Roché Compaan Upfront Systems http://www.upfrontsystems.co.za ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: Caching ZCatalog results
On Fri, 2007-02-23 at 21:25 +0100, Lennart Regebro wrote: On 2/23/07, Philipp von Weitershausen [EMAIL PROTECTED] wrote: It may require a bit of hacking the catalog, of course. Perhaps it's time to start thinking about componentizing the Zope 2 catalog to make such things easier in the future? Yup. It would also be interesting to look into making it faster which huge datasets, something that is a problem now. I think it's because you search each index separately and intersect the results, and only then pick out the 20 first results. It is a making it faster urge that led me to thinking about caching results. I'm curious about your use case, the size of your dataset, and how you think Lucene might help you. We have an application that have about a million objects catalogued. With only a few objects in the catalog, a search take about 1 millisecond. This decreases logarithmically to 20 milliseconds for 500 000 objects and about 21 milliseconds for 1 million objects. 20 milliseconds is fast enough for most of our use cases, except for one use case where we add about 100 objects in a single transaction. These objects have Archetype references that lead to a massive amount of catalog queries. To be fair this is an Archetypes problem and not a catalog one, but it did proof to be an interesting optimisation exercise that lead me to thinking about caching ZCatalog results. In this particular case creating 100 objects lead to about 1000 catalog searches taking 20 milliseconds each. That is 20 seconds in total. So given the above, a application with a million objects using the ZCatalog can basically do 50 catalog searches in a second, if it wants to remain responsive to the user. Maybe this is more than enough, I don't know, but with apps like Plone that relies heavily on the catalog, optimisation of catalog operations can surely help improving scalability. -- Roché Compaan Upfront Systems http://www.upfrontsystems.co.za ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
