On Wed, Aug 11, 2010 at 4:27 PM, Michael Hudson <[email protected]> wrote: > On Wed, 11 Aug 2010 15:02:13 +1200, Robert Collins > <[email protected]> wrote: > >> The latter are trickier because we want to get all the related data we >> want at once, no more than needed, and no less. If we get more things >> go slower, and if we get less we scale poorly as the page gets busier >> (e.g. more builds). The pattern I've tried in registry with >> _all_members should help considerably, once we get the cache coherency >> stuff sorted (which strictly speaking only really affects the test >> suite : production clears storm caches between requests, and so caches >> on model objects have no lifetime in the actual server). > > What's the issue here, generally speaking?
I don't know enough to speak generally. I can tell you about a specific problem. > It's possible to arrange for > caches to be cleared on many boundaries -- for example, the per-request > security policy cache is cleared when the principal changes, which > happens during login. Clearing all @cachedpropertys like this might not > scale very well, but otoh there might be something we can do. Sure. So there are two caches per model object [at a baseline level] *) cachedproperty. This is stored in the model object, as _attributename_cached attributes. *) storm data to determine changes made when flushing a change. And there is a cache /of/ model objects: the storm per-thread cache. Cache invalidation for the first: doesn't exist, unless written adhoc & manually. Cache invalidation for the second is automatic on flush and attribute reset when a query hits the db. Cache invalidation for the third happens: - in production at the end of every request: we throw away the entire cache - in testing, never. This means that things like the cachedproperties I added blow up in testing a lot, and rarely-if-ever in production. We can use __storm_invalidation__ or whatever the hook is to tell storm to invalidate the cached properties, and I'll do that : what I'd like to do long term is: - remove all manual caching - remove the entire overhead of storm caching - *quite a lot* of code runs on every setattr & on so on. It does have a C accelerator but it is 99% of the time waste: we are write-light, read-heavy. - discard all objects at the end of every transaction - explicitly track updated fields, and only updated fields for updates. -Rob _______________________________________________ Mailing list: https://launchpad.net/~launchpad-dev Post to : [email protected] Unsubscribe : https://launchpad.net/~launchpad-dev More help : https://help.launchpad.net/ListHelp
