On Wed, Aug 5, 2009 at 1:27 PM, James Henstridge<[email protected]> wrote:
On Thu, Jul 30, 2009 at 12:05 PM, Stuart Bishop<[email protected]> wrote:On Thu, Jul 30, 2009 at 1:29 AM, Gustavo Niemeyer<[email protected]> wrote:Generally, trying to manage memory yourself inside an application can work against you. See the ArchitectNotes for Varnish for a general overview of why and how:Yeah, I'm kind of concerned about it too, since there are so many details in the way (Python's internal memory buffers, C library allocation behavior, the application's use of objects, etc). Promising an auto-tweaking cache which behaves poorly would be worse than advising developers to tweak their cache size to fit their needs.I agree that auto-tweaking caches that behave poorly are not a good idea :-) I'm fishing for ideas on how to get one that doesn't behave poorly, or at least better than my first experiment. It may be impossible.Perhaps it would be possible to implement some kind of profiling cache implementation would be worthwhile then: something that would record cache hits and misses, and some utilities to analyse that data.
Hmm... yes. That might be a better approach.
I do feel that if I had a half decent implementation and could tell a program 'Use up to 2GB of RAM', that would be useful for almost all of the code we deploy. Our current mechanism is to make an educated guess as to the cache size and forget about it unless performance or RAM consumption is an issue (multiple bits of code sharing a server means swapping is bad). Even if went to the trouble of calculating the ideal cache size, soon it would no longer be ideal as our data is constantly changing - the optimal cache size for importing a pofile with small translations is different to the optimal cache size for importing a pofile with lengthy translations because the objects are radically different in size.Note that a larger cache isn't always going to be better: some operations scan over the list of alive objects (e.g ResultSet.set), so keeping more objects alive could slow those operations down. If you end up caching objects that never get used again, then you'd probably be better off with a smaller cache.
I wasn't aware of that. That would certainly screw up using an external cache like memcached.
And as I said in the previous email, a moderately complex Storm application will likely have many caches, so having each cache make decisions independently will likely give poor results (i.e. one cache ends up really large and the others really small).
Yes - it needs to be treated as a single big cache. My first attempt did this by bumping all caches (not just the one that triggered the event). A profiling approach would be able to profile the individual caches though, so a Store with a high hit rate gets a larger cache than the Store with a terrible hit rate. -- Stuart Bishop <[email protected]> http://www.stuartbishop.net/
signature.asc
Description: OpenPGP digital signature
-- storm mailing list [email protected] Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/storm
