Greetings, I would like to check my understanding is accurate on how KeysCached is understood by Cassandra (0.6.5), and then get some suggestions on settings / OS FS cache interplay.
First - my initial understanding was that if you set KeysCached to 100%, Cassandra would do a best effort to keep all keys in memory on all boxes in the cluster - but it would let some expire if it was going to run out of memory. *Now* I understand that it actually just exhausts all its memory and falls over if you set this value unrealistically high. Is this right? Assuming this is right ... I'm curious on the optimum settings for KeysCached. Previously we've had problems here with boxes doing massive STW GC's, among other things - and we've been working towards a configuration that is a bit more stable and predictable, if not as blazingly fast as a brand new cluster unburdened by much data but with a configuration that means it'll drive itself into oblivion after several days. ;) With that in mind, we're trying to keep JVM heap small - about 3GB is our sweet spot so far - after experimenting with numbers as big as 30GB. Even at 30GB we'd be unable to have 100% KeysCached anyway - so it was always going to require a tradeoff - and now we're trying to guess at the best number. We're using boxes with 16GB, and have an expectation that the OS will cache a lot of this stuff pretty intelligently - but I fear that compactions and other activities will mean that the keys won't always have priority for FS cache. Short of catting the index files to /dev/null periodically (not a serious suggestion) has anyone gleaned some insights on the best tradeoffs for KeysCached where the raw size of your Keys is going to be at least 10x the size of your available memory? Do you go small and hope the OS saves you, or do you try to go as big (but finite) as your JVM will usefully let you and hope Cassandra caches the best set of keys for your usage profile? taa, Jedd.
