Another interesting thing i am seeing is how the heap memory just drops (which I think is due to garbage collection but not certain). If you hit garbage collection manually it will peg the CPU and drop the heap memory to a lower which is why i think it is.
In the attached picture the heap memory takes a *huge* drop all at once which will peg one of the cores at 100% causing clients to timeout for a moment. Is this normal / expected behavior? If this is due to purging values from various caches, wouldn't it be better to incrementally purge? Perhaps this is an effect of the JVM? Suhail On Wed, Feb 3, 2010 at 12:00 PM, Suhail Doshi <suh...@mixpanel.com> wrote: > Something I've been monitoring lately is heap memory because it can cause a > lot of problems if you hit the JVM max memory limit. Usually what happens > when you do is GarbageCollection will run when it hits the max memory and it > will peg one of the cores on the CPU to 100%. That causes clients who are > connecting and actively reading and writing to timeout if you have enough > volume as they appear to be blocked. > > So far I've diagnosed my issues to that and I am thinking about ways to > resolve it. My thought process right now is that KeyCachedFraction and > RowCache values in a ColumnFamily will have a huge impact on this. If > there's anything else, I'd love to know. > > For my needs, our rows are not wide but I have millions, potentially > billions one day. Running on two 8G machines and I am trying to figure out > how to keep the heap memory level and not increasing with time. Any > suggestions or advice would be most helpful. > > Currently I am running an experiment with RowCache set to 0 and > KeyCachedFraction at 0.3. I am assuming KeyCachedFraction will cache up to > 30% of all keys on that given node. If my assumption is right that those two > parameters are the main contributing factors in increased heap memory then > it sounds like you really just have to play with it and monitor it. > > Suhail >
