Re: Out of control memory consumption

2011-02-11 Thread Robert Coli
On Thu, Feb 10, 2011 at 9:17 AM, Huy Le hu...@springpartners.com wrote: Yes, we had setting at 75 but JVM did not have enough time to do GC, so it abort GC'ing.   We lowered it to 50, but still had issue, so we lowered it again to 35. If you lower CMSInitiatingOccupancyFraction below the size

Re: Out of control memory consumption

2011-02-10 Thread Oleg Anastasyev
Huy Le huyle at springpartners.com writes: Our CMS settings are:    -XX:CMSInitiatingOccupancyFraction=35 \    -XX:+UseCMSInitiatingOccupancyOnly \  Occupancy Fraction = 35 seems very low value. You instructed GC to make collection as soon as memory usage is at 35% - i.e. about 1G.

Re: Out of control memory consumption

2011-02-10 Thread Huy Le
Yes, we had setting at 75 but JVM did not have enough time to do GC, so it abort GC'ing. We lowered it to 50, but still had issue, so we lowered it again to 35. On Thu, Feb 10, 2011 at 12:11 PM, Oleg Anastasyev olega...@gmail.comwrote: Huy Le huyle at springpartners.com writes: Our CMS

Re: Out of control memory consumption

2011-02-10 Thread Jonathan Ellis
Sounds like you need to use a larger heap or put less stuff in it (memtables, caches). On Thu, Feb 10, 2011 at 11:17 AM, Huy Le hu...@springpartners.com wrote: Yes, we had setting at 75 but JVM did not have enough time to do GC, so it abort GC'ing.   We lowered it to 50, but still had issue, so

Out of control memory consumption

2011-02-09 Thread Huy Le
Hi, There is already an email thread on memory issue on this email list, but I creating a new thread as we are experiencing a different memory consumption issue. We are 12-server cluster. We use random partitioner with manually generated server tokens. Memory usage on one server keeps growing

Re: Out of control memory consumption

2011-02-09 Thread Chris Burroughs
On 02/09/2011 11:15 AM, Huy Le wrote: There is already an email thread on memory issue on this email list, but I creating a new thread as we are experiencing a different memory consumption issue. We are 12-server cluster. We use random partitioner with manually generated server tokens.

Re: Out of control memory consumption

2011-02-09 Thread Peter Schuller
We are 12-server cluster.  We use random partitioner with manually generated server tokens.  Memory usage on one server keeps growing out of control.  We ran flush and cleared key and row caches but and ran GC but heap memory usage won't go down.  The only way to heap memory usage to go down

Re: Out of control memory consumption

2011-02-09 Thread Peter Schuller
(If you're looking at e.g. jconsole graphs a screenshot of the graph would not hurt.) -- / Peter Schuller

Re: Out of control memory consumption

2011-02-09 Thread Huy Le
If the heap usages continues to grow an OOM will eventually be thrown. Are you experiencing OOMs on these boxes? If you are not OOMing, then what problem are you experiencing (excessive CPU use garbage collection for one example)? No OOM. The JVM just too busy doing GC when the used heap

Re: Out of control memory consumption

2011-02-09 Thread Huy Le
To be clear: You are not talking about the size of the Java process in top, but the actual amount of heap used as reported by the JVM via jmx/jconsole/etc? This is memory usage shows in JMX that we are talking about. Is the memory amount of memory that you consider high, the heap size

Re: Out of control memory consumption

2011-02-09 Thread Robert Coli
On Wed, Feb 9, 2011 at 11:04 AM, Huy Le hu...@springpartners.com wrote: Memory usage grows overtime. It is relatively typical for caches to exert memory pressure over time as they fill. What are your cache settings, for how many columnfamilies, and with what sized memtables? What version of