RE: Garbage collection freezes cassandra node

2012-01-20 Thread Rene Kochen
:23 To: user@cassandra.apache.org Subject: Re: Garbage collection freezes cassandra node On node 172.16.107.46, I see the following: 21:53:27.192+0100: 1335393.834: [GC 1335393.834: [ParNew (promotion failed): 319468K-324959K(345024K), 0.1304456 secs]1335393.964: [CMS: 6000844K-3298251K

Re: Garbage collection freezes cassandra node

2012-01-20 Thread Peter Schuller
Thanks for this very helpful info. It is indeed a production site which I cannot easily upgrade. I will try the various gc knobs and post any positive results. *IF* your data size, or at least hot set, is small enough that you're not extremely reliant on the current size of page cache, and

RE: Garbage collection freezes cassandra node

2012-01-19 Thread Rene Kochen
to decrease the CMSInitiatingOccupancyFraction setting? Thanks! Rene -Original Message- From: sc...@scode.org [mailto:sc...@scode.org] On Behalf Of Peter Schuller Sent: dinsdag 20 december 2011 6:38 To: user@cassandra.apache.org Subject: Re: Garbage collection freezes cassandra node I

Re: Garbage collection freezes cassandra node

2012-01-19 Thread Mohit Anchlia
the CMSInitiatingOccupancyFraction setting? Thanks! Rene -Original Message- From: sc...@scode.org [mailto:sc...@scode.org] On Behalf Of Peter Schuller Sent: dinsdag 20 december 2011 6:38 To: user@cassandra.apache.org Subject: Re: Garbage collection freezes cassandra node I should add: If you

Re: Garbage collection freezes cassandra node

2012-01-19 Thread Peter Schuller
On node 172.16.107.46, I see the following: 21:53:27.192+0100: 1335393.834: [GC 1335393.834: [ParNew (promotion failed): 319468K-324959K(345024K), 0.1304456 secs]1335393.964: [CMS: 6000844K-3298251K(8005248K), 10.8526193 secs] 6310427K-3298251K(8350272K), [CMS Perm :

RE: Garbage collection freezes cassandra node

2011-12-23 Thread Rene Kochen
] On Behalf Of Peter Schuller Sent: dinsdag 20 december 2011 6:36 To: user@cassandra.apache.org Subject: Re: Garbage collection freezes cassandra node During the garbage collections, Cassandra freezes for about ten seconds. I observe the following log entries: “GC for ConcurrentMarkSweep

AW: Garbage collection freezes cassandra node

2011-12-19 Thread Roland Gude
2011 16:35 An: user@cassandra.apache.org Betreff: Garbage collection freezes cassandra node I recently see the following garbage collect behavior in our performance tests (the attached chart shows the heap-size in MB): [cid:image001.jpg@01CCBE73.8B567F80] During the garbage collections, Cassandra

Re: Garbage collection freezes cassandra node

2011-12-19 Thread Mohit Anchlia
Increasing memory in this case may not solve the problem. Share some information about your workload. Cluster configuration, cache sizes etc. You can also try getting java heap historgram to get more info on what's on the heap. On Mon, Dec 19, 2011 at 7:35 AM, Rene Kochen

Re: Garbage collection freezes cassandra node

2011-12-19 Thread Peter Schuller
During the garbage collections, Cassandra freezes for about ten seconds. I observe the following log entries: “GC for ConcurrentMarkSweep: 11597 ms for 1 collections, 1887933144 used; max is 8550678528” Ok, first off: Are you certain that it is actually pausing, or are you assuming that

Re: Garbage collection freezes cassandra node

2011-12-19 Thread Peter Schuller
I should add: If you are indeed actually pausing due to promotion failed or concurrent mode failure (which you will see in the GC log if you enable it with the options I suggested), the first thing I would try to mitigate is: * Decrease the occupancy trigger (search for occupancy) of CMS to a