You node is under significant memory pressure. Look into: * reducing caches * reducing the JVM heap to 8GB
Cheers ----------------- Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 28/03/2012, at 6:03 AM, Feng Qu wrote: > Hi Jonathan, similar problem happens again and there is only one GC running > at that time per system.log. > > This is one node of a 6-node 0.8.6 ring. Heap size on this host is 16GB. > > [fengqu@slcdbx1035 cassandra]$ date; time cnt ring > Tue Mar 27 09:24:20 GMT+7 2012 > Address DC Rack Status State Load Owns > Token > > 141784319550391026443072753096570088106 > 10.89.74.60 slc rack0 Up Normal 343.97 GB 16.67% > 0 > 10.2.128.55 phx rack0 Up Normal 485.48 GB 16.67% > 28356863910078205288614550619314017621 > 10.89.74.67 slc rack0 Up Normal 252.82 GB 16.67% > 56713727820156410577229101238628035242 > 10.2.128.56 phx rack0 Up Normal 258.49 GB 16.67% > 85070591730234615865843651857942052864 > 10.89.74.62 slc rack0 Up Normal 251.02 GB 16.67% > 113427455640312821154458202477256070485 > 10.2.128.57 phx rack0 Up Normal 451.97 GB 16.67% > 141784319550391026443072753096570088106 > > real 0m25.999s > user 0m0.309s > sys 0m0.048s > > WARN [ScheduledTasks:1] 2012-03-27 09:22:39,499 GCInspector.java (line 143) > Heap is 0.8716646568744459 full. You may need to reduce memtable and/or > cache sizes. Cassandra will now flush up to the two largest memtables to > free up memory. Adjust flush_largest_memtables_at threshold in > cassandra.yaml if you don't want Cassandra to do this automatically > INFO [ScheduledTasks:1] 2012-03-27 09:23:21,637 GCInspector.java (line 122) > GC for ConcurrentMarkSweep: 1383 ms for 2 collections, 14756794512 used; max > is 16928210944 > WARN [ScheduledTasks:1] 2012-03-27 09:23:21,637 GCInspector.java (line 143) > Heap is 0.8717279434204102 full. You may need to reduce memtable and/or > cache sizes. Cassandra will now flush up to the two largest memtables to > free up memory. Adjust flush_largest_memtables_at threshold in > cassandra.yaml if you don't want Cassandra to do this automatically > INFO [ScheduledTasks:1] 2012-03-27 09:24:04,844 GCInspector.java (line 122) > GC for ConcurrentMarkSweep: 3090 ms for 2 collections, 14782314472 used; max > is 16928210944 > WARN [ScheduledTasks:1] 2012-03-27 09:24:04,851 GCInspector.java (line 143) > Heap is 0.8732354837083013 full. You may need to reduce memtable and/or > cache sizes. Cassandra will now flush up to the two largest memtables to > free up memory. Adjust flush_largest_memtables_at threshold in > cassandra.yaml if you don't want Cassandra to do this automatically > INFO [ScheduledTasks:1] 2012-03-27 09:24:46,610 GCInspector.java (line 122) > GC for ConcurrentMarkSweep: 3057 ms for 2 collections, 14757982328 used; max > is 16928210944 > WARN [ScheduledTasks:1] 2012-03-27 09:24:46,616 GCInspector.java (line 143) > Heap is 0.871798111260587 full. You may need to reduce memtable and/or cache > sizes. Cassandra will now flush up to the two largest memtables to free up > memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you > don't want Cassandra to do this automatically > INFO [ScheduledTasks:1] 2012-03-27 09:25:36,371 GCInspector.java (line 122) > GC for ConcurrentMarkSweep: 7430 ms for 2 collections, 14719911032 used; max > is 16928210944 > WARN [ScheduledTasks:1] 2012-03-27 09:25:36,377 GCInspector.java (line 143) > Heap is 0.8695491260532345 full. You may need to reduce memtable and/or > cache sizes. Cassandra will now flush up to the two largest memtables to > free up memory. Adjust flush_largest_memtables_at threshold in > cassandra.yaml if you don't want Cassandra to do this automatically > > Feng Qu > From: Jonathan Ellis <jbel...@gmail.com> > To: user <user@cassandra.apache.org> > Cc: Feng Qu <mail...@gmail.com> > Sent: Friday, February 24, 2012 2:29 PM > Subject: Re: nodetool ring runs very slow > > Read the server log and look for GCInspector output. > > On Fri, Feb 24, 2012 at 11:02 AM, Feng Qu <mail...@gmail.com> wrote: > > Hi Jonathan, how to check out whether it's in GC storming? This server > > crashed few time due to Java heap out of memory. We use 8GB heap on a server > > with 96GB ram. This is first node in a 6-node ring and it has opscenter > > community running on it. > > > > Feng Qu > > > > ________________________________ > > From: Jonathan Ellis <jbel...@gmail.com> > > To: user@cassandra.apache.org; Feng Qu <mail...@gmail.com> > > Sent: Thursday, February 23, 2012 1:19 PM > > Subject: Re: nodetool ring runs very slow > > > > The only time I've seen nodetool be that slow is when it was talking > > to a machine that was either swapping or deep into (JVM) GC storming. > > > > On Wed, Feb 22, 2012 at 3:49 PM, Feng Qu <mail...@gmail.com> wrote: > >> We noticed that nodetool ring sometimes returns in 17-20 sec while it > >> normally runs in less than a sec. There were some compaction running when > >> it > >> happened. Did compaction cause nodetool slowness? Anything else I should > >> check? > >>>>> time nodetool -h hostname ring > >> .... > >> real 0m17.595s > >> user 0m0.339s > >> sys 0m0.054s > >> > >> Feng Qu > > > > > > > > -- > > Jonathan Ellis > > Project Chair, Apache Cassandra > > co-founder of DataStax, the source for professional Cassandra support > > http://www.datastax.com > > > > > > > > -- > Jonathan Ellis > Project Chair, Apache Cassandra > co-founder of DataStax, the source for professional Cassandra support > http://www.datastax.com > >
