Re: nodetool ring runs very slow
You node is under significant memory pressure. Look into: * reducing caches * reducing the JVM heap to 8GB Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 28/03/2012, at 6:03 AM, Feng Qu wrote: Hi Jonathan, similar problem happens again and there is only one GC running at that time per system.log. This is one node of a 6-node 0.8.6 ring. Heap size on this host is 16GB. [fengqu@slcdbx1035 cassandra]$ date; time cnt ring Tue Mar 27 09:24:20 GMT+7 2012 Address DC RackStatus State LoadOwns Token 141784319550391026443072753096570088106 10.89.74.60 slc rack0 Up Normal 343.97 GB 16.67% 0 10.2.128.55 phx rack0 Up Normal 485.48 GB 16.67% 28356863910078205288614550619314017621 10.89.74.67 slc rack0 Up Normal 252.82 GB 16.67% 56713727820156410577229101238628035242 10.2.128.56 phx rack0 Up Normal 258.49 GB 16.67% 85070591730234615865843651857942052864 10.89.74.62 slc rack0 Up Normal 251.02 GB 16.67% 113427455640312821154458202477256070485 10.2.128.57 phx rack0 Up Normal 451.97 GB 16.67% 141784319550391026443072753096570088106 real0m25.999s user0m0.309s sys 0m0.048s WARN [ScheduledTasks:1] 2012-03-27 09:22:39,499 GCInspector.java (line 143) Heap is 0.8716646568744459 full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically INFO [ScheduledTasks:1] 2012-03-27 09:23:21,637 GCInspector.java (line 122) GC for ConcurrentMarkSweep: 1383 ms for 2 collections, 14756794512 used; max is 16928210944 WARN [ScheduledTasks:1] 2012-03-27 09:23:21,637 GCInspector.java (line 143) Heap is 0.8717279434204102 full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically INFO [ScheduledTasks:1] 2012-03-27 09:24:04,844 GCInspector.java (line 122) GC for ConcurrentMarkSweep: 3090 ms for 2 collections, 14782314472 used; max is 16928210944 WARN [ScheduledTasks:1] 2012-03-27 09:24:04,851 GCInspector.java (line 143) Heap is 0.8732354837083013 full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically INFO [ScheduledTasks:1] 2012-03-27 09:24:46,610 GCInspector.java (line 122) GC for ConcurrentMarkSweep: 3057 ms for 2 collections, 14757982328 used; max is 16928210944 WARN [ScheduledTasks:1] 2012-03-27 09:24:46,616 GCInspector.java (line 143) Heap is 0.871798111260587 full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically INFO [ScheduledTasks:1] 2012-03-27 09:25:36,371 GCInspector.java (line 122) GC for ConcurrentMarkSweep: 7430 ms for 2 collections, 14719911032 used; max is 16928210944 WARN [ScheduledTasks:1] 2012-03-27 09:25:36,377 GCInspector.java (line 143) Heap is 0.8695491260532345 full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically Feng Qu From: Jonathan Ellis jbel...@gmail.com To: user user@cassandra.apache.org Cc: Feng Qu mail...@gmail.com Sent: Friday, February 24, 2012 2:29 PM Subject: Re: nodetool ring runs very slow Read the server log and look for GCInspector output. On Fri, Feb 24, 2012 at 11:02 AM, Feng Qu mail...@gmail.com wrote: Hi Jonathan, how to check out whether it's in GC storming? This server crashed few time due to Java heap out of memory. We use 8GB heap on a server with 96GB ram. This is first node in a 6-node ring and it has opscenter community running on it. Feng Qu From: Jonathan Ellis jbel...@gmail.com To: user@cassandra.apache.org; Feng Qu mail...@gmail.com Sent: Thursday, February 23, 2012 1:19 PM Subject: Re: nodetool ring runs very slow The only time I've seen nodetool be that slow is when
Re: nodetool ring runs very slow
Haven't received any reply. Resend... Feng Qu From: Feng Qu mail...@gmail.com To: Cassandra User Group user@cassandra.apache.org Sent: Wednesday, February 22, 2012 1:49 PM Subject: nodetool ring runs very slow We noticed that nodetool ring sometimes returns in 17-20 sec while it normally runs in less than a sec. There were some compaction running when it happened. Did compaction cause nodetool slowness? Anything else I should check? time nodetool -h hostname ring real 0m17.595s user 0m0.339s sys 0m0.054s Feng Qu
Re: nodetool ring runs very slow
Hi Jonathan, similar problem happens again and there is only one GC running at that time per system.log. This is one node of a 6-node 0.8.6 ring. Heap size on this host is 16GB. [fengqu@slcdbx1035 cassandra]$ date; time cnt ring Tue Mar 27 09:24:20 GMT+7 2012 Address DC Rack Status State Load Owns Token 141784319550391026443072753096570088106 10.89.74.60 slc rack0 Up Normal 343.97 GB 16.67% 0 10.2.128.55 phx rack0 Up Normal 485.48 GB 16.67% 28356863910078205288614550619314017621 10.89.74.67 slc rack0 Up Normal 252.82 GB 16.67% 56713727820156410577229101238628035242 10.2.128.56 phx rack0 Up Normal 258.49 GB 16.67% 85070591730234615865843651857942052864 10.89.74.62 slc rack0 Up Normal 251.02 GB 16.67% 113427455640312821154458202477256070485 10.2.128.57 phx rack0 Up Normal 451.97 GB 16.67% 141784319550391026443072753096570088106 real 0m25.999s user 0m0.309s sys 0m0.048s WARN [ScheduledTasks:1] 2012-03-27 09:22:39,499 GCInspector.java (line 143) Heap is 0.8716646568744459 full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically INFO [ScheduledTasks:1] 2012-03-27 09:23:21,637 GCInspector.java (line 122) GC for ConcurrentMarkSweep: 1383 ms for 2 collections, 14756794512 used; max is 16928210944 WARN [ScheduledTasks:1] 2012-03-27 09:23:21,637 GCInspector.java (line 143) Heap is 0.8717279434204102 full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically INFO [ScheduledTasks:1] 2012-03-27 09:24:04,844 GCInspector.java (line 122) GC for ConcurrentMarkSweep: 3090 ms for 2 collections, 14782314472 used; max is 16928210944 WARN [ScheduledTasks:1] 2012-03-27 09:24:04,851 GCInspector.java (line 143) Heap is 0.8732354837083013 full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically INFO [ScheduledTasks:1] 2012-03-27 09:24:46,610 GCInspector.java (line 122) GC for ConcurrentMarkSweep: 3057 ms for 2 collections, 14757982328 used; max is 16928210944 WARN [ScheduledTasks:1] 2012-03-27 09:24:46,616 GCInspector.java (line 143) Heap is 0.871798111260587 full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically INFO [ScheduledTasks:1] 2012-03-27 09:25:36,371 GCInspector.java (line 122) GC for ConcurrentMarkSweep: 7430 ms for 2 collections, 14719911032 used; max is 16928210944 WARN [ScheduledTasks:1] 2012-03-27 09:25:36,377 GCInspector.java (line 143) Heap is 0.8695491260532345 full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically Feng Qu From: Jonathan Ellis jbel...@gmail.com To: user user@cassandra.apache.org Cc: Feng Qu mail...@gmail.com Sent: Friday, February 24, 2012 2:29 PM Subject: Re: nodetool ring runs very slow Read the server log and look for GCInspector output. On Fri, Feb 24, 2012 at 11:02 AM, Feng Qu mail...@gmail.com wrote: Hi Jonathan, how to check out whether it's in GC storming? This server crashed few time due to Java heap out of memory. We use 8GB heap on a server with 96GB ram. This is first node in a 6-node ring and it has opscenter community running on it. Feng Qu From: Jonathan Ellis jbel...@gmail.com To: user@cassandra.apache.org; Feng Qu mail...@gmail.com Sent: Thursday, February 23, 2012 1:19 PM Subject: Re: nodetool ring runs very slow The only time I've seen nodetool be that slow is when it was talking to a machine that was either swapping or deep into (JVM) GC storming. On Wed, Feb 22, 2012 at 3:49 PM, Feng Qu mail...@gmail.com wrote: We noticed that nodetool ring sometimes returns in 17-20 sec while it normally runs in less than a sec. There were some compaction running when
Re: nodetool ring runs very slow
Read the server log and look for GCInspector output. On Fri, Feb 24, 2012 at 11:02 AM, Feng Qu mail...@gmail.com wrote: Hi Jonathan, how to check out whether it's in GC storming? This server crashed few time due to Java heap out of memory. We use 8GB heap on a server with 96GB ram. This is first node in a 6-node ring and it has opscenter community running on it. Feng Qu From: Jonathan Ellis jbel...@gmail.com To: user@cassandra.apache.org; Feng Qu mail...@gmail.com Sent: Thursday, February 23, 2012 1:19 PM Subject: Re: nodetool ring runs very slow The only time I've seen nodetool be that slow is when it was talking to a machine that was either swapping or deep into (JVM) GC storming. On Wed, Feb 22, 2012 at 3:49 PM, Feng Qu mail...@gmail.com wrote: We noticed that nodetool ring sometimes returns in 17-20 sec while it normally runs in less than a sec. There were some compaction running when it happened. Did compaction cause nodetool slowness? Anything else I should check? time nodetool -h hostname ring real 0m17.595s user 0m0.339s sys 0m0.054s Feng Qu -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com
Re: nodetool ring runs very slow
The only time I've seen nodetool be that slow is when it was talking to a machine that was either swapping or deep into (JVM) GC storming. On Wed, Feb 22, 2012 at 3:49 PM, Feng Qu mail...@gmail.com wrote: We noticed that nodetool ring sometimes returns in 17-20 sec while it normally runs in less than a sec. There were some compaction running when it happened. Did compaction cause nodetool slowness? Anything else I should check? time nodetool -h hostname ring real 0m17.595s user 0m0.339s sys 0m0.054s Feng Qu -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com
