I have a cassandra 2.0.6 cluster with four nodes as backup database. The only operation is posting data into db. Recently, the full gc of the nodes increases apparently and blocks cluster operation.
The load of each node is 10G. The heap is 8G each with default jvm memory settings. The cpu is 24 cores. The settings of cassandra.yaml are almost default. For debug, all the clients are disconnected, however the gc is still high. The GC is too often that it appears in every minute and blocks for nealy 30s. How to debug this situation The following is the snippets of gc log: before full gc histogram Total Free Space: 0 Max Chunk Size: 0 Number of Blocks: 0 Tree Height: 0 1620.992: [GC 7377342K(8178944K), 0.1380260 secs] Before GC: Statistics for BinaryTreeDictionary: ------------------------------------ Total Free Space: 0 Max Chunk Size: 0 Number of Blocks: 0 Tree Height: 0 Before GC: Statistics for BinaryTreeDictionary: ------------------------------------ Total Free Space: 0 Max Chunk Size: 0 Number of Blocks: 0 Tree Height: 0 ---------------------------------------------- 1: 61030245 1952967840 java.util.concurrent.FutureTask 2: 61031398 1464753552 java.util.concurrent.Executors$RunnableAdapter 3: 61030312 1464727488 java.util.concurrent.LinkedBlockingQueue$Node 4: 61030244 1464725856 org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask 5: 214181 13386992 [B 6: 202890 9738720 java.nio.HeapByteBuffer 7: 37622 5994808 [C 8: 42163 5722096 <constMethodKlass> 9: 42163 5407152 <methodKlass> 10: 4170 4648240 <constantPoolKlass> 11: 100060 4002400 org.apache.cassandra.io.sstable.IndexHelper$IndexInfo 12: 4170 2860816 <instanceKlassKlass> 13: 3664 2702720 <constantPoolCacheKlass> 14: 4817 2701576 [J 15: 2575 1056600 <methodDataKlass> 16: 37431 898344 java.lang.String 17: 2585 627352 [Ljava.lang.Object; 18: 17728 567296 java.util.concurrent.ConcurrentHashMap$HashEntry 19: 22111 530664 javax.management.ObjectName$Property 20: 7595 463824 [S 21: 4534 433560 java.lang.Class 22: 11581 370592 java.util.HashMap$Entry ... Total 289942930 8399196504 1658.022: [Full GC 8178943K->6241829K(8178944K), 27.8562520 secs] CMS: Large block 0x00000007f7da9588 After GC: Statistics for BinaryTreeDictionary: ------------------------------------ Total Free Space: 6335823 Max Chunk Size: 6335823 Number of Blocks: 1 Av. Block Size: 6335823 Tree Height: 1 After GC: Statistics for BinaryTreeDictionary: ------------------------------------ Total Free Space: 0 Max Chunk Size: 0 Number of Blocks: 0 Tree Height: 0 It seems related to objects of Futuretask. -- *陈硕* *Shuo Chen* chenatu2...@gmail.com chens...@whaty.com
