What’s your cfhistograms look like? From: Ruchir Jha <ruchir....@gmail.com<mailto:ruchir....@gmail.com>> Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>> Date: Monday, July 28, 2014 at 10:43 AM To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>> Subject: Re: Full GC in cassandra
Also we do subsequent updates (atleat 4) for each piece of data that we write. On Mon, Jul 28, 2014 at 10:36 AM, Ruchir Jha <ruchir....@gmail.com<mailto:ruchir....@gmail.com>> wrote: Doing about 5K writes / second. Avg Data Size = 1.6 TB / node. Total Data Size = 21 TB. And this is the nodetool cfstats output for one of our busiest column families: SSTable count: 10 Space used (live): 43239294899 Space used (total): 43239419603 SSTable Compression Ratio: 0.2954468408497778 Number of Keys (estimate): 63729152 Memtable Columns Count: 1921620 Memtable Data Size: 257680020 Memtable Switch Count: 9 Read Count: 6167 Read Latency: NaN ms. Write Count: 770984 Write Latency: 0.098 ms. Pending Tasks: 0 Bloom Filter False Positives: 370 Bloom Filter False Ratio: 0.00000 Bloom Filter Space Used: 80103200 Compacted row minimum size: 180 Compacted row maximum size: 3311 Compacted row mean size: 2631 Average live cells per slice (last five minutes): 73.0 Average tombstones per slice (last five minutes): 13.0 On Mon, Jul 28, 2014 at 10:14 AM, Mark Reddy <mark.re...@boxever.com<mailto:mark.re...@boxever.com>> wrote: What is your data size and number of columns in Cassandra. Do you do many deletions? On Mon, Jul 28, 2014 at 2:53 PM, Ruchir Jha <ruchir....@gmail.com<mailto:ruchir....@gmail.com>> wrote: Really curious to know what's causing the spike in Columns and DeletedColums below : 2014-07-28T09:30:27.471-0400: 127335.928: [Full GC 127335.928: [Class Histogram: num #instances #bytes class name ---------------------------------------------- 1: 132626060 6366050880<tel:6366050880> java.nio.HeapByteBuffer 2: 28194918 3920045528 [B 3: 78124737 3749987376 edu.stanford.ppl.concurrent.SnapTreeMap$Node 4: 67650128 2164804096<tel:2164804096> org.apache.cassandra.db.Column 5: 16315310 522089920 org.apache.cassandra.db.DeletedColumn 6: 6818 392489608 [I 7: 2844374 273059904 edu.stanford.ppl.concurrent.CopyOnWriteManager$COWEpoch 8: 5727000 229080000 java.util.TreeMap$Entry 9: 767742 182921376 [J 10: 2932832 140775936 edu.stanford.ppl.concurrent.SnapTreeMap$RootHolder 11: 2844375 91020000 edu.stanford.ppl.concurrent.CopyOnWriteManager$Latch 12: 4145131 66322096 java.util.concurrent.atomic.AtomicReference 13: 437874 64072392 [C 14: 2660844 63860256 java.util.concurrent.ConcurrentSkipListMap$Node 15: 4920 62849864 [[B 16: 1632063 52226016 edu.stanford.ppl.concurrent.SnapTreeMap
