ParNew GC (used by default in cassandra) uses 'stop-the-world' algorithm, which means your application has to be stopped to do gc. You can run jstat command to monitor gc activity and check if your write performance is related to GC, eg: $ jstat -gc <CASSANDRA_PID> 1s But it shouldn't drop throughtput to 0 ops/s.
Very often (almost always) the first bottleneck is storage. 12341.77us avg latency is quite high. How big your inserts are? Is your disk saturated? What kind of storage do you use? Run iostat to take a look what does your disk do. On Wed, Mar 26, 2014 at 2:54 PM, Jiaan Zeng <l.alle...@gmail.com> wrote: > Hi, > > I am doing some performance benchmarks in a *single* node cassandra > 1.2.4. BTW, the machine is dedicated to run one cassandra instance. > The workload is 100% write. The throughput varies dramatically and > sometimes even drops to 0. I have tried several things below and still > got the same observation. There is no errors in the log file. One > thing I spotted in the log is GCInspector reports GC takes more than > 200 ms. I think that is because the size of the memtable setting. If I > lower the memtable size, that kind of report can go away. Any clues > about what is happening in this case and suggestions about how to > achieve a stable write throughput? Thanks a lot. > > 1) Increase heap size from 4 G to 8 G. The total memory is 16 G. > 2) Increase "memtable_total_space_in_mb" and > "commitlog_total_space_in_mb" to decrease the number of memtable > flush. > 3) Disable the compaction to eliminate the impact of compaction on disk. > > Below is an example of throughput. > 280 sec: 865658 operations; 2661.5 current ops/sec; [INSERT > AverageLatency(us)=3640.16] > 290 sec: 865658 operations; 0 current ops/sec; > 300 sec: 903204 operations; 3754.22 current ops/sec; [INSERT > AverageLatency(us)=12341.77] > > > -- > Regards, > Jiaan >
