6s parnew is insane. you're probably swapping. Easiest fix is disabling swap entirely.
P.S. 0.6.3 is ancient. On Thu, May 5, 2011 at 10:51 AM, Ali Ahsan <ali.ah...@panasiangroup.com> wrote: > Hi All > > I have two cassandra node with RF=2,Cassandra start under performing i mean > red write are slow and i see following info in cassandra log.I have centos > 5.5 64 bit with 14 GB memory assigned to cassandra.I am using LVM for > cassandra.When i reboot my cassanrda every thing become normal. > > Below is some nodetoll info > > > # /root/cassandra/bin/nodetool -h 10.0.0.4 info > 222289592293417191096473651402050455 > Load : 43.51 GB > Generation No : 1304495724 > Uptime (seconds) : 114699 > Heap Memory (MB) : 9944.43 / 14319.38 > > > tpstats > > [root@cassandra2 cassandra]# /root/cassandra/bin/nodetool -h 10.0.0.4 > tpstats > Pool Name Active Pending Completed > FILEUTILS-DELETE-POOL 0 0 187 > STREAM-STAGE 0 0 0 > RESPONSE-STAGE 0 0 35901940 > ROW-READ-STAGE 8 9 14580549 > LB-OPERATIONS 0 0 0 > MESSAGE-DESERIALIZER-POOL 0 0 54586854 > GMFD 0 0 308503 > LB-TARGET 0 0 0 > CONSISTENCY-MANAGER 0 0 7637656 > ROW-MUTATION-STAGE 2 2 43329077 > MESSAGE-STREAMING-POOL 0 0 0 > LOAD-BALANCER-STAGE 0 0 0 > FLUSH-SORTER-POOL 0 0 0 > MEMTABLE-POST-FLUSHER 0 0 2028 > FLUSH-WRITER-POOL 0 0 2028 > AE-SERVICE-STAGE 0 0 0 > HINTED-HANDOFF-POOL 1 30 97 > > > > When i see below info cassandra start under performing and happening on both > machines. > > > INFO [GC inspection] 2011-05-05 20:39:08,617 GCInspector.java (line 110) GC > for ParNew: 698 ms, 138706360 reclaimed leaving 10135848048 used; max is > 15241445376 > INFO [GC inspection] 2011-05-05 20:39:09,620 GCInspector.java (line 110) GC > for ParNew: 496 ms, 138960224 reclaimed leaving 10135360416 used; max is > 15241445376 > INFO [GC inspection] 2011-05-05 20:39:10,988 GCInspector.java (line 110) GC > for ParNew: 1051 ms, 137654632 reclaimed leaving 10137177208 used; max is > 15241445376 > INFO [GC inspection] 2011-05-05 20:39:13,370 GCInspector.java (line 110) GC > for ParNew: 907 ms, 137254552 reclaimed leaving 10137977152 used; max is > 15241445376 > INFO [GC inspection] 2011-05-05 20:39:14,631 GCInspector.java (line 110) GC > for ParNew: 635 ms, 135828080 reclaimed leaving 10140229096 used; max is > 15241445376 > INFO [GC inspection] 2011-05-05 20:39:16,872 GCInspector.java (line 110) GC > for ParNew: 1910 ms, 137722664 reclaimed leaving 10141548552 used; max is > 15241445376 > INFO [GC inspection] 2011-05-05 20:39:23,463 GCInspector.java (line 110) GC > for ParNew: 6341 ms, 139920392 reclaimed leaving 10141215712 used; max is > 15241445376 > INFO [GC inspection] 2011-05-05 20:39:25,482 GCInspector.java (line 110) GC > for ParNew: 1712 ms, 139482032 reclaimed leaving 10141321232 used; max is > 15241445376 > > > -- > S.Ali Ahsan > > Senior System Engineer > > e-Business (Pvt) Ltd > > 49-C Jail Road, Lahore, P.O. Box 676 > Lahore 54000, Pakistan > > Tel: +92 (0)42 3758 7140 Ext. 128 > > Mobile: +92 (0)345 831 8769 > > Fax: +92 (0)42 3758 0027 > > Email: ali.ah...@panasiangroup.com > > > > www.ebusiness-pg.com > > www.panasiangroup.com > > Confidentiality: This e-mail and any attachments may be confidential > and/or privileged. If you are not a named recipient, please notify the > sender immediately and do not disclose the contents to another person > use it for any purpose or store or copy the information in any medium. > Internet communications cannot be guaranteed to be timely, secure, error > or virus-free. We do not accept liability for any errors or omissions. > > -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com
