FYI. after upgrading jdk from 1.6.0_16 to 1.6.0_20, the problem solved.
On Fri, Apr 16, 2010 at 00:33, Ingram Chen <[email protected]> wrote: > Hi all, > > We setup two nodes and simply set replication factor=2 for test run. > > After both nodes, say, node A and node B, serve several hours, we found > that "node A" always keep 300% cpu usage. > (the other node is under 100% cpu, which is normal) > > thread dump on "node A" shows that there are 3 busy threads related to > IncomingStreamReader: > > ========================== > > "Thread-66" prio=10 tid=0x00002aade4018800 nid=0x69e7 runnable > [0x000000004030a000] > java.lang.Thread.State: RUNNABLE > at sun.misc.Unsafe.setMemory(Native Method) > at sun.nio.ch.Util.erase(Util.java:202) > at > sun.nio.ch.FileChannelImpl.transferFromArbitraryChannel(FileChannelImpl.java:560) > at > sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:603) > at > org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:62) > at > org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66) > > "Thread-65" prio=10 tid=0x00002aade4017000 nid=0x69e6 runnable > [0x000000004d44b000] > java.lang.Thread.State: RUNNABLE > at sun.misc.Unsafe.setMemory(Native Method) > at sun.nio.ch.Util.erase(Util.java:202) > at > sun.nio.ch.FileChannelImpl.transferFromArbitraryChannel(FileChannelImpl.java:560) > at > sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:603) > at > org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:62) > at > org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66) > > "Thread-62" prio=10 tid=0x00002aade4014800 nid=0x4150 runnable > [0x000000004d34a000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.FileChannelImpl.size0(Native Method) > at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:309) > - locked <0x00002aaac450dcd0> (a java.lang.Object) > at > sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:597) > at > org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:62) > at > org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66) > > =========================== > > Is there anyone experience similar issue ? > > environments: > > OS --- CentOS 5.4, Linux 2.6.18-164.15.1.el5 SMP x86_64 GNU/Linux > Java --- build 1.6.0_16-b01, Java HotSpot(TM) 64-Bit Server VM (build > 14.2-b01, mixed mode) > Cassandra --- 0.6.0 > Node configuration --- node A and node B. both nodes use node A as Seed > client --- Java thrift clients pick one node randomly to do read and write. > > > -- > Ingram Chen > online share order: http://dinbendon.net > blog: http://www.javaworld.com.tw/roller/page/ingramchen > -- Ingram Chen online share order: http://dinbendon.net blog: http://www.javaworld.com.tw/roller/page/ingramchen
