That is consistent with the https://issues.apache.org/jira/browse/CASSANDRA-1169 bug I mentioned, that is fixed in the 0.6 svn branch.
On Wed, Jun 16, 2010 at 10:51 PM, Julie <julie.su...@nextcentury.com> wrote: > Jonathan Ellis <jbellis <at> gmail.com> writes: > >> > Another thing that is odd is that even when the server nodes are quiescent >> > because compacting is complete, I am still seeing cpu usage stay at >> > about 40% . Even after several hours, no reading or writing to the > database >> > and all compactions complete, the cpu usage is staying around 40%. >> >> Possibly this is Hinted Handoff scanning going on. You can rm >> data/system/Hint* (while the node is shut down) if you want to take a >> shot in the dark. Otherwise you'll want to follow >> http://publib.boulder.ibm.com/infocenter/javasdk/tools/index.jsp? > topic=/com.ibm.java.doc.igaa/_1vg0001475cb4a-1190e2e0f74-8000_1007.html >> to figure out which thread is actually consuming the CPU. >> > > Thank you for all of the helpful advice. We upgraded to 0.6.2 to see if any > problems would resolve but we're still seeing the 40% cpu usage on certain > nodes several hours after any reads or writes have taken place. We attached to > one of these super busy cassandra nodes and have isolated it to a single > thread “Thread-32” that seems to be buzzing in a tight loop. > > The loop is in IncomingStreamReader.java, line 62, a 3-line while loop. > bytesRead is not changing. pendingFile.getExpectedBytes() returns > 7,161,538,639 but bytesRead is stuck at 2,147,483,647. > > The stack is: > > Sun.nio.ch.Util.getTemporaryDirectBuffer() – line 50 > Sun.nio.ch.FileChannelImpl.transferFromArbitraryChannel() – line 556 > Sun.nio.ch.FileChannelImpl.transferFrom() – line 603 > Org.apache.cassandra.streaming.IncomingStreamReader.read() – line 62 > Org.apache.cassandra.net.IncomingTcpConnection.run() – line 66 > > This same 3 line loop in IncomingStreamReader was also getting stuck last > night with 0.6.1 so whatever it is is still happening in 0.6.2. > > Thanks for your help, > Julie > > > > -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of Riptano, the source for professional Cassandra support http://riptano.com
