There is always this old chestnut http://wiki.apache.org/cassandra/FAQ#ubuntu_hangs
A ----------------- Aaron Morton Freelance Cassandra Developer New Zealand @aaronmorton http://www.thelastpickle.com On 16/02/2013, at 8:22 AM, Edward Capriolo <edlinuxg...@gmail.com> wrote: > With hyper threading a core can show up as two or maybe even four > physical system processors, this is something the kernel does. > > On Fri, Feb 15, 2013 at 11:41 AM, Hiller, Dean <dean.hil...@nrel.gov> wrote: >> We ran into an issue today where website became around 10 times slower. We >> found out node 5 out of our 6 nodes was hitting 2100% cpu (cat /proc/cpuinfo >> reveals a 16 processor machine). I am really not sure how to hit 2100% >> unless we had 21 processors. It bounces between 300% and 2100% so I tried >> to a do a thread dump and had to use –F which then hotspot hit a nullpointer >> :(. >> >> I copied off all my logs after restarting(should have done it before >> restarting it). Any ideas what I could even look for as to what went wrong >> with this node? >> >> Also, we know our astyanax for some reason is not setup properly yet so we >> probably would not have seen an issue had we had all nodes in the seed >> list(which we changed today) as astyanax is supposed to be measuring time >> per request and changing which nodes it hits but we know it only hits nodes >> in our seedlist right now as we have not fixed that yet. Our astyanax was >> hitting 3,4,5,6 and did not have 1 and 2 in the seed list (we rollout a new >> version next wed. with the new seedlist including the last two delaying the >> dynamic discovery config we need to look at). >> >> Thanks, >> Dean >> >> Commands I ran with jstack that didn't work out too well…. >> >> [cassandra@a5 ~]$ jstack -l 20907 > threads.txt >> 20907: Unable to open socket file: target process not responding or HotSpot >> VM not loaded >> The -F option can be used when the target process is not responding >> [cassandra@a5 ~]$ jstack -l -F 20907 > threads.txt >> Attaching to process ID 20907, please wait... >> Debugger attached successfully. >> Server compiler detected. >> JVM version is 20.7-b02 >> java.lang.NullPointerException >> at >> sun.jvm.hotspot.oops.InstanceKlass.computeSubtypeOf(InstanceKlass.java:426) >> at sun.jvm.hotspot.oops.Klass.isSubtypeOf(Klass.java:137) >> at sun.jvm.hotspot.oops.Oop.isA(Oop.java:100) >> at sun.jvm.hotspot.runtime.DeadlockDetector.print(DeadlockDetector.java:93) >> at sun.jvm.hotspot.runtime.DeadlockDetector.print(DeadlockDetector.java:39) >> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:52) >> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45) >> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60) >> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221) >> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >> at java.lang.reflect.Method.invoke(Method.java:597) >> at sun.tools.jstack.JStack.runJStackTool(JStack.java:118) >> at sun.tools.jstack.JStack.main(JStack.java:84) >> [cassandra@a5 ~]$ java -version >> java version "1.6.0_32"
