Hi
I encounter the High CPU problem, Cassandra 1.0.3, happened on both
sized and leveled compaction, 6G heap, 64bit Oracle java. For normal
traffic, Cassandra will use 15% CPU.
But every half a hour, Cassandra will use almost 100% total cpu (SUSE,
12 Core).
And here is the top information for that moment.
#top -H -p 12451
top - 12:30:14 up 15 days, 12:49, 6 users, load average: 10.52, 8.92, 8.14
Tasks: 706 total, 21 running, 685 sleeping, 0 stopped, 0 zombie
Cpu(s): 25.7%us, 14.0%sy, 48.9%ni, 6.5%id, 0.0%wa, 0.0%hi, 4.9%si,
0.0%st
Mem: 24150M total, 12218M used, 11932M free, 142M buffers
Swap: 0M total, 0M used, 0M free, 3714M cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
20291 casadm 24 4 8003m 5.4g 167m R 92 22.7 0:42.46 java
20276 casadm 24 4 8003m 5.4g 167m R 88 22.7 0:43.88 java
20181 casadm 24 4 8003m 5.4g 167m R 86 22.7 0:52.97 java
20213 casadm 24 4 8003m 5.4g 167m R 85 22.7 0:49.21 java
20188 casadm 24 4 8003m 5.4g 167m R 82 22.7 0:54.34 java
20268 casadm 24 4 8003m 5.4g 167m R 81 22.7 0:46.25 java
20269 casadm 24 4 8003m 5.4g 167m R 41 22.7 0:15.11 java
20316 casadm 24 4 8003m 5.4g 167m S 20 22.7 0:02.35 java
20191 casadm 24 4 8003m 5.4g 167m R 15 22.7 0:16.85 java
12500 casadm 20 0 8003m 5.4g 167m R 6 22.7 1:07.86 java
15245 casadm 20 0 8003m 5.4g 167m D 5 22.7 0:36.45 java
Jstack can not print the stack.
Thread 20291: (state = IN_JAVA)
Error occurred during stack walking:
...
Thread 20276: (state = IN_JAVA)
Error occurred during stack walking:
After it come back, the stack shows:
Thread 20291: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
- java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=196 (Compiled frame)
-
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.util.concurrent.SynchronousQueue$TransferStack$SNode,
boolean, long) @bci=174, line=424 (Compiled frame)
-
java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.lang.Object,
boolean, long) @bci=102, line=323 (Compiled frame)
- java.util.concurrent.SynchronousQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=11, line=874 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.getTask() @bci=62, line=945
(Compiled frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907
(Compiled frame)
- java.lang.Thread.run() @bci=11, line=662 (Interpreted frame
And after this happened, the data is not correct, some
large column which suppose to be deleted, come back again.
Here is the suspect thread when it use up 100%
Thread 20191: (state = IN_VM)
- sun.misc.Unsafe.unpark(java.lang.Object) @bci=0 (Compiled frame;
information may be imprecise)
- java.util.concurrent.locks.LockSupport.unpark(java.lang.Thread) @bci=8,
line=122 (Compiled frame)
-
java.util.concurrent.SynchronousQueue$TransferStack$SNode.tryMatch(java.util.concurrent.SynchronousQueue$TransferStack$SNode)
@bci=34, line=242 (Compiled frame)
-
java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.lang.Object,
boolean, long) @bci=268, line=344 (Compiled frame)
- java.util.concurrent.SynchronousQueue.offer(java.lang.Object) @bci=19,
line=846 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.execute(java.lang.Runnable)
@bci=43, line=653 (Compiled frame)
-
java.util.concurrent.AbstractExecutorService.submit(java.util.concurrent.Callable)
@bci=20, line=92 (Compiled frame)
-
org.apache.cassandra.db.compaction.ParallelCompactionIterable$Reducer.getCompactedRow(java.util.List)
@bci=86, line=190 (Compiled frame) -
org.apache.cassandra.db.compaction.ParallelCompactionIterable$Reducer.getReduced()
@bci=31, line=164 (Compiled frame)
-
org.apache.cassandra.db.compaction.ParallelCompactionIterable$Reducer.getReduced()
@bci=1, line=144 (Compiled frame)
- org.apache.cassandra.utils.MergeIterator$ManyToOne.consume() @bci=88,
line=116 (Compiled frame)
- org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext() @bci=5,
line=99 (Compiled frame)
- com.google.common.collect.AbstractIterator.tryToComputeNext() @bci=9,
line=140 (Compiled frame)
- com.google.common.collect.AbstractIterator.hasNext() @bci=61, line=135
(Compiled frame)
-
org.apache.cassandra.db.compaction.ParallelCompactionIterable$Unwrapper.computeNext()
@bci=4, line=103 (Compiled frame)
-
org.apache.cassandra.db.compaction.ParallelCompactionIterable$Unwrapper.computeNext()
@bci=1, line=90 (Compiled frame)
- com.google.common.collect.AbstractIterator.tryToComputeNext() @bci=9,
line=140 (Compiled frame)
- com.google.common.collect.AbstractIterator.hasNext() @bci=61, line=135
(Compiled frame)
- com.google.common.collect.Iterators$7.computeNext() @bci=4, line=614
(Compiled frame)
- com.google.common.collect.AbstractIterator.tryToComputeNext() @bci=9,
line=140 (Compiled frame)
- com.google.common.collect.AbstractIterator.hasNext() @bci=61, line=135
(Compiled frame)
-
org.apache.cassandra.db.compaction.CompactionTask.execute(org.apache.cassandra.db.compaction.CompactionManager$CompactionExecutorStatsCollector)
@bci=772, line=172 (Compiled frame)
-
org.apache.cassandra.db.compaction.LeveledCompactionTask.execute(org.apache.cassandra.db.compaction.CompactionManager$CompactionExecutorStatsCollector)
@bci=2, line=57 (Interpreted frame)
- org.apache.cassandra.db.compaction.CompactionManager$1.call() @bci=117,
line=134 (Interpreted frame)
- org.apache.cassandra.db.compaction.CompactionManager$1.call() @bci=1,
line=114 (Interpreted frame)
- java.util.concurrent.FutureTask$Sync.innerRun() @bci=30, line=303
(Compiled frame)
- java.util.concurrent.FutureTask.run() @bci=4, line=138 (Compiled frame)
-
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(java.lang.Runnable)
@bci=59, line=886 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=28, line=908
(Compiled frame)
- java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
Thread 20269: (state = BLOCKED)
- org.apache.cassandra.utils.obs.OpenBitSet.<init>(long, boolean) @bci=51,
line=104 (Compiled frame)
- org.apache.cassandra.utils.obs.OpenBitSet.<init>(long) @bci=3, line=92
(Compiled frame)
- org.apache.cassandra.utils.BloomFilter.bucketsFor(long, int) @bci=12,
line=54 (Compiled frame)
- org.apache.cassandra.utils.BloomFilter.getFilter(long, int) @bci=110,
line=73 (Compiled frame)
-
org.apache.cassandra.db.ColumnIndexer.serialize(org.apache.cassandra.io.util.IIterableColumns)
@bci=10, line=83 (Compiled frame)
-
org.apache.cassandra.db.ColumnIndexer.serialize(org.apache.cassandra.io.util.IIterableColumns,
java.io.DataOutput) @bci=5, line=51 (Compiled frame)
-
org.apache.cassandra.db.compaction.PrecompactedRow.write(java.io.DataOutput)
@bci=42, line=140 (Compiled frame)
-
org.apache.cassandra.io.sstable.SSTableWriter.append(org.apache.cassandra.db.compaction.AbstractCompactedRow)
@bci=43, line=160 (Compiled frame)
-
org.apache.cassandra.db.compaction.CompactionTask.execute(org.apache.cassandra.db.compaction.CompactionManager$CompactionExecutorStatsCollector)
@bci=685, line=158 (Compiled frame)
-
org.apache.cassandra.db.compaction.LeveledCompactionTask.execute(org.apache.cassandra.db.compaction.CompactionManager$CompactionExecutorStatsCollector)
@bci=2, line=57 (Interpreted frame)
- org.apache.cassandra.db.compaction.CompactionManager$1.call() @bci=117,
line=134 (Interpreted frame)
- org.apache.cassandra.db.compaction.CompactionManager$1.call() @bci=1,
line=114 (Interpreted frame)
- java.util.concurrent.FutureTask$Sync.innerRun() @bci=30, line=303
(Compiled frame)
- java.util.concurrent.FutureTask.run() @bci=4, line=138 (Compiled frame)
-
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(java.lang.Runnable)
@bci=59, line=886 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=28, line=908
(Compiled frame)
- java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
BRs
//Tang Weiqiang
