I'm running 2.0.11. These nodes are on bare metal at Soft Layer. So after I sent my first post logs a RuntimeException popped up in the logs, not sure if this might be related?
ERROR 14:13:17,648 Exception in thread Thread[CompactionExecutor:505,1,main] java.lang.RuntimeException: Last written key DecoratedKey(7750448305743929847, 003076697369746f725f706167655f766965613162616364663133646265316336373266363363666466366361383330323100004930303030616336342d303030302d303033302d343030302d3030303030303030616336343a64323065323964632d643836342d313165342d383030302d30303030306666636666366500) >= current key DecoratedKey(-8457751561836812744, 002c6f626a6563745f6175646974386164626664623835326465363732633935346338313261306638373264663700005230303030326263322d303030302d303033302d343030302d3030303030303030326263323a30303030326263322d303030302d303033302d343030302d3030303030336331373630303a50726f737065637400) writing into /var/lib/cassandra/data/pi/__shardindex/pi-__shardindex-tmp-jb-657-Data.db at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:143) at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:166) at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:170) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60) at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59) at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:198) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) On Tue, Aug 4, 2015 at 2:14 PM, Robert Coli <rc...@eventbrite.com> wrote: > On Tue, Aug 4, 2015 at 11:02 AM, Stan Lemon <sle...@salesforce.com> wrote: > >> I am attempting to add a 13th node in one of the datacenters. I have been >> monitoring this process from the node itself with nodetool netstats and >> from one of the existing nodes using nodetool status. >> >> On the existing node I see the new node as UJ. I have watched the load >> steadily climb up to about 203.4gb, and then over the last two hours it has >> fluctuated a bit and has been steadily dropping to about 203.1gb >> > > It's probably hung. If I were you I'd probably wipe the node and > re-bootstrap. > > (what version of cassandra/what network are you on (AWS?)/etc.) > > =Rob > >
