If it's helps, here's the log with debug log statements. Possibly issue with that exception?
INFO [RMI TCP Connection(2)-10.116.111.143] 2013-03-09 03:54:32,402 StorageService.java (line 774) DRAINING: starting drain process INFO [RMI TCP Connection(2)-10.116.111.143] 2013-03-09 03:54:32,403 CassandraDaemon.java (line 218) Stop listening to thrift clients INFO [RMI TCP Connection(2)-10.116.111.143] 2013-03-09 03:54:32,404 Gossiper.java (line 1133) Announcing shutdown DEBUG [GossipTasks:1] 2013-03-09 03:54:33,328 DebuggableThreadPoolExecutor.java (line 190) Task cancelled java.util.concurrent.CancellationException at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:220) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.extractThrowable(DebuggableThreadPoolExecutor.java:182) at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.logExceptionsAfterExecute(DebuggableThreadPoolExecutor.java:146) at org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor.afterExecute(DebuggableScheduledThreadPoolExecutor.java:50) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) DEBUG [RMI TCP Connection(2)-10.116.111.143] 2013-03-09 03:54:33,406 StorageService.java (line 776) DRAINING: shutting down MessageService INFO [RMI TCP Connection(2)-10.116.111.143] 2013-03-09 03:54:33,406 MessagingService.java (line 534) Waiting for messaging service to quiesce INFO [ACCEPT-ip-10-116-111-143.ec2.internal/10.116.111.143] 2013-03-09 03:54:33,407 MessagingService.java (line 690) MessagingService shutting down server thread. DEBUG [RMI TCP Connection(2)-10.116.111.143] 2013-03-09 03:54:33,408 StorageService.java (line 776) DRAINING: waiting for streaming DEBUG [RMI TCP Connection(2)-10.116.111.143] 2013-03-09 03:54:33,408 StorageService.java (line 776) DRAINING: clearing mutation stage DEBUG [Thread-5] 2013-03-09 03:54:33,408 Gossiper.java (line 221) Reseting version for /10.83.55.44 DEBUG [RMI TCP Connection(2)-10.116.111.143] 2013-03-09 03:54:33,409 StorageService.java (line 776) DRAINING: flushing column families DEBUG [RMI TCP Connection(2)-10.116.111.143] 2013-03-09 03:54:33,409 ColumnFamilyStore.java (line 713) forceFlush requested but everything is clean in Counter1 DEBUG [Thread-6] 2013-03-09 03:54:33,410 Gossiper.java (line 221) Reseting version for /10.80.187.124 DEBUG [RMI TCP Connection(2)-10.116.111.143] 2013-03-09 03:54:33,410 ColumnFamilyStore.java (line 713) forceFlush requested but everything is clean in Super1 DEBUG [RMI TCP Connection(2)-10.116.111.143] 2013-03-09 03:54:33,410 ColumnFamilyStore.java (line 713) forceFlush requested but everything is clean in SuperCounter1 DEBUG [RMI TCP Connection(2)-10.116.111.143] 2013-03-09 03:54:33,410 ColumnFamilyStore.java (line 713) forceFlush requested but everything is clean in Standard1 INFO [RMI TCP Connection(2)-10.116.111.143] 2013-03-09 03:54:33,510 StorageService.java (line 774) DRAINED On Fri, Mar 8, 2013 at 10:36 PM, Andrew Bialecki <[email protected]>wrote: > Hey all, > > We're getting ready to upgrade our cluster to 1.2.2 from 1.1.5 and we're > testing the upgrade process on our dev cluster. We turned off all client > access to the cluster and then ran "nodetool drain" on the first instance > with the intention of running "nodetool snapshot" once it finished. > However, after running the drain, didn't see any errors, but the Cassandra > process was no longer running. Is that expected? From everything I've read > it doesn't seem like it, but maybe I'm mistaken. > > Here's the relevant portion of the log from that node (notice it says it's > shutting down the server thread in there): > > INFO [RMI TCP Connection(38)-10.116.111.143] 2013-03-09 03:26:48,288 > StorageService.java (line 774) DRAINING: starting drain process > INFO [RMI TCP Connection(38)-10.116.111.143] 2013-03-09 03:26:48,288 > CassandraDaemon.java (line 218) Stop listening to thrift clients > INFO [RMI TCP Connection(38)-10.116.111.143] 2013-03-09 03:26:48,315 > Gossiper.java (line 1133) Announcing shutdown > INFO [RMI TCP Connection(38)-10.116.111.143] 2013-03-09 03:26:49,318 > MessagingService.java (line 534) Waiting for messaging service to quiesce > INFO [ACCEPT-ip-10-116-111-143.ec2.internal/10.116.111.143] 2013-03-09 > 03:26:49,319 MessagingService.java (line 690) MessagingService shutting > down server thread. > INFO [RMI TCP Connection(38)-10.116.111.143] 2013-03-09 03:26:49,338 > ColumnFamilyStore.java (line 659) Enqueuing flush of > Memtable-Counter1@177255852(14810190/60139556 serialized/live bytes, > 243550 ops) > INFO [FlushWriter:7] 2013-03-09 03:26:49,338 Memtable.java (line 264) > Writing Memtable-Counter1@177255852(14810190/60139556 serialized/live > bytes, 243550 ops) > INFO [FlushWriter:7] 2013-03-09 03:26:49,899 Memtable.java (line 305) > Completed flushing > /var/lib/cassandra/data/Keyspace1/Counter1/Keyspace1-Counter1-he-104-Data.db > (15204741 bytes) for commitlog position > ReplayPosition(segmentId=1362797442799, position=27621115) > INFO [CompactionExecutor:11] 2013-03-09 03:26:49,900 CompactionTask.java > (line 109) Compacting > [SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Counter1/Keyspace1-Counter1-he-102-Data.db'), > SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Counter1/Keyspace1-Counter1-he-103-Data.db'), > SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Counter1/Keyspace1-Counter1-he-104-Data.db'), > SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Counter1/Keyspace1-Counter1-he-101-Data.db')] > INFO [RMI TCP Connection(38)-10.116.111.143] 2013-03-09 03:26:50,193 > StorageService.java (line 774) DRAINED > > > Thanks in advanced for any help. > > Cheers, > Andrew >
