Well, since my last post, about 10 minutes later, the node goes into bootstrap mode. It's kind of worrying that a lot of time goes by where it seems like nothing is happening, then all of a sudden things get going again.
22,584 keys. Time: 20,276ms. INFO [HintedHandoff:1] 2011-04-05 22:29:23,167 HintedHandOffManager.java (line 304) Started hinted handoff for endpoint /10.46.108.101 INFO [HintedHandoff:1] 2011-04-05 22:29:23,167 HintedHandOffManager.java (line 360) Finished hinted handoff of 0 rows to endpoint /10.46.108.101 ---- LONG PAUSE WHERE NOTHING HAPPENS ---- INFO [RMI TCP Connection(4)-10.46.108.102] 2011-04-05 22:43:38,770 StorageService.java (line 1637) Announcing that I have left the ring for 30000ms INFO [RMI TCP Connection(4)-10.46.108.102] 2011-04-05 22:44:08,770 StorageService.java (line 1747) re-bootstrapping to new token 85070591730234615865843651857942052863 INFO [RMI TCP Connection(4)-10.46.108.102] 2011-04-05 22:44:08,771 ColumnFamilyStore.java (line 695) switching in a fresh Memtable for LocationInfo at CommitLogContext(file='/var/lib/cassandra/commitlog/CommitLog-1302035265949.log', position=25920946) INFO [RMI TCP Connection(4)-10.46.108.102] 2011-04-05 22:44:08,771 ColumnFamilyStore.java (line 1006) Enqueuing flush of Memtable-LocationInfo@1358281533(53 bytes, 2 operations) INFO [FlushWriter:1] 2011-04-05 22:44:08,772 Memtable.java (line 157) Writing Memtable-LocationInfo@1358281533(53 bytes, 2 operations) INFO [FlushWriter:1] 2011-04-05 22:44:08,825 Memtable.java (line 164) Completed flushing /var/lib/cassandra/data/system/LocationInfo-f-22-Data.db (163 bytes) INFO [RMI TCP Connection(4)-10.46.108.102] 2011-04-05 22:44:08,826 StorageService.java (line 505) Joining: sleeping 30000 ms for pending range setup INFO [RMI TCP Connection(4)-10.46.108.102] 2011-04-05 22:44:38,826 StorageService.java (line 505) Bootstrapping INFO [CompactionExecutor:1] 2011-04-05 22:44:43,952 SSTableReader.java (line 154) Opening /var/lib/cassandra/data/DFS/main-f-128 INFO [CompactionExecutor:1] 2011-04-05 22:44:43,978 SSTableReader.java (line 154) Opening /var/lib/cassandra/data/DFS/main-f-129 INFO [CompactionExecutor:1] 2011-04-05 22:46:02,228 SSTableReader.java (line 154) Opening /var/lib/cassandra/data/DFS/main-f-130 On Apr 5, 2011, at 10:46 PM, Jonathan Colby wrote: > > I added a node to the cluster and I am having a difficult time reassigning > the new tokens. > > It seems after a while nothing shows up in the new node's logs and it just > stays in status "Leaving". "nodetool netstats" on all nodes shows > "Nothing streaming to/from". > > There is no activity in the other logs related to the move. > > The data size is not even that big, around 5 GB. What could be happening? > Seems like the move is frozen.
