Is it possible to have true "drop in" node replacements? For example, I have a cluster of 51 Cassandra nodes, 17 in each data center. I had one host go down on DC3, and when it came back up, it joined the ring, etc., but was not receiving any data. Even after multiple restarts and forcing a repair on the entire fleet, it still holds maybe ~30MB on a cluster that is absorbing ~1.2TB a day.
On top of that, I decided to see if I could recreate it--by taking down a node, reprovisioning it, and then throwing it back in WITHOUT having it "take over" the old node's tokens, it never seems to ever absorb any of the old data after a full repair, and it never seems to start loading new data (I now have 3 nodes using ~30MB). Am I doing something wrong? I would imagine a repair on the entire cluster (across 3 DCs) would force C* to put some copies onto the other node--but this doesn't seem to be the case. What can I do? Andrew
