What I want to do is kind of replacing a dead node - http://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_replace_node_t.html But replacing it with a clean node with the same IP and hostname.
On Sun, Dec 21, 2014 at 9:53 AM, Or Sher <or.sh...@gmail.com> wrote: > Thanks guys. > I have to replace all data disks, so I don't have another large enough > local disk to move the data to. > If I'll have no choice, I will backup the data before on some other node > or something, but I'd like to avoid it. > I would really love letting Cassandra do it thing and rebuild itself. > Did anybody handled such cases that way (Letting Cassandra rebuild it's > data?) > Although there are no documented procedure for it, It should be possible > right? > > On Fri, Dec 19, 2014 at 8:41 AM, Jan Kesten <j.kes...@enercast.de> wrote: > >> Hi Or, >> >> I did some sort of this a while ago. If your machines do have a free disk >> slot - just put another disk there and use it as another >> data_file_directory. >> >> If not - as in my case: >> >> - grab an usb dock for disks >> - put the new one in there, plug in, format, mount to /mnt etc. >> - I did an online rsync from /var/lib/cassandra/data to /mnt >> - after that, bring cassandra down >> - do another rsync from /var/lib/cassandra/data to /mnt (should be >> faster, as sstables do not change, minimizes downtime) >> - if you need adjust /etc/fstab if needed >> - shutdown the node >> - swap disks >> - power on the node >> - everything should be fine ;-) >> >> Of course you will need a replication factor > 1 for this to work ;-) >> >> Just my 2 cents, >> Jan >> >> rsync the full contents there, >> >> Am 18.12.2014 um 16:17 schrieb Or Sher: >> >> Hi all, >>> >>> We have a situation where some of our nodes have smaller disks and we >>> would like to align all nodes by replacing the smaller disks to bigger ones >>> without replacing nodes. >>> We don't have enough space to put data on / disk and copy it back to the >>> bigger disks so we would like to rebuild the nodes data from other replicas. >>> >>> What do you think should be the procedure here? >>> >>> I'm guessing it should be something like this but I'm pretty sure it's >>> not enough. >>> 1. shutdown C* node and server. >>> 2. replace disks + create the same vg lv etc. >>> 3. start C* (Normally?) >>> 4. nodetool repair/rebuild? >>> *I think I might get some consistency issues for use cases relying on >>> Quorum reads and writes for strong consistency. >>> What do you say? >>> >>> Another question is (and I know it depends on many factors but I'd like >>> to hear an experienced estimation): How much time would take to rebuild a >>> 250G data node? >>> >>> Thanks in advance, >>> Or. >>> >>> -- >>> Or Sher >>> >> >> > > > -- > Or Sher > -- Or Sher
