Re: Single cluster node restore

[Rajath Subramanyam]

Hi Petr,

>From the question, it looks like you are trying to find a shortcut to avoid
longer restore times without having data loss. The TLDR answer to your
question is that you cannot workaround by doing node level restore since
the corrupted data will eventually get propagated. But there are some other
ways you can make the restore quicker and ensure that you don't have any
data loss.

1) Fine-grained Recovery (to save you time)
If you know the column family(ies) or at least the keyspace(s) where the
data corruption happened, you can save time by doing a fine-grained restore
of only those column(ies) or keyspace(s).

2) Point-in-time Recovery (to prevent data loss)
Assuming, you had a last correct snapshot, you can restore that version.
However, all the newer data that came in after the snapshot and before the
data corruption was introduced will still be lost. In order to avoid this,
you will need to devise a point-in-time recovery strategy.

3) Repair-free Recovery (to save you time)
By going back to last correct snapshot, you might undo the corruption, but
you will also undo all the good work done by Cassandra via repairs since
the snapshot was taken and before the corruption was introduced. For this,
you will have to devise a repair-free recovery strategy.

Hope this helps.

Thank you !

Regards,
Rajath


Rajath Subramanyam


On Fri, Dec 2, 2016 at 10:19 AM, Anuj Wadehra 
wrote:

> Hi Petr,
>
> If data corruption means accidental data deletions via Cassandra commands,
> you have to restore entire cluster with latest snapshots. This may lead to
> data loss as there may be valid updates after the snapshot was taken but
> before the data deletion. Restoring single node with snapshot wont help as
> Cassandra replicated the accidental deletes to all nodes.
>
> If data corruption means accidental deletion of some sstable files from
> file system of a node, repair would fix it.
>
> If data corruption means unreadable data due to hardware issues etc, you
> will have two options after replacing the disk: bootstrap or restore
> snapshot on the single affected node. If you have huge data per node e.g.
> 300Gb , you may want to restore from Snapshot followed by repair. Restoring
> snapshot on single node is faster than streaming all data via bootstrap.
> If the node is not recoverable and must be replaced, you should be able to
> do auto-boostrap or restore from snapshot with auto-bootstrap set to false.
> I havent replaced a dead node with snapshot but there should not be any
> issues as token ranges dont change when you replace a node.
>
>
>
> Thanks
> Anuj
>
> On Tue, 29 Nov, 2016 at 11:08 PM, Petr Malik
>  wrote:
>
>
> Hi.
>
> I have a question about Cassandra backup-restore strategies.
>
> As far as I understand Cassandra has been designed to survive hardware
> failures by relying on data replication.
>
>
> It seems like people still want backup/restore for case when somebody
> accidentally deletes data or the data gets otherwise corrupted.
>
> In that case restoring all keyspace/table snapshots on all nodes should
> bring it back.
>
>
> I am asking because I often read directions on restoring a single node in
> a cluster. I am just wondering under what circumstances could this be done
> safely.
>
>
> Please correct me if i am wrong but restoring just a single node does not
> really roll back the data as the newer (corrupt) data will be served by
> other replicas and eventually propagated to the restored node. Right?
>
> In fact by doing so one may end up reintroducing deleted data back...
>
>
> Also since Cassandra distributes the data throughout the cluster it is not
> clear on which mode any particular (corrupt) data resides and hence which
> to restore.
>
>
> I guess this is a long way of asking whether there is an advantage of
> trying to restore just a single node in a Cassandra cluster as opposed to
> say replacing the dead node and letting Cassandra handle the replication.
>
>
> Thanks.
>
>

Re: Single cluster node restore

[Anuj Wadehra]

Hi Petr,
If data corruption means accidental data deletions via Cassandra commands, you 
have to restore entire cluster with latest snapshots. This may lead to data 
loss as there may be valid updates after the snapshot was taken but before the 
data deletion. Restoring single node with snapshot wont help as Cassandra 
replicated the accidental deletes to all nodes.
If data corruption means accidental deletion of some sstable files from file 
system of a node, repair would fix it.
If data corruption means unreadable data due to hardware issues etc, you will 
have two options after replacing the disk: bootstrap or restore snapshot on the 
single affected node. If you have huge data per node e.g. 300Gb , you may want 
to restore from Snapshot followed by repair. Restoring snapshot on single node 
is faster than streaming all data via bootstrap. If the node is not recoverable 
and must be replaced, you should be able to do auto-boostrap or restore from 
snapshot with auto-bootstrap set to false. I havent replaced a dead node with 
snapshot but there should not be any issues as token ranges dont change when 
you replace a node.



Thanks
Anuj 
 
  On Tue, 29 Nov, 2016 at 11:08 PM, Petr Malik wrote:   


Hi.

I have a question about Cassandra backup-restore strategies.


As far as I understand Cassandra has been designed to survive hardware failures 
by relying on data replication.




It seems like people still want backup/restore for case when somebody 
accidentally deletes data or the data gets otherwise corrupted.

In that case restoring all keyspace/table snapshots on all nodes should bring 
it back.




I am asking because I often read directions on restoring a single node in a 
cluster. I am just wondering under what circumstances could this be done safely.





Please correct me if i am wrong but restoring just a single node does not 
really roll back the data as the newer (corrupt) data will be served by other 
replicas and eventually propagated to the restored node. Right?

In fact by doing so one may end up reintroducing deleted data back...




Also since Cassandra distributes the data throughout the cluster it is not 
clear on which mode any particular (corrupt) data resides and hence which to 
restore.




I guess this is a long way of asking whether there is an advantage of trying to 
restore just a single node in a Cassandra cluster as opposed to say replacing 
the dead node and letting Cassandra handle the replication.




Thanks.
  

Re: Single cluster node restore

[Ben Slater]

You can have situations where rebuilding a node via streaming is painful
and slow (generally because there is something bad about the data model
like misused secondary indexes or massive partitions). Also, overstreaming
can mean you need more disk space to bootstrap a node than you’ll require
once it’s fully streamed and compacted - this can be hard to work around in
some environments. In this case you might want to restore a single node
from backup.

However, in general you’re right - it’s not something that tends to be done
very often.

Cheers
Ben

On Wed, 30 Nov 2016 at 04:38 Petr Malik  wrote:

>
> Hi.
>
> I have a question about Cassandra backup-restore strategies.
>
> As far as I understand Cassandra has been designed to survive hardware
> failures by relying on data replication.
>
>
> It seems like people still want backup/restore for case when somebody
> accidentally deletes data or the data gets otherwise corrupted.
>
> In that case restoring all keyspace/table snapshots on all nodes should
> bring it back.
>
>
> I am asking because I often read directions on restoring a single node in
> a cluster. I am just wondering under what circumstances could this be done
> safely.
>
>
> Please correct me if i am wrong but restoring just a single node does not
> really roll back the data as the newer (corrupt) data will be served by
> other replicas and eventually propagated to the restored node. Right?
>
> In fact by doing so one may end up reintroducing deleted data back...
>
>
> Also since Cassandra distributes the data throughout the cluster it is not
> clear on which mode any particular (corrupt) data resides and hence which
> to restore.
>
>
> I guess this is a long way of asking whether there is an advantage of
> trying to restore just a single node in a Cassandra cluster as opposed to
> say replacing the dead node and letting Cassandra handle the replication.
>
>
> Thanks.
>

Single cluster node restore

[Petr Malik]

Hi.

I have a question about Cassandra backup-restore strategies.


As far as I understand Cassandra has been designed to survive hardware failures 
by relying on data replication.


It seems like people still want backup/restore for case when somebody 
accidentally deletes data or the data gets otherwise corrupted.

In that case restoring all keyspace/table snapshots on all nodes should bring 
it back.


I am asking because I often read directions on restoring a single node in a 
cluster. I am just wondering under what circumstances could this be done safely.


Please correct me if i am wrong but restoring just a single node does not 
really roll back the data as the newer (corrupt) data will be served by other 
replicas and eventually propagated to the restored node. Right?

In fact by doing so one may end up reintroducing deleted data back...


Also since Cassandra distributes the data throughout the cluster it is not 
clear on which mode any particular (corrupt) data resides and hence which to 
restore.


I guess this is a long way of asking whether there is an advantage of trying to 
restore just a single node in a Cassandra cluster as opposed to say replacing 
the dead node and letting Cassandra handle the replication.


Thanks.