[
https://issues.apache.org/jira/browse/CASSANDRA-8132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Minh Do updated CASSANDRA-8132:
-------------------------------
Description:
Often, we need to replace a node with a new instance in the cloud environment
where we have all nodes are still alive. To be safe without losing data, we
usually make sure all hints are gone before we do this operation.
Replacement means we just want to shutdown C* process on a node and bring up
another instance to take over that node's token.
However, if a node to be replaced has a lot of stored hints, its
HintedHandofManager seems very slow to send the hints to other nodes. In our
case, we tried to replace a node and had to wait for several days before its
stored hints are clear out. As mentioned above, we need all hints on this node
to clear out before we can terminate it and replace it by a new node.
Since this is not a decommission, I am proposing that we have the same
hints-streaming mechanism as in the decommission code. Furthermore, there
needs to be a cmd for NodeTool to trigger this.
was:
Often, we need to replace a node with a new instance in the cloud environment
where we have all nodes are still alive. To be safe without losing data, we
usually make sure all hints are gone before we do this operation.
Replacement means we just want to shutdown C* process on a node and bring up
another instance to take over that node's token.
However, if a node has a lot of stored hints, HintedHandofManager seems very
slow to play the hints. In our case, we tried to replace a node and had to
wait for several days.
Since this is not a decommission, I am proposing that we have the same
hints-streaming mechanism as in the decommission code. Furthermore, there
needs to be a cmd for NodeTool to trigger this.
> Save or stream hints to a safe place in node replacement
> --------------------------------------------------------
>
> Key: CASSANDRA-8132
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8132
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Minh Do
> Assignee: Minh Do
> Fix For: 2.1.1
>
>
> Often, we need to replace a node with a new instance in the cloud environment
> where we have all nodes are still alive. To be safe without losing data, we
> usually make sure all hints are gone before we do this operation.
> Replacement means we just want to shutdown C* process on a node and bring up
> another instance to take over that node's token.
> However, if a node to be replaced has a lot of stored hints, its
> HintedHandofManager seems very slow to send the hints to other nodes. In our
> case, we tried to replace a node and had to wait for several days before its
> stored hints are clear out. As mentioned above, we need all hints on this
> node to clear out before we can terminate it and replace it by a new node.
> Since this is not a decommission, I am proposing that we have the same
> hints-streaming mechanism as in the decommission code. Furthermore, there
> needs to be a cmd for NodeTool to trigger this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
