Re: Adding a New Node With The Same IP of an Old Node

[Shalom Sagges]

Thanks Nitan!

Eventually it was a firewall issue related to the Centos7 node.

Once fixed, the rolling restart resolved the issue completely.

Thanks again!



Shalom Sagges
DBA
T: +972-74-700-4035
 
 We Create Meaningful Connections



On Tue, Jul 4, 2017 at 7:52 PM, Nitan Kainth  wrote:

> Try rolling restart of cluster as solution for schema version mismatch.
>
>
> Sent from my iPhone
>
> On Jul 4, 2017, at 8:31 AM, Shalom Sagges  wrote:
>
> Hi Experts,
>
> My plan is to upgrade the C* nodes' OS from Centos6 to Centos7.
> Since an upgrade wasn't recommended I needed to install new machines with
> Centos7 and join them to the cluster.
> I didn't want to decommission/bootstrap dozens of nodes, so I decided to
> do the following:
>
>- Create a new machine
>- Copy all data, schema data and commit logs using rsync from the old
>node to the new one
>- Stop Cassandra on the old node, copy the delta and shut the node down
>- Change the hostname and IP address of the new node to the hostname
>and IP of the old node.
>- Start Cassandra on the new node.
>
> My thought was that the "new" node will join the cluster as the "old" node
> since it has all of its data and metadata, however, after Cassandra started
> up, it saw all its peers in DN state. The same goes for the other nodes,
> that saw the new node is DN state.
>
> I didn't find any errors or warnings in the logs, but I did see that the 
> *schema
> version* on the new node was different from the others( I'm assuming
> that's the issue here?)
>
>
> *Functioning node*
> nodetool describecluster
> Cluster Information:
> Name: MyCluster
> Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
> Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
> Schema versions:
> 503bf151-c59c-35a0-8350-ca73e82098f5: [x.x.x.32,
> x.x.x.126, x.x.x.35, x.x.x.1, x.x.x.28, x.x.x.15, x.x.x.252, x.x.x.253,
> x.x.x.31]
>
> UNREACHABLE: [x.x.x.2]
>
>
> *New node*
> nodetool describecluster
> Cluster Information:
> Name: MyCluster
> Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
> Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
> Schema versions:
> dab67163-65d4-3895-82e6-ffc07bf5c17a: [x.x.x.2]
>
> UNREACHABLE: [x.x.x.31, x.x.x.1, x.x.x.28, x.x.x.252,
> x.x.x.253, x.x.x.15, x.x.x.126, x.x.x.35, x.x.x.32]
>
>
> I'd really REALLY appreciate some guidance. Did I do something wrong? Is
> there a way to fix this?
>
>
> Thanks a lot!
>
> Shalom Sagges
> DBA
>  
>  We Create Meaningful Connections
>
>
>
> This message may contain confidential and/or privileged information.
> If you are not the addressee or authorized to receive this on behalf of
> the addressee you must not use, copy, disclose or take action based on this
> message or any information herein.
> If you have received this message in error, please advise the sender
> immediately by reply email and delete this message. Thank you.
>
>

-- 
This message may contain confidential and/or privileged information. 
If you are not the addressee or authorized to receive this on behalf of the 
addressee you must not use, copy, disclose or take action based on this 
message or any information herein. 
If you have received this message in error, please advise the sender 
immediately by reply email and delete this message. Thank you.

Re: Adding a New Node With The Same IP of an Old Node

[Nitan Kainth]

Try rolling restart of cluster as solution for schema version mismatch.


Sent from my iPhone

> On Jul 4, 2017, at 8:31 AM, Shalom Sagges  wrote:
> 
> Hi Experts, 
> 
> My plan is to upgrade the C* nodes' OS from Centos6 to Centos7. 
> Since an upgrade wasn't recommended I needed to install new machines with 
> Centos7 and join them to the cluster. 
> I didn't want to decommission/bootstrap dozens of nodes, so I decided to do 
> the following:
> Create a new machine
> Copy all data, schema data and commit logs using rsync from the old node to 
> the new one
> Stop Cassandra on the old node, copy the delta and shut the node down
> Change the hostname and IP address of the new node to the hostname and IP of 
> the old node.
> Start Cassandra on the new node. 
> My thought was that the "new" node will join the cluster as the "old" node 
> since it has all of its data and metadata, however, after Cassandra started 
> up, it saw all its peers in DN state. The same goes for the other nodes, that 
> saw the new node is DN state. 
> 
> I didn't find any errors or warnings in the logs, but I did see that the 
> schema version on the new node was different from the others( I'm assuming 
> that's the issue here?) 
> 
> 
> Functioning node
> nodetool describecluster
> Cluster Information:
> Name: MyCluster
> Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
> Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
> Schema versions:
> 503bf151-c59c-35a0-8350-ca73e82098f5: [x.x.x.32, x.x.x.126, 
> x.x.x.35, x.x.x.1, x.x.x.28, x.x.x.15, x.x.x.252, x.x.x.253, x.x.x.31]
> 
> UNREACHABLE: [x.x.x.2]
> 
> 
> New node
> nodetool describecluster
> Cluster Information:
> Name: MyCluster
> Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
> Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
> Schema versions:
> dab67163-65d4-3895-82e6-ffc07bf5c17a: [x.x.x.2]
> 
> UNREACHABLE: [x.x.x.31, x.x.x.1, x.x.x.28, x.x.x.252, 
> x.x.x.253, x.x.x.15, x.x.x.126, x.x.x.35, x.x.x.32]
> 
> 
> I'd really REALLY appreciate some guidance. Did I do something wrong? Is 
> there a way to fix this? 
> 
> 
> Thanks a lot!
> 
> Shalom Sagges
> DBA
>   
> We Create Meaningful Connections
> 
>  
> 
> This message may contain confidential and/or privileged information. 
> If you are not the addressee or authorized to receive this on behalf of the 
> addressee you must not use, copy, disclose or take action based on this 
> message or any information herein. 
> If you have received this message in error, please advise the sender 
> immediately by reply email and delete this message. Thank you.

Adding a New Node With The Same IP of an Old Node

[Shalom Sagges]

Hi Experts,

My plan is to upgrade the C* nodes' OS from Centos6 to Centos7.
Since an upgrade wasn't recommended I needed to install new machines with
Centos7 and join them to the cluster.
I didn't want to decommission/bootstrap dozens of nodes, so I decided to do
the following:

   - Create a new machine
   - Copy all data, schema data and commit logs using rsync from the old
   node to the new one
   - Stop Cassandra on the old node, copy the delta and shut the node down
   - Change the hostname and IP address of the new node to the hostname and
   IP of the old node.
   - Start Cassandra on the new node.

My thought was that the "new" node will join the cluster as the "old" node
since it has all of its data and metadata, however, after Cassandra started
up, it saw all its peers in DN state. The same goes for the other nodes,
that saw the new node is DN state.

I didn't find any errors or warnings in the logs, but I did see that
the *schema
version* on the new node was different from the others( I'm assuming that's
the issue here?)


*Functioning node*
nodetool describecluster
Cluster Information:
Name: MyCluster
Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
Schema versions:
503bf151-c59c-35a0-8350-ca73e82098f5: [x.x.x.32, x.x.x.126,
x.x.x.35, x.x.x.1, x.x.x.28, x.x.x.15, x.x.x.252, x.x.x.253, x.x.x.31]

UNREACHABLE: [x.x.x.2]


*New node*
nodetool describecluster
Cluster Information:
Name: MyCluster
Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
Schema versions:
dab67163-65d4-3895-82e6-ffc07bf5c17a: [x.x.x.2]

UNREACHABLE: [x.x.x.31, x.x.x.1, x.x.x.28, x.x.x.252,
x.x.x.253, x.x.x.15, x.x.x.126, x.x.x.35, x.x.x.32]


I'd really REALLY appreciate some guidance. Did I do something wrong? Is
there a way to fix this?


Thanks a lot!

Shalom Sagges
DBA
 
 We Create Meaningful Connections

-- 
This message may contain confidential and/or privileged information. 
If you are not the addressee or authorized to receive this on behalf of the 
addressee you must not use, copy, disclose or take action based on this 
message or any information herein. 
If you have received this message in error, please advise the sender 
immediately by reply email and delete this message. Thank you.