Thanks Nitan! Eventually it was a firewall issue related to the Centos7 node.
Once fixed, the rolling restart resolved the issue completely. Thanks again! Shalom Sagges DBA T: +972-74-700-4035 <http://www.linkedin.com/company/164748> <http://twitter.com/liveperson> <http://www.facebook.com/LivePersonInc> We Create Meaningful Connections <https://liveperson.docsend.com/view/8iiswfp> On Tue, Jul 4, 2017 at 7:52 PM, Nitan Kainth <ni...@bamlabs.com> wrote: > Try rolling restart of cluster as solution for schema version mismatch. > > > Sent from my iPhone > > On Jul 4, 2017, at 8:31 AM, Shalom Sagges <shal...@liveperson.com> wrote: > > Hi Experts, > > My plan is to upgrade the C* nodes' OS from Centos6 to Centos7. > Since an upgrade wasn't recommended I needed to install new machines with > Centos7 and join them to the cluster. > I didn't want to decommission/bootstrap dozens of nodes, so I decided to > do the following: > > - Create a new machine > - Copy all data, schema data and commit logs using rsync from the old > node to the new one > - Stop Cassandra on the old node, copy the delta and shut the node down > - Change the hostname and IP address of the new node to the hostname > and IP of the old node. > - Start Cassandra on the new node. > > My thought was that the "new" node will join the cluster as the "old" node > since it has all of its data and metadata, however, after Cassandra started > up, it saw all its peers in DN state. The same goes for the other nodes, > that saw the new node is DN state. > > I didn't find any errors or warnings in the logs, but I did see that the > *schema > version* on the new node was different from the others( I'm assuming > that's the issue here?) > > > *Functioning node* > nodetool describecluster > Cluster Information: > Name: MyCluster > Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch > Partitioner: org.apache.cassandra.dht.Murmur3Partitioner > Schema versions: > 503bf151-c59c-35a0-8350-ca73e82098f5: [x.x.x.32, > x.x.x.126, x.x.x.35, x.x.x.1, x.x.x.28, x.x.x.15, x.x.x.252, x.x.x.253, > x.x.x.31] > > UNREACHABLE: [x.x.x.2] > > > *New node* > nodetool describecluster > Cluster Information: > Name: MyCluster > Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch > Partitioner: org.apache.cassandra.dht.Murmur3Partitioner > Schema versions: > dab67163-65d4-3895-82e6-ffc07bf5c17a: [x.x.x.2] > > UNREACHABLE: [x.x.x.31, x.x.x.1, x.x.x.28, x.x.x.252, > x.x.x.253, x.x.x.15, x.x.x.126, x.x.x.35, x.x.x.32] > > > I'd really REALLY appreciate some guidance. Did I do something wrong? Is > there a way to fix this? > > > Thanks a lot! > > Shalom Sagges > DBA > <http://www.linkedin.com/company/164748> <http://twitter.com/liveperson> > <http://www.facebook.com/LivePersonInc> We Create Meaningful Connections > > > > This message may contain confidential and/or privileged information. > If you are not the addressee or authorized to receive this on behalf of > the addressee you must not use, copy, disclose or take action based on this > message or any information herein. > If you have received this message in error, please advise the sender > immediately by reply email and delete this message. Thank you. > > -- This message may contain confidential and/or privileged information. If you are not the addressee or authorized to receive this on behalf of the addressee you must not use, copy, disclose or take action based on this message or any information herein. If you have received this message in error, please advise the sender immediately by reply email and delete this message. Thank you.
