Hello Rahul, Basically the issue is running nodetool status on the host rebooted node, shows itself as UN and all other nodes in the cluster as DN.
And running nodetool status on any other node in the cluster shows this rebooted node as DN. Correct me if I am wrong. Is this the issue. Also attach screenshot of the observation you are talking about. You may choose to replace the ip address of the hosts Thanks On Fri, 19 Jul, 2019, 9:36 PM Rahul Reddy, <rahulreddy1...@gmail.com> wrote: > Thanks for quick response rajshekar. > > Correct same cassandra.yml and same java > > On Fri, Jul 19, 2019, 11:56 AM Rajsekhar Mallick <raj.mallic...@gmail.com> > wrote: > >> Hello Rahul, >> >> May you please confirm on below things. >> >> 1. Cassandra.yaml file of the node which was started after the machine >> reboot is same as that of rest of the nodes in the cluster. >> 2. Java version is consistent across all nodes in the cluster. >> >> Do check and revert >> >> Thanks >> >> On Fri, 19 Jul, 2019, 9:19 PM Rahul Reddy, <rahulreddy1...@gmail.com> >> wrote: >> >>> Here , >>> >>> We have 6 nodes each in 2 data centers us-east-1 and us-west-2 . We >>> have RF 3 and cl set to local quorum. And gossip snitch. All our instance >>> are c5.2xlarge and data files and comit logs are stored in gp2 ebs. C5 >>> instance type had a bug which aws asked to set nvme_timeout to higher >>> number in etc/grub.conf. after setting the parameter and did run nodetool >>> drain and reboot the node in east >>> >>> Instance cameup but Cassandra didn't come up normal had to start the >>> Cassandra. Cassandra cameup but it shows other instances down. Even though >>> didn't reboot the other node down same was observed in one other node. How >>> could that happen and don't any errors in system.log which is set to info. >>> Without any intervention gossip settled in 10 mins entire cluster became >>> normal. >>> >>> Tried same thing West it happened again >>> >>> >>> >>> I'm concerned how to check what caused it and if a reboot happens again >>> how to avoid this. >>> If I just STOP Cassandra instead of reboot I don't see this issue. >>> >>>
