Thanks Alex. We have 6 nodes in each DC with RF=3 with CL local qourum . and we stopped and started only one instance at a time . Tough nodetool status says all nodes UN and system.log says canssandra started and started listening . Jmx explrter shows instance stayed down longer how do we determine what caused the Cassandra unavialbe though log says its stared and listening ?
On Tue, Oct 29, 2019, 4:44 PM Oleksandr Shulgin < oleksandr.shul...@zalando.de> wrote: > On Tue, Oct 29, 2019 at 9:34 PM Rahul Reddy <rahulreddy1...@gmail.com> > wrote: > >> >> We have our infrastructure on aws and we use ebs storage . And aws was >> retiring on of the node. Since our storage was persistent we did nodetool >> drain and stopped and start the instance . This caused 500 errors in the >> service. We have local_quorum and rf=3 why does stopping one instance cause >> application to have issues? >> > > Can you still look up what was the underlying error from Cassandra driver > in the application logs? Was it request timeout or not enough replicas? > > For example, if you only had 3 Cassandra nodes, restarting one of them > reduces your cluster capacity by 33% temporarily. > > Cheers, > -- > Alex > >
