Thanks for you reply Malte. I replaced the nodes and the new nodes have new IP, but I have removed the dead nodes using nodetool. Does Cassandra still keep the info of dead nodes for 72 hours even I have removed them?
Thanks, Guan From: Malte Pickhan <malte.pick...@zalando.de<mailto:malte.pick...@zalando.de>> Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>> Date: Tuesday, January 17, 2017 at 7:48 AM To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>> Subject: Re: Unreliable JMX metrics Hey Guan, If it's the case that you actually replaced the nodes and already assigned new IP's to them this is probably common behaviour. Since Cassandra has a retention policy to keep dead nodes for 72 hours. Best, Malte 2017-01-17 0:29 GMT+01:00 Sun, Guan <guan...@amazon.com<mailto:guan...@amazon.com>>: Hi all, We have a 6 node Cassandra cluster running on Amazon EC2. We replaced the nodes one by one while Cassandra was running several days ago. After the replacement, the nodetool status show that all the nodes were up and running. However, the JMX metrics about down nodes kept showing there were 6 nodes down for about 2.5 days before it became 0. Has anyone seen this before? Any idea why the JMX metrics takes so long to reflect the actual status of the cluster? Thanks, Guan -- Malte Pickhan Software Engineer Zalando Payments SE & Co. KG POSTAL ADDRESS Zalando Payments SE & Co. KG 11501 Berlin OFFICE Zalando Payments SE & Co. KG Hörder Hafenstraße 1-3 44263 Dortmund Germany E-Mail: malte.pick...@zalando.de<mailto:malte.pick...@zalando.de> Web: corporate.zalando.de<http://corporate.zalando.de/> Jobs: jobs.zalando.de<http://jobs.zalando.de/> Zalando Payments SE & Co. KG Company registration: Amtsgericht Charlottenburg, HRA 48907 B VAT registration number: DE 289683421 General partner: Zalando SE, registered office: Berlin Company registration: Amtsgericht Charlottenburg, HRA 48907 B Management Board: Robert Gentz, David Schneider, Rubin Ritter Chairperson of Supervisory: Lothar Lanz