Re: Unreliable JMX metrics

[kurt Greaves]

Yes. You likely will still be able to see the nodes in nodetool gossipinfo

Re: Unreliable JMX metrics

[Sun, Guan]

Thanks for you reply Malte.
I replaced the nodes and the new nodes have new IP, but I have removed the dead 
nodes using nodetool. Does Cassandra still keep the info of dead nodes for 72 
hours even I have removed them?

Thanks,
Guan

From: Malte Pickhan <malte.pick...@zalando.de<mailto:malte.pick...@zalando.de>>
Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" 
<user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
Date: Tuesday, January 17, 2017 at 7:48 AM
To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" 
<user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
Subject: Re: Unreliable JMX metrics

Hey Guan,

If it's the case that you actually replaced the nodes and already assigned new 
IP's to them this is probably common behaviour. Since Cassandra has a retention 
policy to keep dead nodes for 72 hours.

Best,

Malte

2017-01-17 0:29 GMT+01:00 Sun, Guan 
<guan...@amazon.com<mailto:guan...@amazon.com>>:
Hi all,

We have a 6 node Cassandra cluster running on Amazon EC2. We replaced the nodes 
one by one while Cassandra was running several days ago. After the replacement, 
the nodetool status show that all the nodes were up and running. However, the 
JMX metrics about down nodes kept showing there were 6 nodes down for about 2.5 
days before it became 0.

Has anyone seen this before? Any idea why the JMX metrics takes so long to 
reflect the actual status of the cluster?

Thanks,
Guan



--
Malte Pickhan

Software Engineer

Zalando Payments SE & Co. KG

POSTAL ADDRESS
Zalando Payments SE & Co. KG
11501 Berlin

OFFICE
Zalando Payments SE & Co. KG
Hörder Hafenstraße 1-3
44263 Dortmund
Germany


E-Mail: malte.pick...@zalando.de<mailto:malte.pick...@zalando.de>
Web: corporate.zalando.de<http://corporate.zalando.de/>
Jobs: jobs.zalando.de<http://jobs.zalando.de/>

Zalando Payments SE & Co. KG
Company registration: Amtsgericht Charlottenburg, HRA 48907 B

VAT registration number: DE 289683421



General partner: Zalando SE, registered office: Berlin

Company registration: Amtsgericht Charlottenburg, HRA 48907 B

Management Board: Robert Gentz, David Schneider, Rubin Ritter
Chairperson of Supervisory: Lothar Lanz

Re: Unreliable JMX metrics

[Malte Pickhan]

Hey Guan,

If it's the case that you actually replaced the nodes and already assigned
new IP's to them this is probably common behaviour. Since Cassandra has a
retention policy to keep dead nodes for 72 hours.

Best,

Malte

2017-01-17 0:29 GMT+01:00 Sun, Guan :

> Hi all,
>
> We have a 6 node Cassandra cluster running on Amazon EC2. We replaced the
> nodes one by one while Cassandra was running several days ago. After the
> replacement, the nodetool status show that all the nodes were up and
> running. However, the JMX metrics about down nodes kept showing there were
> 6 nodes down for about 2.5 days before it became 0.
>
> Has anyone seen this before? Any idea why the JMX metrics takes so long to
> reflect the actual status of the cluster?
>
> Thanks,
> Guan
>



-- 
Malte Pickhan

Software Engineer

Zalando Payments SE & Co. KG

*POSTAL ADDRESS*
Zalando Payments SE & Co. KG
11501 Berlin

*OFFICE*
Zalando Payments SE & Co. KG
Hörder Hafenstraße 1-3
44263 Dortmund
Germany


E-Mail: malte.pick...@zalando.de
Web: *corporate.zalando.de *
Jobs: jobs.zalando.de

Zalando Payments SE & Co. KG
Company registration: Amtsgericht Charlottenburg, HRA 48907 B

VAT registration number: DE 289683421



General partner: Zalando SE, registered office: Berlin

Company registration: Amtsgericht Charlottenburg, HRA 48907 B

Management Board: Robert Gentz, David Schneider, Rubin Ritter
Chairperson of Supervisory: Lothar Lanz