Re: Repeated messages about Removing tokens

[Jai Bheemsen Rao Dhanwada]

Yes correct, for the same node the log message is repeating

On Monday, October 5, 2020, Erick Ramirez 
wrote:

> Yes, This log entry is expected during the decommission and happens only
>> once. But in my case I see these messages repeating several 1000 times a
>> day and continuing. Even the restart of C* process didn't work.
>>
>
> Are you saying that the same node is repeatedly reported as going down?
>

Re: Repeated messages about Removing tokens

[Erick Ramirez]

>
> Yes, This log entry is expected during the decommission and happens only
> once. But in my case I see these messages repeating several 1000 times a
> day and continuing. Even the restart of C* process didn't work.
>

Are you saying that the same node is repeatedly reported as going down?

Re: Repeated messages about Removing tokens

[Jai Bheemsen Rao Dhanwada]

Yes, This log entry is expected during the decommission and happens only
once. But in my case I see these messages repeating several 1000 times a
day and continuing. Even the restart of C* process didn't work.

On Sun, Oct 4, 2020 at 11:49 PM Erick Ramirez 
wrote:

> yes correct, I gracefully decommissioned a few nodes from the cluster and
>> I see the below log messages for all the decommissioned servers.
>>
>> INFO  [GossipStage:51] 2020-10-05 05:54:10,429 Gossiper.java:1010 -
>>> InetAddress /x.x.x.x is now DOWN
>>> INFO  [GossipStage:51] 2020-10-05 05:54:10,430 StorageService.java:1965
>>> - Removing tokens [100] for /x.x.x.x
>>
>>
> That's expected everytime you decommission a node. The tokens the node
> used to own get "removed" meaning the decommissioned node no longer owns
> it. Those log entries are normal behaviour for a cluster. Cheers!
>

Re: Repeated messages about Removing tokens

[Erick Ramirez]

>
> yes correct, I gracefully decommissioned a few nodes from the cluster and
> I see the below log messages for all the decommissioned servers.
>
> INFO  [GossipStage:51] 2020-10-05 05:54:10,429 Gossiper.java:1010 -
>> InetAddress /x.x.x.x is now DOWN
>> INFO  [GossipStage:51] 2020-10-05 05:54:10,430 StorageService.java:1965 -
>> Removing tokens [100] for /x.x.x.x
>
>
That's expected everytime you decommission a node. The tokens the node used
to own get "removed" meaning the decommissioned node no longer owns it.
Those log entries are normal behaviour for a cluster. Cheers!

Re: Repeated messages about Removing tokens

[Jai Bheemsen Rao Dhanwada]

yes correct, I gracefully decommissioned a few nodes from the cluster and I
see the below log messages for all the decommissioned servers.

INFO  [GossipStage:51] 2020-10-05 05:54:10,429 Gossiper.java:1010 -
> InetAddress /x.x.x.x is now DOWN
> INFO  [GossipStage:51] 2020-10-05 05:54:10,430 StorageService.java:1965 -
> Removing tokens [100] for /x.x.x.x


On Sun, Oct 4, 2020 at 10:39 PM Erick Ramirez 
wrote:

> any other pointers here? I have performed rolling restart of all the nodes
>> in the cluster but still see these messages repeatedly. it's been more than
>> 30 days the nodes were decommissioned gracefully.
>>
>
> Can you provide an update on what you're seeing? Are you saying you're
> still seeing "Removing tokens" for the same token *AND* node IP?
>

Re: Repeated messages about Removing tokens

[Erick Ramirez]

>
> any other pointers here? I have performed rolling restart of all the nodes
> in the cluster but still see these messages repeatedly. it's been more than
> 30 days the nodes were decommissioned gracefully.
>

Can you provide an update on what you're seeing? Are you saying you're
still seeing "Removing tokens" for the same token *AND* node IP?

Re: Repeated messages about Removing tokens

[Jai Bheemsen Rao Dhanwada]

any other pointers here? I have performed rolling restart of all the nodes
in the cluster but still see these messages repeatedly. it's been more than
30 days the nodes were decommissioned gracefully.

C* version: 2.1.21, is there any known bug with this version?

On Wed, Aug 26, 2020 at 9:44 AM Jai Bheemsen Rao Dhanwada <
jaibheem...@gmail.com> wrote:

> No, I always performed addition and decommission one node at a time.
>
> On Wed, Aug 26, 2020 at 5:42 AM Hossein Ghiyasi Mehr <
> ghiyasim...@gmail.com> wrote:
>
>> Is it possible to remove/decommission/add nodes at the same time? You
>> can't add twos nodes at the same time.
>> Do you decommission a node while you are adding another node?
>>
>> *---*
>> *VafaTech  : A Total Solution for Data Gathering
>> & Analysis*
>> *---*
>>
>>
>> On Mon, Aug 24, 2020 at 2:37 AM Paulo Motta 
>> wrote:
>>
>>> These messages should go away when the decommission/removenode is
>>> complete. Are you seeing them repeating for the same nodes after they've
>>> left or do they eventually stop?
>>>
>>> If not this is expected behavior but perhaps a bit too verbose if the
>>> message is being printed more than once per node removal (in which case you
>>> should probably open a JIRA to fix it).
>>>
>>> Em qua., 19 de ago. de 2020 às 02:30, Jai Bheemsen Rao Dhanwada <
>>> jaibheem...@gmail.com> escreveu:
>>>
 Yes correct i have seen these messages but in one of cluster i see
 these messages are repeated ~3000 times per day

 On Tuesday, August 18, 2020, Erick Ramirez 
 wrote:

> Yes, it is normal to see that message when you are decommissioning
> nodes. It just means that the token ownership(s) is getting updated. 
> Cheers!
>
>>

Re: Repeated messages about Removing tokens

[Jai Bheemsen Rao Dhanwada]

No, I always performed addition and decommission one node at a time.

On Wed, Aug 26, 2020 at 5:42 AM Hossein Ghiyasi Mehr 
wrote:

> Is it possible to remove/decommission/add nodes at the same time? You
> can't add twos nodes at the same time.
> Do you decommission a node while you are adding another node?
>
> *---*
> *VafaTech  : A Total Solution for Data Gathering
> & Analysis*
> *---*
>
>
> On Mon, Aug 24, 2020 at 2:37 AM Paulo Motta 
> wrote:
>
>> These messages should go away when the decommission/removenode is
>> complete. Are you seeing them repeating for the same nodes after they've
>> left or do they eventually stop?
>>
>> If not this is expected behavior but perhaps a bit too verbose if the
>> message is being printed more than once per node removal (in which case you
>> should probably open a JIRA to fix it).
>>
>> Em qua., 19 de ago. de 2020 às 02:30, Jai Bheemsen Rao Dhanwada <
>> jaibheem...@gmail.com> escreveu:
>>
>>> Yes correct i have seen these messages but in one of cluster i see these
>>> messages are repeated ~3000 times per day
>>>
>>> On Tuesday, August 18, 2020, Erick Ramirez 
>>> wrote:
>>>
 Yes, it is normal to see that message when you are decommissioning
 nodes. It just means that the token ownership(s) is getting updated. 
 Cheers!

>

Re: Repeated messages about Removing tokens

[Hossein Ghiyasi Mehr]

Is it possible to remove/decommission/add nodes at the same time? You can't
add twos nodes at the same time.
Do you decommission a node while you are adding another node?

*---*
*VafaTech  : A Total Solution for Data Gathering &
Analysis*
*---*


On Mon, Aug 24, 2020 at 2:37 AM Paulo Motta 
wrote:

> These messages should go away when the decommission/removenode is
> complete. Are you seeing them repeating for the same nodes after they've
> left or do they eventually stop?
>
> If not this is expected behavior but perhaps a bit too verbose if the
> message is being printed more than once per node removal (in which case you
> should probably open a JIRA to fix it).
>
> Em qua., 19 de ago. de 2020 às 02:30, Jai Bheemsen Rao Dhanwada <
> jaibheem...@gmail.com> escreveu:
>
>> Yes correct i have seen these messages but in one of cluster i see these
>> messages are repeated ~3000 times per day
>>
>> On Tuesday, August 18, 2020, Erick Ramirez 
>> wrote:
>>
>>> Yes, it is normal to see that message when you are decommissioning
>>> nodes. It just means that the token ownership(s) is getting updated. Cheers!
>>>

Re: Repeated messages about Removing tokens

[Paulo Motta]

These messages should go away when the decommission/removenode is complete.
Are you seeing them repeating for the same nodes after they've left or do
they eventually stop?

If not this is expected behavior but perhaps a bit too verbose if the
message is being printed more than once per node removal (in which case you
should probably open a JIRA to fix it).

Em qua., 19 de ago. de 2020 às 02:30, Jai Bheemsen Rao Dhanwada <
jaibheem...@gmail.com> escreveu:

> Yes correct i have seen these messages but in one of cluster i see these
> messages are repeated ~3000 times per day
>
> On Tuesday, August 18, 2020, Erick Ramirez 
> wrote:
>
>> Yes, it is normal to see that message when you are decommissioning nodes.
>> It just means that the token ownership(s) is getting updated. Cheers!
>>
>>>

Re: Repeated messages about Removing tokens

[Jai Bheemsen Rao Dhanwada]

Yes correct i have seen these messages but in one of cluster i see these
messages are repeated ~3000 times per day

On Tuesday, August 18, 2020, Erick Ramirez 
wrote:

> Yes, it is normal to see that message when you are decommissioning nodes.
> It just means that the token ownership(s) is getting updated. Cheers!
>
>>

Re: Repeated messages about Removing tokens

[Erick Ramirez]

Yes, it is normal to see that message when you are decommissioning nodes.
It just means that the token ownership(s) is getting updated. Cheers!

>

Repeated messages about Removing tokens

[Jai Bheemsen Rao Dhanwada]

Hello,

I am trying to add and decommission some nodes into the cluster. While some
nodes are decommissioned I see the below error messages repeatedly on the
rest of C* nodes logs. It's not for all the IPs but for a few of them. Is
this an issue?

INFO  [GossipStage:7] 2020-08-18 04:22:07,168 StorageService.java:1965 -
> Removing tokens [127605887595351923798765477786913079396] for /x.x.x.x


While I see the C* cluster is healthy, sometimes I see the clients are
getting NoHostsAvailableExceptions.

C* Version: 2.1.13
Client: Thrift protocol