[ 
https://issues.apache.org/jira/browse/KUDU-2323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16438143#comment-16438143
 ] 

Mike Percy commented on KUDU-2323:
----------------------------------

Based on a conversation with Todd in Slack the underlying issue still looks 
like a race between adding and removing a replica by the master in cluster. 
Having a delayed or retried DeleteTablet() RPC seems more likely under heavy 
load, and adding back the same node after evicting it seems more likely the 
smaller the cluster is. However we still have not been able to reproduce this 
particular bug.

> NON_VOTER replica flapping (repeatedly added and evicted)
> ---------------------------------------------------------
>
>                 Key: KUDU-2323
>                 URL: https://issues.apache.org/jira/browse/KUDU-2323
>             Project: Kudu
>          Issue Type: Bug
>          Components: consensus
>    Affects Versions: 1.7.0
>            Reporter: Todd Lipcon
>            Assignee: Alexey Serbin
>            Priority: Major
>
> In running a YCSB stress workload I see a tablet got into some state where 
> the master flapped back and forth adding and then removing a replica as a 
> NON_VOTER:
> {code}
> I0221 21:54:35.341892 28047 catalog_manager.cc:3274] Sending 
> ChangeConfig:REMOVE_PEER on tablet 4ffc930e9f2148c38e441f19aa230872 (attempt 
> 1)
> I0221 21:54:35.360297 28045 catalog_manager.cc:3162] Sending 
> ChangeConfig:ADD_PEER:NON_VOTER on tablet 4ffc930e9f2148c38e441f19aa230872 
> (attempt 1)
> I0221 21:54:35.612417 28048 catalog_manager.cc:3274] Sending 
> ChangeConfig:REMOVE_PEER on tablet 4ffc930e9f2148c38e441f19aa230872 (attempt 
> 1)
> I0221 21:54:35.713057 28045 catalog_manager.cc:3162] Sending 
> ChangeConfig:ADD_PEER:NON_VOTER on tablet 4ffc930e9f2148c38e441f19aa230872 
> (attempt 1)
> I0221 21:54:35.725723 28045 catalog_manager.cc:3274] Sending 
> ChangeConfig:REMOVE_PEER on tablet 4ffc930e9f2148c38e441f19aa230872 (attempt 
> 1)
> I0221 21:54:35.752959 28052 catalog_manager.cc:3162] Sending 
> ChangeConfig:ADD_PEER:NON_VOTER on tablet 4ffc930e9f2148c38e441f19aa230872 
> (attempt 1)
> I0221 21:54:35.767974 28047 catalog_manager.cc:3274] Sending 
> ChangeConfig:REMOVE_PEER on tablet 4ffc930e9f2148c38e441f19aa230872 (attempt 
> 1)
> I0221 21:54:35.772202 28045 catalog_manager.cc:3162] Sending 
> ChangeConfig:ADD_PEER:NON_VOTER on tablet 4ffc930e9f2148c38e441f19aa230872 
> (attempt 1)
> I0221 21:54:36.291569 28046 catalog_manager.cc:3274] Sending 
> ChangeConfig:REMOVE_PEER on tablet 4ffc930e9f2148c38e441f19aa230872 (attempt 
> 1)
> I0221 21:54:36.296468 28046 catalog_manager.cc:3162] Sending 
> ChangeConfig:ADD_PEER:NON_VOTER on tablet 4ffc930e9f2148c38e441f19aa230872 
> (attempt 1)
> I0221 21:54:36.328945 28045 catalog_manager.cc:3274] Sending 
> ChangeConfig:REMOVE_PEER on tablet 4ffc930e9f2148c38e441f19aa230872 (attempt 
> 1)
> I0221 21:54:36.339675 28045 catalog_manager.cc:3162] Sending 
> ChangeConfig:ADD_PEER:NON_VOTER on tablet 4ffc930e9f2148c38e441f19aa230872 
> (attempt 1)
> I0221 21:54:36.387465 28045 catalog_manager.cc:3274] Sending 
> ChangeConfig:REMOVE_PEER on tablet 4ffc930e9f2148c38e441f19aa230872 (attempt 
> 1)
> I0221 21:54:36.394716 28047 catalog_manager.cc:3162] Sending 
> ChangeConfig:ADD_PEER:NON_VOTER on tablet 4ffc930e9f2148c38e441f19aa230872 
> (attempt 1)
> I0221 21:54:36.398644 28047 catalog_manager.cc:3274] Sending 
> ChangeConfig:REMOVE_PEER on tablet 4ffc930e9f2148c38e441f19aa230872 (attempt 
> 1)
> I0221 21:54:36.405082 28047 catalog_manager.cc:3162] Sending 
> ChangeConfig:ADD_PEER:NON_VOTER on tablet 4ffc930e9f2148c38e441f19aa230872 
> (attempt 1)
> I0221 21:54:36.409888 28048 catalog_manager.cc:3274] Sending 
> ChangeConfig:REMOVE_PEER on tablet 4ffc930e9f2148c38e441f19aa230872 (attempt 
> 1)
> I0221 21:54:36.414216 28046 catalog_manager.cc:3162] Sending 
> ChangeConfig:ADD_PEER:NON_VOTER on tablet 4ffc930e9f2148c38e441f19aa230872 
> (attempt 1)
> I0221 21:54:36.417915 28048 catalog_manager.cc:3274] Sending 
> ChangeConfig:REMOVE_PEER on tablet 4ffc930e9f2148c38e441f19aa230872 (attempt 
> 1)
> I0221 21:54:36.423548 28048 catalog_manager.cc:3162] Sending 
> ChangeConfig:ADD_PEER:NON_VOTER on tablet 4ffc930e9f2148c38e441f19aa230872 
> (attempt 1)
> I0221 21:54:36.453407 28045 catalog_manager.cc:3274] Sending 
> ChangeConfig:REMOVE_PEER on tablet 4ffc930e9f2148c38e441f19aa230872 (attempt 
> 1)
> I0221 21:54:36.552772 28048 catalog_manager.cc:3162] Sending 
> ChangeConfig:ADD_PEER:NON_VOTER on tablet 4ffc930e9f2148c38e441f19aa230872 
> (attempt 1)
> I0221 21:58:01.300199 28053 catalog_manager.cc:3274] Sending 
> ChangeConfig:REMOVE_PEER on tablet 4ffc930e9f2148c38e441f19aa230872 (attempt 
> 1)
> I0221 21:58:01.426921 28046 catalog_manager.cc:3162] Sending 
> ChangeConfig:ADD_PEER:NON_VOTER on tablet 4ffc930e9f2148c38e441f19aa230872 
> (attempt 1)
> I0221 22:01:37.779790 28051 catalog_manager.cc:3274] Sending 
> ChangeConfig:REMOVE_PEER on tablet 4ffc930e9f2148c38e441f19aa230872 (attempt 
> 1)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to