[
https://issues.apache.org/jira/browse/KUDU-2323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16414801#comment-16414801
]
Mike Percy commented on KUDU-2323:
----------------------------------
While the speed of this cycle was likely fixed by the patch to fix KUDU-2320,
it appears there is no code path to remove a TrackedPeer when it gets evicted.
While this could cause a minor resource leak until a leader was evicted in a
3-2-3 world, in a 3-4-3 world it affects last_communcation_time and can
therefore make a downed NON_VOTER to be considered FAILED as soon as it is
added to the config.
Maybe this is also interacting with KUDU-2354, in which there are certain cases
that can cause a catalog manager task to endlessly retry adding a new replica.
> NON_VOTER replica flapping (repeatedly added and evicted)
> ---------------------------------------------------------
>
> Key: KUDU-2323
> URL: https://issues.apache.org/jira/browse/KUDU-2323
> Project: Kudu
> Issue Type: Bug
> Components: consensus
> Affects Versions: 1.7.0
> Reporter: Todd Lipcon
> Assignee: Alexey Serbin
> Priority: Major
>
> In running a YCSB stress workload I see a tablet got into some state where
> the master flapped back and forth adding and then removing a replica as a
> NON_VOTER:
> {code}
> I0221 21:54:35.341892 28047 catalog_manager.cc:3274] Sending
> ChangeConfig:REMOVE_PEER on tablet 4ffc930e9f2148c38e441f19aa230872 (attempt
> 1)
> I0221 21:54:35.360297 28045 catalog_manager.cc:3162] Sending
> ChangeConfig:ADD_PEER:NON_VOTER on tablet 4ffc930e9f2148c38e441f19aa230872
> (attempt 1)
> I0221 21:54:35.612417 28048 catalog_manager.cc:3274] Sending
> ChangeConfig:REMOVE_PEER on tablet 4ffc930e9f2148c38e441f19aa230872 (attempt
> 1)
> I0221 21:54:35.713057 28045 catalog_manager.cc:3162] Sending
> ChangeConfig:ADD_PEER:NON_VOTER on tablet 4ffc930e9f2148c38e441f19aa230872
> (attempt 1)
> I0221 21:54:35.725723 28045 catalog_manager.cc:3274] Sending
> ChangeConfig:REMOVE_PEER on tablet 4ffc930e9f2148c38e441f19aa230872 (attempt
> 1)
> I0221 21:54:35.752959 28052 catalog_manager.cc:3162] Sending
> ChangeConfig:ADD_PEER:NON_VOTER on tablet 4ffc930e9f2148c38e441f19aa230872
> (attempt 1)
> I0221 21:54:35.767974 28047 catalog_manager.cc:3274] Sending
> ChangeConfig:REMOVE_PEER on tablet 4ffc930e9f2148c38e441f19aa230872 (attempt
> 1)
> I0221 21:54:35.772202 28045 catalog_manager.cc:3162] Sending
> ChangeConfig:ADD_PEER:NON_VOTER on tablet 4ffc930e9f2148c38e441f19aa230872
> (attempt 1)
> I0221 21:54:36.291569 28046 catalog_manager.cc:3274] Sending
> ChangeConfig:REMOVE_PEER on tablet 4ffc930e9f2148c38e441f19aa230872 (attempt
> 1)
> I0221 21:54:36.296468 28046 catalog_manager.cc:3162] Sending
> ChangeConfig:ADD_PEER:NON_VOTER on tablet 4ffc930e9f2148c38e441f19aa230872
> (attempt 1)
> I0221 21:54:36.328945 28045 catalog_manager.cc:3274] Sending
> ChangeConfig:REMOVE_PEER on tablet 4ffc930e9f2148c38e441f19aa230872 (attempt
> 1)
> I0221 21:54:36.339675 28045 catalog_manager.cc:3162] Sending
> ChangeConfig:ADD_PEER:NON_VOTER on tablet 4ffc930e9f2148c38e441f19aa230872
> (attempt 1)
> I0221 21:54:36.387465 28045 catalog_manager.cc:3274] Sending
> ChangeConfig:REMOVE_PEER on tablet 4ffc930e9f2148c38e441f19aa230872 (attempt
> 1)
> I0221 21:54:36.394716 28047 catalog_manager.cc:3162] Sending
> ChangeConfig:ADD_PEER:NON_VOTER on tablet 4ffc930e9f2148c38e441f19aa230872
> (attempt 1)
> I0221 21:54:36.398644 28047 catalog_manager.cc:3274] Sending
> ChangeConfig:REMOVE_PEER on tablet 4ffc930e9f2148c38e441f19aa230872 (attempt
> 1)
> I0221 21:54:36.405082 28047 catalog_manager.cc:3162] Sending
> ChangeConfig:ADD_PEER:NON_VOTER on tablet 4ffc930e9f2148c38e441f19aa230872
> (attempt 1)
> I0221 21:54:36.409888 28048 catalog_manager.cc:3274] Sending
> ChangeConfig:REMOVE_PEER on tablet 4ffc930e9f2148c38e441f19aa230872 (attempt
> 1)
> I0221 21:54:36.414216 28046 catalog_manager.cc:3162] Sending
> ChangeConfig:ADD_PEER:NON_VOTER on tablet 4ffc930e9f2148c38e441f19aa230872
> (attempt 1)
> I0221 21:54:36.417915 28048 catalog_manager.cc:3274] Sending
> ChangeConfig:REMOVE_PEER on tablet 4ffc930e9f2148c38e441f19aa230872 (attempt
> 1)
> I0221 21:54:36.423548 28048 catalog_manager.cc:3162] Sending
> ChangeConfig:ADD_PEER:NON_VOTER on tablet 4ffc930e9f2148c38e441f19aa230872
> (attempt 1)
> I0221 21:54:36.453407 28045 catalog_manager.cc:3274] Sending
> ChangeConfig:REMOVE_PEER on tablet 4ffc930e9f2148c38e441f19aa230872 (attempt
> 1)
> I0221 21:54:36.552772 28048 catalog_manager.cc:3162] Sending
> ChangeConfig:ADD_PEER:NON_VOTER on tablet 4ffc930e9f2148c38e441f19aa230872
> (attempt 1)
> I0221 21:58:01.300199 28053 catalog_manager.cc:3274] Sending
> ChangeConfig:REMOVE_PEER on tablet 4ffc930e9f2148c38e441f19aa230872 (attempt
> 1)
> I0221 21:58:01.426921 28046 catalog_manager.cc:3162] Sending
> ChangeConfig:ADD_PEER:NON_VOTER on tablet 4ffc930e9f2148c38e441f19aa230872
> (attempt 1)
> I0221 22:01:37.779790 28051 catalog_manager.cc:3274] Sending
> ChangeConfig:REMOVE_PEER on tablet 4ffc930e9f2148c38e441f19aa230872 (attempt
> 1)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
