[
https://issues.apache.org/jira/browse/KUDU-1391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15225845#comment-15225845
]
Todd Lipcon commented on KUDU-1391:
-----------------------------------
Here's some analysis of the logs:
{noformat}
nodes:
---------------------
bfdd1 (st198)
edc90 (st212)
a8f15 (st216)
39195 (st172)
timeline:
--------------------------
while edc906 was leader:
21:07:20 bfdd1 went down (starts getting conn refused)
21:12:20 opid 1997024: remove peer bfdd1 (dead for 5 minutes)
- master sees under-replication, suggests to add 39195
21:12:20 opid 1997025: add peer 39195
21:12:20 39195 starts bootstrapping data from edc906
21:12:22: a8f15 went down (started getting conn refused)
-- at this point, there are no live followers (a8f15 is down, 39195 is still
bootstrapping)
21:17:21: tries to remove a8f15, but can't commit anything yet
21:26:38 39195 went down before it finished bootstrapping
22:05:57 39195 restarted with the incomplete copy, so it deleted it on startup
current state
------------------
peer a8f15 still has config:
peers in config: edc906, a8f15, 39195
- this _should_ be able to commit by getting a vote from itself and edc906
peer edc906 has a pending config which removed a8f15
peers in config: edc906, 39195
- this can't commit, because it needs 2/2 majority and 39195 is not done
bootstrapping
peer 39195 (st172.bj)
got restarted when it was in the middle of bootstrapping, so it
deleted the data on reboot
- this can't vote or elect because it has no data.
why is election stuck?
-------------------------
- a8f15 and edc906 seem to be taking turns attempting elections
- edc906 has a higher term, so rejects all vote requests by a8f15
- edc906 can't win an election because its pending config is itself and a
missing node
- a8f15 should be able to win an election by getting a vote from edc906, but
there's
an apparent timing issue: edc906 has a higher term, and they are both
incrementing
their terms at the same rate. a "I have a higher term" vote rejection doesn't
seem
to force incrementing the term on the candidate.
{noformat}
> 2 of 3 replica alive but failed to elect leader
> -----------------------------------------------
>
> Key: KUDU-1391
> URL: https://issues.apache.org/jira/browse/KUDU-1391
> Project: Kudu
> Issue Type: Bug
> Reporter: Binglin Chang
> Attachments: 6a32cfa0353e4175809c2aa67e16ac9e.log.st172,
> 6a32cfa0353e4175809c2aa67e16ac9e.log.st212,
> 6a32cfa0353e4175809c2aa67e16ac9e.log.st212.before,
> 6a32cfa0353e4175809c2aa67e16ac9e.log.st216
>
>
> Last weekend many TS have a lot too many open files error(haven't upgrade to
> , when using our internal deploy tool to restart cluster (stop all ts, then
> start all ts), the control machine have some issue which seems to block or
> write to ssh terminal(maybe usb driver issue, not related to this bug), so
> only half (about 30) of the TS is shutdown, then after maybe 10 minutes, I
> switch to another control host and perform the whole restart.
> Then I see writes are blocked, because 1 tablet is in no leader state, from
> web-ui, 2 of 3 replicas is in follower state, 1 TABLET_DATA_TOMBSTONED, but
> all election failed, will attach the log of the 2 followers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
