[
https://issues.apache.org/jira/browse/KUDU-3010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16984114#comment-16984114
]
Andrew Wong commented on KUDU-3010:
-----------------------------------
I spent some time rummaging through logs and pulled out the relevant snippet
targeting just a specific tablet ID.
As we suspected, what's happening is related to the lack of any leader-checking
done by {{unsafe_change_config}}:
Starting around, 08:08:55:
* [f06, f03, f04] are in the same Raft group for term 60.
* f06 gets an {{unsafe_change_config}} request to become [f06], and because
it's a single-replica election, it wins immediately and becomes leader of term
61 for [f06].
* At the same time, f04 requests to become leader of term 61. f03 votes yes and
f04 becomes leader of term 61 for [f06, f03, f04].
* Raft crashes on f06 and f03 because there's a mismatch of who is leader.
Why was {{unsafe_change_config}} run, when there were healthy replicas?
Unclear. I'm not sure how we make this better, given it's documented as unsafe.
Maybe better publicity of how unsafe it is? Warnings to only run the tool to
remove servers that are down?
> unsafe_change_config can lead to a crash
> ----------------------------------------
>
> Key: KUDU-3010
> URL: https://issues.apache.org/jira/browse/KUDU-3010
> Project: Kudu
> Issue Type: Bug
> Components: consensus, ops-tooling
> Reporter: Andrew Wong
> Priority: Major
> Attachments: foo03.out, foo04.out, foo06.out
>
>
> I've seen a case of running the {{unsafe_change_config}} tool, per the steps
> laid out in the ["Bringing a tablet that has lost a majority of
> replicas"|https://kudu.apache.org/docs/administration.html#tablet_majority_down_recovery]
> steps, crashing a tserver with the following error:
> {code:java}
> I1028 08:24:31.241361 38436 raft_consensus.cc:684] T
> b90b0429806747a6b993d8543ab5fd50 P f344ade17ed94072b8839007ccc7570a [term 40
> FOLLOWER]: Illegal state: RaftConfig change currently pending. Only one is
> allowed at a time.
> W1028 08:24:31.241379 38436 raft_consensus.cc:1373] T
> b90b0429806747a6b993d8543ab5fd50 P f344ade17ed94072b8839007ccc7570a [term 40
> FOLLOWER]: Could not prepare transaction for op 34.48 and following 69 ops.
> Status for this op: Illegal state: RaftConfig change currently pending. Only
> one is allowed at a time.
> I1028 08:26:07.300520 38436 raft_consensus.cc:1058] T
> a6bfa86e43b74cfaa6feba4631879251 P f344ade17ed94072b8839007ccc7570a [term 17
> FOLLOWER]: Refusing update from remote peer f1a7fb14b7b44a5c8b31e93114d79a8d:
> Log matching property violated. Preceding OpId in replica: term: 15 index:
> 93. Preceding OpId from leader: term: 17 index: 112. (index mismatch)
> I1028 08:26:07.301476 38436 raft_consensus.cc:2819] T
> a6bfa86e43b74cfaa6feba4631879251 P f344ade17ed94072b8839007ccc7570a [term 17
> NON_PARTICIPANT]: Allowing unsafe config change even though there is a
> pending config! Existing pending config: opid_index: 95 OBSOLETE_local: false
> peers { permanent_uuid: "7875cc5598a44bd893998cba7bd2cc47" member_type: VOTER
> last_known_addr{ host: "foo01.server.net" port: 7050 }
> attrs{ promote: false }
> } peers { permanent_uuid: "f1a7fb14b7b44a5c8b31e93114d79a8d" member_type:
> NON_VOTER last_known_addr{ host: "foo04.server.net" port: 7050 }
> attrs{ promote: true }
> } unsafe_config_change: true; New pending config: opid_index: 96
> OBSOLETE_local: false peers { permanent_uuid:
> "7875cc5598a44bd893998cba7bd2cc47" member_type: VOTER last_known_addr{ host:
> "foo01.server.net" port: 7050 }
> attrs{ promote: false }
> } peers { permanent_uuid: "f1a7fb14b7b44a5c8b31e93114d79a8d" member_type:
> NON_VOTER last_known_addr{ host: "foo04.server.net" port: 7050 }
> attrs{ promote: true }
> } peers { permanent_uuid: "231e6fdad22647978c9a76c07407da4c" member_type:
> NON_VOTER last_known_addr{ host: "foo02.server.net" port: 7050 }
> attrs{ promote: true }
> } unsafe_config_change: true
> F1028 08:26:07.302338 38436 pending_rounds.cc:179] Check failed: _s.ok() Bad
> status: Corruption: New operation's term is not >= than the previous op's
> term. Current: 14.94. Previous: 15.93
> {code}
> It seems like the tool is permitting the persistence of a bad op, considering
> there's already a config change in flight.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
