[
https://issues.apache.org/jira/browse/KUDU-1391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15231510#comment-15231510
]
Mike Percy commented on KUDU-1391:
----------------------------------
Just thought through this some more. Currently we're not protected from split
brain if we let non-leaders remote bootstrap because of the following RPC
signature:
{code}
message StartRemoteBootstrapRequestPB {
// UUID of server this request is addressed to.
optional bytes dest_uuid = 5;
required bytes tablet_id = 1;
// Identification for the host we are bootstrapping from.
// TODO: Consider renaming these to bootstrap_source_*.
required bytes bootstrap_peer_uuid = 2;
required HostPortPB bootstrap_peer_addr = 3;
// The caller's term. In the case that the target of this request has a
// TOMBSTONED replica with a term higher than this one, the request will fail.
optional int64 caller_term = 4 [ default = -1 ];
}
{code}
In this case, caller_term is not good enough (if we're not leader) since term
is not serialized through the cluster. To avoid a zombie candidate from
auto-vivifying his own split-brain tablet, we need to do a CAS based on
committed config version, not caller term. Caller term is only good enough if
we have a zombie leader and the leader is the only one who can initiate remote
bootstrap.
> 2 of 3 replica alive but failed to elect leader
> -----------------------------------------------
>
> Key: KUDU-1391
> URL: https://issues.apache.org/jira/browse/KUDU-1391
> Project: Kudu
> Issue Type: Bug
> Reporter: Binglin Chang
> Attachments: 6a32cfa0353e4175809c2aa67e16ac9e.log.st172,
> 6a32cfa0353e4175809c2aa67e16ac9e.log.st212,
> 6a32cfa0353e4175809c2aa67e16ac9e.log.st212.before,
> 6a32cfa0353e4175809c2aa67e16ac9e.log.st216, remote-bootstrap-tool.patch
>
>
> Last weekend many TS have a lot too many open files error(haven't upgrade to
> , when using our internal deploy tool to restart cluster (stop all ts, then
> start all ts), the control machine have some issue which seems to block or
> write to ssh terminal(maybe usb driver issue, not related to this bug), so
> only half (about 30) of the TS is shutdown, then after maybe 10 minutes, I
> switch to another control host and perform the whole restart.
> Then I see writes are blocked, because 1 tablet is in no leader state, from
> web-ui, 2 of 3 replicas is in follower state, 1 TABLET_DATA_TOMBSTONED, but
> all election failed, will attach the log of the 2 followers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
