[
https://issues.apache.org/jira/browse/KUDU-1358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15402965#comment-15402965
]
Adar Dembo commented on KUDU-1358:
----------------------------------
Mostly fixed in commits fb2022c, 284591a, and 8618ae2.
But I'm leaving the JIRA open because the window of time is still open if:
# The leader restarts quickly enough that none of its followers notice that it
"failed".
# The leader is restarted and is immediately reelected leader.
To address this, we're thinking of adding a "safe mode" to the master. Safe
mode would be entered automatically upon restart and exited once the master is
confident that it can make forward progress in the cluster (i.e. after some
percentage of tservers have heartbeated, or perhaps after some amount of time
has passed). When in safe mode, CreateTable() requests will fail and clients
should retry them with some backoff.
> Following a master leader election, create table may fail
> ---------------------------------------------------------
>
> Key: KUDU-1358
> URL: https://issues.apache.org/jira/browse/KUDU-1358
> Project: Kudu
> Issue Type: Sub-task
> Components: master
> Affects Versions: 0.7.0
> Reporter: Adar Dembo
> Assignee: Adar Dembo
> Priority: Critical
>
> In the current multi-master design and implementation, tservers only
> heartbeat to the leader master. After a master leader election, there's a
> short window of time in which the new leader master may not be aware of the
> existence of some (or even all) of the tservers. Attempts to create a table
> during this window may fail, as the tservers known to the new leader master
> may be too few to satisfy the new table's replication factor. Whether the
> window exists in the first place depends on whether the new leader master had
> been leader before, and whether any of the tservers had sent heartbeats to it
> during that time.
> Some possible solutions include:
> # Modifying the heartbeat protocol so that tservers heartbeat to _all_
> masters, leaders and followers alike. Doing this will ensure that the "soft
> state" belonging to any master is always up-to-date at the cost of network
> bandwidth lost to heartbeating. Additionally, changes may need to be made to
> ensure that a follower master can't cause a tserver to take any actions.
> # Never actually failing a create table request due to too few tservers,
> instead allowing it to linger until such a time when more tservers exist. For
> this to actually be practical we'd need to allow clients to "cancel" a
> previously issued create table request.
> Both approaches probably include additional ramifications; this problem needs
> to be thought through carefully.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
