[
https://issues.apache.org/jira/browse/KUDU-3818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18141562#comment-18141562
]
Alexey Serbin commented on KUDU-3818:
-------------------------------------
IIUC, [this changelist|https://gerrit.cloudera.org/#/c/20262/] should address
the issue, and the approach used there is a bit more universal and broader than
the one proposed in the description of this JIRA item.
> Master can't restart after interrupted sys catalog creation
> ------------------------------------------------------------
>
> Key: KUDU-3818
> URL: https://issues.apache.org/jira/browse/KUDU-3818
> Project: Kudu
> Issue Type: Improvement
> Reporter: Chovan Zoltan
> Assignee: Chovan Zoltan
> Priority: Minor
> Attachments: test-output.txt
>
>
> When a new multi-master cluster starts, each master creates the system
> catalog tablet in SysCatalogTable::CreateNew(). The steps run in this order:
> 1. Write the tablet metadata to disk.
> 2. Resolve the other masters' UUIDs over RPC. This blocks until those masters
> are up, for up to --raft_get_node_instance_timeout_ms (30s by default).
> 3. Write the consensus metadata to disk.
> If the master is stopped or crashes between steps 1 and 3, or if step 2 times
> out, the tablet metadata is on disk but the consensus metadata isn't. Each
> later start then takes the "load existing" path and fails for good:
> {noformat}
> E20261027 13:19:53.309684 31360 master.cc:417] Unable to init master catalog
> manager: Not found: Unable to initialize catalog manager: Failed to
> initialize sys tables async: Unable to load consensus metadata for tablet
> 00000000000000000000000000000000: ...
> /wal/consensus-meta/00000000000000000000000000000000: No such file or
> directory (error 2)
> {noformat}
> The master can't recover without manual action (wiping its data directories),
> even though it never joined the Raft config.
> This showed up as a flaky
> {{TestSecurity.testRenewAndReacquireKeberosCredentials}} in a TSAN pre-commit
> run. The test restarts all masters about a second after the mini cluster
> reports it has started. The mini cluster only waits for each master's RPC
> server to come up, not for the catalog to be created. Under TSAN, two of the
> three masters were still resolving peer UUIDs when they were killed. After
> the restart both failed with the error above, no leader could be elected, and
> {{listTabletServers()}} timed out after 30s ("Master config ... has no
> leader"). The same error also appears earlier in the same log during
> {{testConnectToNonLeaderMasters}}. That test passed only because just one
> master was affected, so the other two still had a majority.
> The full test output from the failed run is attached. The relevant section
> starts at 13:19:40, the masters are restarted at 13:19:51, and the test fails
> at 13:20:31.
> Proposed fix: build the Raft config (including resolving the peer UUIDs)
> before writing anything to disk. Also let {{SysCatalogTable::Load()}} finish
> an interrupted creation, but only when no Raft state (consensus metadata or
> WAL) exists for the sys catalog tablet, i.e. the master has never been part
> of the Raft config.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)