Mike Percy has submitted this change and it was merged. Change subject: KUDU-2123. Auto-vivify cmeta on tombstoned replicas if doesn't exist at startup ......................................................................
KUDU-2123. Auto-vivify cmeta on tombstoned replicas if doesn't exist at startup It is possible for tombstoned replicas to legitimately not have a cmeta file as a result of crashing during a first tablet copy, or failing a tablet copy operation in an older version of Kudu. Not having a cmeta file results in those tombstoned replicas being unable to vote in Raft leader elections. We remedy this by creating a cmeta object (with an empty config) at startup time. The empty config is safe for a tombstoned replica, because the config doesn't affect a replica's ability to vote in a leader election. Additionally, if the tombstoned replica were ever to be overwritten by a tablet copy operation, that would also result in overwriting the config stored in the local cmeta with a valid Raft config. Finally, all of this assumes that the nonexistence of a cmeta file guarantees that the replica has never voted in a leader election. As an optimization, the cmeta is created with the NO_FLUSH_ON_CREATE flag, meaning that it will only be flushed to disk if the replica ever votes. The following changes had to be made to ConsensusMetadata and the ConsensusMetadataManager to support the above functionality: * Enable deferred flush on Create() by defining a flag called NO_FLUSH_ON_CREATE * Made some additional method arguments optional, for convenience. The following tests have been added: * A unit test for ConsensusMetadataManager::LoadOrCreate(). * A unit test for ConsensusMetadataCreateMode::NO_FLUSH_ON_CREATE. * A test that crashes the target of a tablet copy after writing the superblock and before writing the cmeta file. The tablet server is restarted and the replica is expected to be able to vote while tombstoned. Previously-written tests that verify ConsensusMetadata::Create() will not clobber an existing file still pass, and an additional test was added for unflushed cmeta instances. Change-Id: I8ff6255b1fcbb12417b82853bcde9b239291492b Reviewed-on: http://gerrit.cloudera.org:8080/7988 Reviewed-by: Andrew Wong <[email protected]> Reviewed-by: Alexey Serbin <[email protected]> Tested-by: Kudu Jenkins --- M src/kudu/consensus/consensus_meta-test.cc M src/kudu/consensus/consensus_meta.cc M src/kudu/consensus/consensus_meta.h M src/kudu/consensus/consensus_meta_manager-test.cc M src/kudu/consensus/consensus_meta_manager.cc M src/kudu/consensus/consensus_meta_manager.h M src/kudu/consensus/raft_consensus_quorum-test.cc M src/kudu/integration-tests/CMakeLists.txt A src/kudu/integration-tests/tombstoned_voting-itest.cc M src/kudu/master/sys_catalog.cc M src/kudu/tablet/tablet_bootstrap-test.cc M src/kudu/tablet/tablet_replica-test.cc M src/kudu/tserver/tablet_copy_client.cc M src/kudu/tserver/tablet_copy_source_session-test.cc M src/kudu/tserver/ts_tablet_manager.cc 15 files changed, 370 insertions(+), 67 deletions(-) Approvals: Andrew Wong: Looks good to me, but someone else must approve Alexey Serbin: Looks good to me, approved Kudu Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/7988 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: merged Gerrit-Change-Id: I8ff6255b1fcbb12417b82853bcde9b239291492b Gerrit-PatchSet: 10 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Mike Percy <[email protected]> Gerrit-Reviewer: Adar Dembo <[email protected]> Gerrit-Reviewer: Alexey Serbin <[email protected]> Gerrit-Reviewer: Andrew Wong <[email protected]> Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Mike Percy <[email protected]> Gerrit-Reviewer: Tidy Bot Gerrit-Reviewer: Todd Lipcon <[email protected]>
