Alexey Serbin has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/17691 )
Change subject: [master] make automatic AddMaster resilient to leadership changes ...................................................................... [master] make automatic AddMaster resilient to leadership changes AutoAddMasterTest.TestAddWithOnGoingDdl would previously fail with the new master shutting down with the following error: Remote error: Failed to perform AddMaster RPC: Illegal state: Failed initiating master Raft ChangeConfig request, error: unknown: Leader has not yet committed an operation in its own term This patch addresses this by catching errors commonly seen in the Raft layer and retrying, as we might do for DML operations. Without this patch, the test failed ~10% of the time in debug mode with --stress_cpu_threads=3. With the patch, the test only fails ~1% of the time, which will be addressed in a later patch. This patch also addresses some leftover feedback from 7e66534d0e62fb850bf300d52da4a0a76889f4b8 regarding verification of masters in the presence of DNS failures. I didn't add a new test for this because it turned out to not change the behavior of things -- the new master would still fail upon attempting to resolve replicas. Change-Id: Ie38453c6fc41ce98c59c010902e2d9fe9db62dee Reviewed-on: http://gerrit.cloudera.org:8080/17691 Tested-by: Kudu Jenkins Reviewed-by: Bankim Bhavsar <[email protected]> Reviewed-by: Alexey Serbin <[email protected]> --- M src/kudu/master/dynamic_multi_master-test.cc M src/kudu/master/master.cc M src/kudu/master/master_runner.cc M src/kudu/master/master_service.cc 4 files changed, 38 insertions(+), 12 deletions(-) Approvals: Kudu Jenkins: Verified Bankim Bhavsar: Looks good to me, approved Alexey Serbin: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/17691 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ie38453c6fc41ce98c59c010902e2d9fe9db62dee Gerrit-Change-Number: 17691 Gerrit-PatchSet: 3 Gerrit-Owner: Andrew Wong <[email protected]> Gerrit-Reviewer: Alexey Serbin <[email protected]> Gerrit-Reviewer: Bankim Bhavsar <[email protected]> Gerrit-Reviewer: Kudu Jenkins (120)
