Todd Lipcon has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/9867 )
Change subject: KUDU-2361. Fix flaky MasterTest.TestDumpStacksOnRpcQueueOverflow ...................................................................... KUDU-2361. Fix flaky MasterTest.TestDumpStacksOnRpcQueueOverflow This test was flaky in TSAN builds for two reasons: (1) even though taking the stack trace is always pretty fast, symbolization can sometimes take more than a second. If that was the case, the assertion that looked at the diagnostics log content would fail just because the expected log was not there _yet_. Fixing this issue using ASSERT_EVENTUALLY brought the test failure rate with 4 stress threads down from about 60% to about 0.3%. (2) with periodic stack dumping enabled, it was possible that the periodic dump was already in progress at the time that the overflow-triggered dump was triggered. In that case, the overflow-triggered dump would be ignored. That was expected behavior, but the test would fail due to not seeing the overflow listed as the "reason". This patch fixes this issue by disabling periodic dumping in the test. In order to disable periodic dumping but leave triggered dumping enabled, I changed the semantics of the configuration slightly. Previously, setting it to '0' would disable all dumping. Now, setting it to '0' disables periodic dumping but still leaves triggered dumps enabled. Setting it to '-1' disables all dumping. With both these changes I was able to loop the test 300 times with 4 stress threads in TSAN mode with no failures. Change-Id: If7078bc87d2cf724d5f2fddf1173ac1786279e7e Reviewed-on: http://gerrit.cloudera.org:8080/9867 Reviewed-by: Adar Dembo <a...@cloudera.com> Tested-by: Todd Lipcon <t...@apache.org> --- M src/kudu/master/master-test.cc M src/kudu/server/diagnostics_log.cc 2 files changed, 26 insertions(+), 10 deletions(-) Approvals: Adar Dembo: Looks good to me, approved Todd Lipcon: Verified -- To view, visit http://gerrit.cloudera.org:8080/9867 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: If7078bc87d2cf724d5f2fddf1173ac1786279e7e Gerrit-Change-Number: 9867 Gerrit-PatchSet: 3 Gerrit-Owner: Todd Lipcon <t...@apache.org> Gerrit-Reviewer: Adar Dembo <a...@cloudera.com> Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Mike Percy <mpe...@apache.org> Gerrit-Reviewer: Todd Lipcon <t...@apache.org>
