Andrew Wong has submitted this change and it was merged. (
http://gerrit.cloudera.org:8080/15113 )
Change subject: KUDU-3046: deflake TabletServerQuiescingITest
......................................................................
KUDU-3046: deflake TabletServerQuiescingITest
The test was flaky for a number of reasons including:
- Slowness in TSAN mode along with a low Raft timeout meant workloads
would fail to even create tablets.
- Addressed this by increasing the heartbeat interval in TSAN mode.
- Not hitting the exact number of scanners when running the tool because
of a TOCTOU race between checking the number of scanners and running
the tool.
- Addressed this by reducing the number of read threads and thus
reducing the degrees of freedom with which the tool can run (either
0 scanners or 1 scanner).
- TestAbruptStepdownWhileAllQuiescing failed because the test would step
down a leader without the guarantee that it was the latest leader, so
a leader could still exist even after stepping down.
- Addressed this by stepping down on all tablet servers just to be
sure, and retrying if necessary via ASSERT_EVENTUALLY.
There appears to be another source of flakiness that are less specific
to this test, but this dropped flakiness from failing 4/100 to failing
9/2000 (all due to a TSAN issue in the TestWorkload that I'm still
getting to the bottom of).
Change-Id: I3f9ef531062c4b66648840e04962070768fbad5d
Reviewed-on: http://gerrit.cloudera.org:8080/15113
Reviewed-by: Adar Dembo <[email protected]>
Tested-by: Kudu Jenkins
Reviewed-by: Alexey Serbin <[email protected]>
---
M src/kudu/integration-tests/tablet_server_quiescing-itest.cc
1 file changed, 43 insertions(+), 11 deletions(-)
Approvals:
Adar Dembo: Looks good to me, approved
Kudu Jenkins: Verified
Alexey Serbin: Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/15113
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I3f9ef531062c4b66648840e04962070768fbad5d
Gerrit-Change-Number: 15113
Gerrit-PatchSet: 6
Gerrit-Owner: Andrew Wong <[email protected]>
Gerrit-Reviewer: Adar Dembo <[email protected]>
Gerrit-Reviewer: Alexey Serbin <[email protected]>
Gerrit-Reviewer: Andrew Wong <[email protected]>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
