[
https://issues.apache.org/jira/browse/KUDU-3789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18097023#comment-18097023
]
Gabriella Lotz commented on KUDU-3789:
--------------------------------------
I ran the auto_leader_rebalancer-test under TSAN dist-test 1000x
multiple times to work through the flakiness. Below are the tests that
were run and the actions taken.
h4. 1. Test (http://dist-test.cloudera.org/job?job_id=root.1783598847.260469) -
464 failed
Highest number of fails:
approx. 57x - AddTserver (in functions CheckLeaderBalance and
RunLeaderRebalancer)
approx. 17x - FilterSoftDeletedTableTest
Actions taken: added AssertEventually to the convergence checks
(AddTserver etc.) and made the leader step-down best-effort in
auto_leader_rebalancer.cc.
h4. 2. Test (http://dist-test.cloudera.org/job?job_id=root.1783601546.275653) -
113 failed
approx. 78x - AddTserver
Actions taken: I tried stripping the replica rebalancer during the test
to remove the extra leadership moves.
h4. 3. Test (http://dist-test.cloudera.org/job?job_id=root.1783608008.290841) -
621 failed
Since this was a regression, I quickly reverted the previous change and
reduced the tablet count from 59 to 24 instead in AddTserver.
h4. 4. Test (http://dist-test.cloudera.org/job?job_id=root.1783681035.306066) -
9 failed
3x - tserver crash: hdr_histogram.cc value >= 0 at line 181
2x - crash: LeaderElection has_responded_ at leader_election.cc line 206
3x - TestMaintenanceMode
1x - MultiTableLeaderBalance
Actions taken: fixed the two crashes (real product bugs, filed
separately, see below) and gave MultiTableLeaderBalance a longer wait
time.
h4. 5. Test (http://dist-test.cloudera.org/job?job_id=root.1783688995.321310) -
3 failed
2x - MultiTableLeaderBalance
1x - FilterSoftDeletedTableTest
Actions taken: raised leader_failure_max_missed_heartbeat_periods,
relaxed the soft-deleted check and rewrote TestMaintenanceMode.
h4. 6. Test (http://dist-test.cloudera.org/job?job_id=root.1783932881.336514) -
0 failed
Actions
- Deflake the test: [https://gerrit.cloudera.org/c/24591/] (this ticket)
- Best-effort leader step-down: [https://gerrit.cloudera.org/c/24594/]
Product bugs raised
- [KUDU-3792|https://issues.apache.org/jira/browse/KUDU-3792] - tablet server
crash from negative op_duration (hdr_histogram CHECK)
- [KUDU-3793|https://issues.apache.org/jira/browse/KUDU-3793] - LeaderElection
has_responded_ CHECK on shutdown from a dropped async DNS callback
> Investigate and fix flaky tests
> -------------------------------
>
> Key: KUDU-3789
> URL: https://issues.apache.org/jira/browse/KUDU-3789
> Project: Kudu
> Issue Type: Sub-task
> Reporter: Gabriella Lotz
> Assignee: Gabriella Lotz
> Priority: Major
>
> # In auto_rebalancer-test.cc
> # In auto_leader_rebalancer-test.cc
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
