[
https://issues.apache.org/jira/browse/HBASE-30466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jose Luis López updated HBASE-30466:
------------------------------------
Description:
Some tests do not stop their test clusters when they finish. When surefire
retries a failed test, the retry runs in the same JVM, so it finds the old
cluster still running and fails straight away with "A mini-cluster is already
running". One real failure therefore turns into several fake ones.
We saw this with TestSyncReplicationStandbyKillMaster on apache/hbase#8743. The
first run hit a known flake (HBASE-30344). The first retry could not start its
cluster, and the cleanup skipped the shutdown, so the half-started cluster was
left running. The second retry then failed because of that leftover cluster.
h3. Affected tests
* The TestSyncReplication* tests: cleanup skips shutting down a cluster that
failed to start, and one cleanup error stops the remaining clusters from being
shut down.
* TestClaimReplicationQueue, TestRemovePeerProcedureWaitForSCP and
TestAsyncWALReplay (plus its 3 subclasses): their cleanup method replaces the
parent class's cleanup instead of adding to it, so their clusters are never
shut down.
h3. Fix
Always shut down every cluster, even when starting it or an earlier cleanup
step failed. Test-only change.
was:
On apache/hbase#8743 (large-wave-3), TestSyncReplicationStandbyKillMaster
failed three times in a row, but only the first failure was real:
# Run 1 failed in the test body (the flake tracked in HBASE-30344).
# Rerun 1: {{UTIL1.startMiniCluster}}
({{SyncReplicationTestBaseNoBeforeAll.startClusters:111}}) started DFS, then
the master did not initialise in 200 s, so it threw "IOException: Shutting
down". DFS stayed up and the util stayed marked running.
# {{@AfterAll}} called {{shutdown(UTIL1)}}, which returns early when
{{util.getHBaseCluster() == null}} and so never called
{{shutdownMiniCluster()}}.
# Rerun 2 failed at once with "IllegalStateException: A mini-cluster is already
running"; its log still shows the leaked master writing to the leaked DFS.
h3. Problems
* {{SyncReplicationTestBaseNoBeforeAll.shutdown(util)}} skips
{{shutdownMiniCluster()}} exactly in the state a failed start leaves behind.
* {{tearDown()}} runs admin calls and then shuts down UTIL1, UTIL2 and ZK in
sequence with no finally, so one failing call leaks the rest. This affects all
15 subclasses.
* {{TestClaimReplicationQueue}} and {{TestRemovePeerProcedureWaitForSCP}}
declare a static {{tearDownAfterClass()}} that hides
{{TestReplicationBaseNoBeforeAll.tearDownAfterClass()}} without calling it, so
their two clusters are never shut down.
* {{TestAsyncWALReplay}} does the same with
{{AbstractTestWALReplay.tearDownAfterClass()}}, which affects it and its 3
subclasses.
h3. Fix
Always call {{shutdownMiniCluster()}} in finally, chain UTIL1 -> UTIL2 -> ZK
through finally, and make the hiding methods call the base teardown in finally.
Test-only change.
> TestSyncReplication* and replication/WAL replay tests do not shut down mini
> clusters after a failed start or a hidden @AfterAll
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-30466
> URL: https://issues.apache.org/jira/browse/HBASE-30466
> Project: HBase
> Issue Type: Test
> Reporter: Jose Luis López
> Priority: Major
> Labels: pull-request-available
>
> Some tests do not stop their test clusters when they finish. When surefire
> retries a failed test, the retry runs in the same JVM, so it finds the old
> cluster still running and fails straight away with "A mini-cluster is already
> running". One real failure therefore turns into several fake ones.
> We saw this with TestSyncReplicationStandbyKillMaster on apache/hbase#8743.
> The first run hit a known flake (HBASE-30344). The first retry could not
> start its cluster, and the cleanup skipped the shutdown, so the half-started
> cluster was left running. The second retry then failed because of that
> leftover cluster.
> h3. Affected tests
> * The TestSyncReplication* tests: cleanup skips shutting down a cluster that
> failed to start, and one cleanup error stops the remaining clusters from
> being shut down.
> * TestClaimReplicationQueue, TestRemovePeerProcedureWaitForSCP and
> TestAsyncWALReplay (plus its 3 subclasses): their cleanup method replaces the
> parent class's cleanup instead of adding to it, so their clusters are never
> shut down.
> h3. Fix
> Always shut down every cluster, even when starting it or an earlier cleanup
> step failed. Test-only change.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
