Huang Kuan Hao created HDDS-16884:
-------------------------------------
Summary: Intermittent timeout in
TestSnapshotBackgroundServices#testBackupCompactionFilesPruningBackgroundService
Key: HDDS-16884
URL: https://issues.apache.org/jira/browse/HDDS-16884
Project: Apache Ozone
Issue Type: Bug
Reporter: Huang Kuan Hao
Assignee: Huang Kuan Hao
{{TestSnapshotBackgroundServices#testBackupCompactionFilesPruningBackgroundService}}
intermittently times out waiting for the backup SST directory's file count to
decrease after pruning is resumed.
Observed CI failure:
https://github.com/apache/ozone/actions/runs/35732766354/job/106767352881
{code}
TestSnapshotBackgroundServices.testBackupCompactionFilesPruningBackgroundService:504
-> checkIfCompactionBackupFilesWerePruned:574
-> TimeoutException
{code}
The unchanged full class also reproduced the timeout locally on master
{{6818707698}} with JDK 21 (4 tests, 1 error):
{code}
mvn -pl :ozone-integration-test test -Dtest=TestSnapshotBackgroundServices
-DskipShade -DskipRecon -DskipDocs
{code}
Snapshot creation does not guarantee a completed tracked compaction. The
fixture relies on automatic compaction and shared-cluster state to produce
eligible backup SST inputs. Passing diagnostic runs removed inputs from a
previous test. A net directory count can also hide removals when new backup
files are created.
Disabling automatic compaction with the existing {{DBProfile.TEST}} reproduced
the missing precondition: the backup directory, compaction log and DAG remained
empty, and the original wait timed out. This establishes a fixture defect; the
historical CI log does not contain enough metadata to confirm its exact trigger.
The proposed fix forces a keyTable compaction using
{{CompactDBUtil.compactTable}} with bottommost {{kForce}} while pruning is
suspended, captures existing backup inputs from newly recorded compactions,
asserts that they are non-empty, and waits for those files to disappear after
pruning resumes. It preserves the leadership-transfer and snapshot-diff checks
and the existing 10-second timeout. Default {{compactDB()}} produced no new
inputs in a verification run, so forcing bottommost compaction is needed.
Local validation of the test-only fix:
* The controlled no-automatic-compaction probe fails with the original test and
passes with the fix.
* Five separate runs of the normal-profile full class passed: 20 tests, 0
failures/errors/skipped.
* Checkstyle passed with 0 violations.
Related: HDDS-9206 (resolved older pruning timeout), HDDS-10306 (shared test
cluster).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
