YUNG_CHENG created HDDS-16273:
---------------------------------
Summary: OM bootstrap never completes when a single SST file is
larger than ozone.om.ratis.snapshot.max.total.sst.size
Key: HDDS-16273
URL: https://issues.apache.org/jira/browse/HDDS-16273
Project: Apache Ozone
Issue Type: Bug
Reporter: YUNG_CHENG
Assignee: YUNG_CHENG
h2. Problem
When a follower OM bootstraps or installs a Ratis snapshot, a single SST file
whose size is greater than or equal to
ozone.om.ratis.snapshot.max.total.sst.size can never be transferred, and the
follower retries forever without any error.
h2. Code path
In OMDBCheckpointServletInodeBasedXfer.collectFilesFromDir() the size budget is
checked *before*
the file is recorded and *before* it is added to sstFilesToExclude:
long fileSize = Files.size(dbFile);
if (maxTotalSstSize.get() - fileSize <= 0) {
return false;
}
bytesRecorded += omdbArchiver.recordFileEntry(dbFile.toFile(), fileId);
...
sstFilesToExclude.add(fileId);
maxTotalSstSize is rebuilt at its full configured value on every HTTP request,
so a file with
size >= budget fails the check in every round. The false return propagates to
shouldContinue, which skips omdbArchiver.setCompleted(true), so the tarball
carries neither the file nor the completion marker.
OMDBArchiver.writeToArchive() does not throw on an empty file map and the
servlet still returns HTTP 200.
On the follower, RDBSnapshotProvider.downloadDBSnapshotFromLeader() is an
unbounded while(true) whose only exit is ratisSnapshotComplete(). The exclusion
list for the next request is rebuilt from
HAUtils.getExistingFiles(candidateDir), which is unchanged, so the leader makes
the same zero-progress decision indefinitely. No error, no bound, no progress.
The wedged thread also holds the synchronized OzoneManager monitor via
installSnapshotFromLeader().
Note the comparison is "<= 0", so a file exactly equal to the budget also
stalls.
h2. Regression
The legacy servlet does not have this bug. OMDBCheckpointServlet.processDir()
calls processFile()
first and evaluates the budget afterwards, so it overshoots by one file and
always makes progress.
The check-before-record ordering was introduced with the inode-based transfer
in HDDS-12984 (commit 96390ac142), which became the default in HDDS-14721
(ozone.om.db.checkpoint.use.inode.based.transfer=true).
HDDS-14403 raised the default budget from 100 MB to 10 GiB, which masks the bug
but does not fix it.
h2. Impact
OM follower bootstrap / install-snapshot never completes. Reachable whenever
ozone.om.ratis.snapshot.max.total.sst.size is tuned down, or when a snapshot
SST exceeds the
configured value. Only the snapshot-data phases are bounded (snapshot dirs, SST
backup dir,
compaction log dir); the active om.db phase is unbounded.
h2. Existing coverage
TestOMRatisSnapshotTransfer sets the limit to the *sum* of SST sizes halved,
which by
construction never produces a single file larger than the whole budget, so the
failing case is untested.
h2. Proposed fix
Only enforce the budget once the current request has already collected at least
one file, so every round transfers at least one file and the transfer always
makes progress. This restores the legacy overshoot-by-one behaviour.
Additionally, bound the follower loop so a zero-progress round fails with a
diagnosable IOException instead of spinning forever.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
