[
https://issues.apache.org/jira/browse/HDDS-16273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
YUNG_CHENG updated HDDS-16273:
------------------------------
Description:
h2. Problem
When a follower OM bootstraps or installs a Ratis snapshot, a single SST file
whose size is greater than or equal to
ozone.om.ratis.snapshot.max.total.sst.size can never be transferred, and the
follower retries forever without any error.
h2. Code path
In OMDBCheckpointServletInodeBasedXfer.collectFilesFromDir() the size budget is
checked *before*
the file is recorded and *before* it is added to sstFilesToExclude:
long fileSize = Files.size(dbFile);
if (maxTotalSstSize.get() - fileSize <= 0)
{ return false; }
bytesRecorded += omdbArchiver.recordFileEntry(dbFile.toFile(), fileId);
...
sstFilesToExclude.add(fileId);
maxTotalSstSize is rebuilt at its full configured value on every HTTP request,
so a file with
size >= budget fails the check in every round. The false return propagates to
shouldContinue, which skips omdbArchiver.setCompleted(true), so the tarball
carries neither the file nor the completion marker.
OMDBArchiver.writeToArchive() does not throw on an empty file map and the
servlet still returns HTTP 200.
On the follower, RDBSnapshotProvider.downloadDBSnapshotFromLeader() is an
unbounded while(true) whose only exit is ratisSnapshotComplete(). The exclusion
list for the next request is rebuilt from
HAUtils.getExistingFiles(candidateDir), which is unchanged, so the leader makes
the same zero-progress decision indefinitely. No error, no bound, no progress.
The wedged thread also holds the synchronized OzoneManager monitor via
installSnapshotFromLeader().
Note the comparison is "<= 0", so a file exactly equal to the budget also
stalls.
h2. Regression
The legacy servlet does not have this bug. OMDBCheckpointServlet.processDir()
calls processFile()
first and evaluates the budget afterwards, so it overshoots by one file and
always makes progress.
The check-before-record ordering was introduced with the inode-based transfer
in HDDS-12984 (commit 96390ac142), which became the default in HDDS-14721
(ozone.om.db.checkpoint.use.inode.based.transfer=true).
HDDS-14403 raised the default budget from 100 MB to 10 GiB, which masks the bug
but does not fix it.
h2. Impact
OM follower bootstrap / install-snapshot never completes. Reachable whenever
ozone.om.ratis.snapshot.max.total.sst.size is tuned down, or when a snapshot
SST exceeds the
configured value. Only the snapshot-data phases are bounded (snapshot dirs, SST
backup dir,
compaction log dir); the active om.db phase is unbounded.
h2. Existing coverage
TestOMRatisSnapshotTransfer sets the limit to the *sum* of SST sizes halved,
which by
construction never produces a single file larger than the whole budget, so the
failing case is untested.
h2. Proposed fix
Only enforce the budget once the current request has already collected at least
one file, so every round transfers at least one file and the transfer always
makes progress. This restores the legacy overshoot-by-one behaviour.
Additionally, bound the follower loop so a zero-progress round fails with a
diagnosable IOException instead of spinning forever.
was:
h2. Problem
When a follower OM bootstraps or installs a Ratis snapshot, a single SST file
whose size is greater than or equal to
ozone.om.ratis.snapshot.max.total.sst.size can never be transferred, and the
follower retries forever without any error.
h2. Code path
In OMDBCheckpointServletInodeBasedXfer.collectFilesFromDir() the size budget is
checked *before*
the file is recorded and *before* it is added to sstFilesToExclude:
long fileSize = Files.size(dbFile);
if (maxTotalSstSize.get() - fileSize <= 0)
{ return false; }
bytesRecorded += omdbArchiver.recordFileEntry(dbFile.toFile(), fileId);
...sstFilesToExclude.add(fileId);
maxTotalSstSize is rebuilt at its full configured value on every HTTP request,
so a file with
size >= budget fails the check in every round. The false return propagates to
shouldContinue, which skips omdbArchiver.setCompleted(true), so the tarball
carries neither the file nor the completion marker.
OMDBArchiver.writeToArchive() does not throw on an empty file map and the
servlet still returns HTTP 200.
On the follower, RDBSnapshotProvider.downloadDBSnapshotFromLeader() is an
unbounded while(true) whose only exit is ratisSnapshotComplete(). The exclusion
list for the next request is rebuilt from
HAUtils.getExistingFiles(candidateDir), which is unchanged, so the leader makes
the same zero-progress decision indefinitely. No error, no bound, no progress.
The wedged thread also holds the synchronized OzoneManager monitor via
installSnapshotFromLeader().
Note the comparison is "<= 0", so a file exactly equal to the budget also
stalls.
h2. Regression
The legacy servlet does not have this bug. OMDBCheckpointServlet.processDir()
calls processFile()
first and evaluates the budget afterwards, so it overshoots by one file and
always makes progress.
The check-before-record ordering was introduced with the inode-based transfer
in HDDS-12984 (commit 96390ac142), which became the default in HDDS-14721
(ozone.om.db.checkpoint.use.inode.based.transfer=true).
HDDS-14403 raised the default budget from 100 MB to 10 GiB, which masks the bug
but does not fix it.
h2. Impact
OM follower bootstrap / install-snapshot never completes. Reachable whenever
ozone.om.ratis.snapshot.max.total.sst.size is tuned down, or when a snapshot
SST exceeds the
configured value. Only the snapshot-data phases are bounded (snapshot dirs, SST
backup dir,
compaction log dir); the active om.db phase is unbounded.
h2. Existing coverage
TestOMRatisSnapshotTransfer sets the limit to the *sum* of SST sizes halved,
which by
construction never produces a single file larger than the whole budget, so the
failing case is untested.
h2. Proposed fix
Only enforce the budget once the current request has already collected at least
one file, so every round transfers at least one file and the transfer always
makes progress. This restores the legacy overshoot-by-one behaviour.
Additionally, bound the follower loop so a zero-progress round fails with a
diagnosable IOException instead of spinning forever.
> OM bootstrap never completes when a single SST file is larger than
> ozone.om.ratis.snapshot.max.total.sst.size
> -------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16273
> URL: https://issues.apache.org/jira/browse/HDDS-16273
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: YUNG_CHENG
> Assignee: YUNG_CHENG
> Priority: Major
>
> h2. Problem
> When a follower OM bootstraps or installs a Ratis snapshot, a single SST file
> whose size is greater than or equal to
> ozone.om.ratis.snapshot.max.total.sst.size can never be transferred, and the
> follower retries forever without any error.
> h2. Code path
> In OMDBCheckpointServletInodeBasedXfer.collectFilesFromDir() the size budget
> is checked *before*
> the file is recorded and *before* it is added to sstFilesToExclude:
> long fileSize = Files.size(dbFile);
> if (maxTotalSstSize.get() - fileSize <= 0)
> { return false; }
> bytesRecorded += omdbArchiver.recordFileEntry(dbFile.toFile(), fileId);
> ...
> sstFilesToExclude.add(fileId);
> maxTotalSstSize is rebuilt at its full configured value on every HTTP
> request, so a file with
> size >= budget fails the check in every round. The false return propagates to
> shouldContinue, which skips omdbArchiver.setCompleted(true), so the tarball
> carries neither the file nor the completion marker.
> OMDBArchiver.writeToArchive() does not throw on an empty file map and the
> servlet still returns HTTP 200.
> On the follower, RDBSnapshotProvider.downloadDBSnapshotFromLeader() is an
> unbounded while(true) whose only exit is ratisSnapshotComplete(). The
> exclusion list for the next request is rebuilt from
> HAUtils.getExistingFiles(candidateDir), which is unchanged, so the leader
> makes the same zero-progress decision indefinitely. No error, no bound, no
> progress. The wedged thread also holds the synchronized OzoneManager monitor
> via installSnapshotFromLeader().
> Note the comparison is "<= 0", so a file exactly equal to the budget also
> stalls.
> h2. Regression
> The legacy servlet does not have this bug. OMDBCheckpointServlet.processDir()
> calls processFile()
> first and evaluates the budget afterwards, so it overshoots by one file and
> always makes progress.
> The check-before-record ordering was introduced with the inode-based transfer
> in HDDS-12984 (commit 96390ac142), which became the default in HDDS-14721
> (ozone.om.db.checkpoint.use.inode.based.transfer=true).
> HDDS-14403 raised the default budget from 100 MB to 10 GiB, which masks the
> bug but does not fix it.
> h2. Impact
> OM follower bootstrap / install-snapshot never completes. Reachable whenever
> ozone.om.ratis.snapshot.max.total.sst.size is tuned down, or when a snapshot
> SST exceeds the
> configured value. Only the snapshot-data phases are bounded (snapshot dirs,
> SST backup dir,
> compaction log dir); the active om.db phase is unbounded.
> h2. Existing coverage
> TestOMRatisSnapshotTransfer sets the limit to the *sum* of SST sizes halved,
> which by
> construction never produces a single file larger than the whole budget, so
> the failing case is untested.
> h2. Proposed fix
> Only enforce the budget once the current request has already collected at
> least one file, so every round transfers at least one file and the transfer
> always makes progress. This restores the legacy overshoot-by-one behaviour.
> Additionally, bound the follower loop so a zero-progress round fails with a
> diagnosable IOException instead of spinning forever.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
