Siyao Meng created HDDS-16880:
---------------------------------
Summary: Recon reuses the truncated file left by a failed OM DB
checkpoint untar, so its retry from the same OM leader builds a DB with a
truncated SST
Key: HDDS-16880
URL: https://issues.apache.org/jira/browse/HDDS-16880
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: CR-16-recon-clear-candidate-dir-after-failed-fetch.patch,
TestBugCR16TruncatedSstReusedOnReconRetry.java
h3. Mechanism
When Recon fetches the OM DB,
{{RDBSnapshotProvider.downloadDBSnapshotFromLeader}} untars the downloaded part
into {{om.snapshot.db.candidate}} with {{FileUtil.unTar}}, which runs GNU tar
on Linux. If the archive ends in the middle of an entry, or extraction runs out
of space, tar exits with code 2 and leaves the partly written file in the
candidate dir. Since HDDS-15766,
{{OzoneManagerServiceProviderImpl.getOzoneManagerDBSnapshot}} deliberately
keeps the candidate dir after a failure, and
{{RDBSnapshotProvider.checkLeaderConsistency}} keeps it for the same leader. On
the next fetch {{ReconRDBSnapshotProvider.downloadSnapshot}} sends every file
name in the candidate dir ({{HAUtils.getExistingFiles}}, names only) as already
received. {{OMDBCheckpointServletInodeBasedXfer.collectFilesFromDir}} skips
files whose {{<inode>-<mtime>}} id is in that list, which still matches as long
as the SST is live on the leader, but still lists them in the hardLinkFile.
{{InodeMetadataRocksDBCheckpoint}} then links the truncated file as
{{om.db/<n>.sst}}, and {{ReconRDBSnapshotProvider.getCheckpointFromUntarredDb}}
promotes it to {{om.snapshot.db_<ts>}}. Nothing on the Recon side compares file
sizes or checksums.
The comment that keeps the dir says this "matches how the OM follower handles a
failed download", but it does not: {{OzoneManager.installSnapshotFromLeader}}
deletes the candidate dir after any failed download
({{cleanupCheckpoint(null)}}). The OM follower and {{ozone repair om}} are not
affected.
h3. Trigger
# Recon fetches a full OM DB snapshot (first start, or fallback after delta
updates fail).
# The untar of the response fails part way. One way is the leader side:
{{OMDBCheckpointServletInodeBasedXfer.processMetadataSnapshotRequest}} catches
an {{IOException}} from {{OMDBArchiver.writeToArchive}} after the response is
committed, logs it and returns normally, so Recon gets HTTP 200 and a tar cut
in the middle of an entry (see HDDS-16864). A dropped connection does not
trigger this, because {{ReconRDBSnapshotProvider.downloadSnapshot}} deletes the
partial tar on a read error. The other way, more likely on Recon, is the Recon
OM snapshot volume filling up during extraction: the tar stays on the volume
until the untar succeeds, so extraction needs about twice the checkpoint size.
Neither was run end to end; the reproduction uses a response cut in the middle
of an SST entry.
# The next fetch goes to the same OM leader while that SST is still live there.
h3. Impact
* The retry builds an OM DB with a truncated SST. RocksDB refuses to open it
({{Corruption: Sst file size mismatch between expected 24743 and file system
12288 sstable: .../om.snapshot.db_<ts>/000010.sst}}), so Recon does not read
the truncated data.
* {{ReconOmMetadataManagerImpl.initializeNewRdbStore}} only logs the open
failure. {{updateReconOmDBWithNewSnapshot}} returns true, logs "Successfully
updated Recon OM DB with new snapshot." and keeps the previous OM DB handle
(none on first start). {{getOzoneManagerDBSnapshot}} already deleted that
directory before the first fetch, so a later flush on it fails. In the
scheduled sync the run is then marked failed and {{GET_OM_DB_SNAPSHOT_FAILED}}
is set again, but only because the task reinitialization that follows cannot
checkpoint that deleted DB (or the missing one on first start).
* The retry that should have repaired Recon is wasted, and the corrupt
{{om.snapshot.db_<ts>}} stays on disk. Recon recovers at the next full fetch,
which deletes that dir and starts from an empty candidate dir, or after a
restart (the next full fetch was run; the restart was argued, not run).
* No effect on OM. Affects Version: 2.3.0. The affected code is only on master
(HDDS-15766 is in no release; 2.2.1 and earlier extract the OM DB into a fresh
dir without an exclusion list).
h3. Reproduction
Deterministic, unmodified source.
[^TestBugCR16TruncatedSstReusedOnReconRetry.java] ({{hadoop-ozone/recon}}) runs
Recon's own {{OzoneManagerServiceProviderImpl.updateReconOmDBWithNewSnapshot}}
and {{ReconRDBSnapshotProvider}}, with the real request form data and shell
untar, against a real OM RocksDB. Only the HTTP transport, the OM RPC client
and Recon's task wiring are mocked; the snapshot provider, the form data, the
shell untar, the hard link install and the RocksDB open are the real code. The
leader side is a stand-in for {{OMDBCheckpointServletInodeBasedXfer}} with
{{includeSnapshotData=false}}: a fresh checkpoint per request, entries named by
inode and mtime, files in the request's exclusion list skipped, then the
hardLinkFile and the completion flag. The first response ends in the middle of
the largest SST. The test asserts the correct behaviour, so it fails while the
defect is present. On the unfixed code the second request excludes the
truncated file, and the promoted DB has {{000010.sst}} at 12288 bytes instead
of 24743. The same run logs the RocksDB size mismatch, then "Successfully
updated Recon OM DB with new snapshot.". With the patch it passes.
h3. Patch
[^CR-16-recon-clear-candidate-dir-after-failed-fetch.patch], against
a6b7bdb937109ba2893688b41a89470bc95c88cf. It also applies to master at
6818707698c.
After a failed fetch, {{getOzoneManagerDBSnapshot}} now calls
{{reconSnapshotProvider.init()}}, which empties the candidate dir, as the OM
follower does. A {{RuntimeException}} from {{init()}} (for example
{{HddsUtils.createDir}} failing) is logged, so the method still returns null.
This puts back the reset that an earlier revision of HDDS-15766
(apache/ozone#10696) had in the same place, with the same guard. It was dropped
on review so that a future batched transfer could resume the parts already
received. Keeping the dir is only safe if every file in it is complete, and a
failed untar breaks that. Resuming a batched transfer would need each part
extracted atomically (to a staging dir, moved in only on success) or verified
(HDDS-14651). The OM follower, which does use batches, also wipes the dir after
a failure. Today Recon requests the checkpoint with
{{includeSnapshotData=false}}, which the leader sends in a single part, so
after a failure the dir only holds the output of a failed part and nothing that
is safe to resume is lost. This holds for the v1 endpoint too
({{ozone.om.db.checkpoint.use.inode.based.transfer=false}}), where Recon sends
SST paths and the leader also excludes by name (argued, not run). A leader side
fix would need sizes or checksums on the wire (see HDDS-14651). That
{{initializeNewRdbStore}} reports a failed open as success is not changed here;
it is filed as a separate issue.
Covered by {{testPartiallyUntarredFileNotReportedOnRetry}} in the existing
{{TestOzoneManagerServiceProviderImpl}}: after an untar failure on a truncated
SST, the next request must list no files and the candidate dir must be empty.
Without the change it fails with {{[[], ["000010.sst"]]}}.
{{testGetOzoneManagerDBSnapshotReturnsNullWhenCandidateDirResetFails}} covers
the guard. With the change, {{TestOzoneManagerServiceProviderImpl}},
{{TestReconRDBSnapshotProvider}} and {{TestTriggerDBSyncEndpoint}} (23 tests)
and the mini-cluster {{TestReconWithOzoneManagerHA}} pass, and checkstyle is
clean.
Related: HDDS-15766 (introduced the retention), HDDS-16864 (the same failed
untar also leaves the downloaded tarball behind), HDDS-14651 (manifest and
checksum validation would detect the short file).
Found by code review of the OM Ratis snapshot (checkpoint) install paths, as
part of the TLA+ verification effort under HDDS-15926, on commit
a6b7bdb937109ba2893688b41a89470bc95c88cf. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
