Siyao Meng created HDDS-17743:
---------------------------------
Summary: DAG snapshot diff reads value pruned compaction backup
SST files and drops every change in them
Key: HDDS-17743
URL: https://issues.apache.org/jira/browse/HDDS-17743
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: CR-5-full-diff-fallback-for-pruned-delta-files.patch,
TestBugCR5PrunedBackupSst.java
h3. Mechanism
{{RocksDBCheckpointDiffer.getSSTFullPath}} resolves a delta file to its copy in
the compaction backup directory whenever that copy exists.
{{RocksDBCheckpointDiffer.pruneSstFileValues}} rewrites the compaction inputs
in that directory with an SST file writer: empty values and sequence number 0.
{{DagDiffSequentialReader}} reads the delta through
{{LatestVersionedKWayMergeIterator.overRawSstFilesFromSequence}}, which keeps
only rows with a sequence number above the from snapshot's
{{dbTxSequenceNumber}}, so every row of a pruned file is dropped, tombstones
included. The values are gone too: even without the sequence check the DAG
reader could not use these rows, because it parses the values, so relaxing the
check for sequence 0 rows is not a fix. Nothing falls back to a full diff, and
the job ends DONE.
h3. Trigger
{{ozone.om.snapshot.diff.optimized.enabled=true}}, native library loaded (DAG
path, the default; the pruner runs whenever the library loads, every 10 minutes
by default):
# Take snapshot snap1, write keys, take snapshot snap2.
# A compaction merges the SST file flushed for snap2 with other files. The file
is kept in the backup directory and later pruned.
# Request the diff snap1 to snap2, or to a later snapshot.
A trivial move of the file to another level does not record a compaction and
does not queue the file for pruning, so tiny test data may need keys on both
sides of the new ones to get a merging compaction.
h3. Impact
The report misses every CREATE, MODIFY, DELETE and RENAME held in the pruned
files; in the reproduction both reports are empty instead of five CREATEs. A
diff over a pair whose delta files have since been compacted and pruned is
affected, whether the file is still in the to snapshot (snap1 to snap2) or
exists only in the backup directory (snap1 to a snapshot taken after the
compaction). The report is persisted and returned for every later read of the
pair.
h3. Reproduction
[^TestBugCR5PrunedBackupSst.java] ({{hadoop-ozone/integration-test}}, needs
{{-Drocks_tools_native}}) writes keys through the client on a mini cluster,
takes the snapshots, then runs the OM DB compaction and the pruner directly for
timing and checks that the pruner pruned files. A control diff before pruning
reports the five CREATEs. After pruning, snap1 to snap2 (ordered) and snap1 to
snap3 are both empty on the unfixed code ({{Expecting actual: [] to contain
exactly in any order: [+ ./m0, + ./m1, + ./m2, + ./m3, + ./m4]}}); with the
patch both list the five CREATEs.
h3. Patch
[^CR-5-full-diff-fallback-for-pruned-delta-files.patch], against
d3b75c75bf9ad475551202477c3a21a3f4b968b7. It also applies to master at
bbc8505e00d.
New {{SstFileSetReader.hasExternalSstFile}} checks for the
{{rocksdb.external_sst_file.version}} table property, which the SST file writer
adds to every file it writes and flush or compaction outputs do not have. When
a table's DAG delta contains such a file,
{{SnapshotDiffManager.scanTablesForDiff}} logs it and uses the full diff reader
for that table, which scans both snapshot tables and does not read the delta
files. Any delta file written by an SST file writer takes this fallback: value
pruned backup files, and by code reading also the files that snapshot
defragmentation ingests (off by default; not run). Delta files produced by
flush or compaction keep the DAG path. The fallback goes through the full diff
reader, whose updateID gate has its own defect, HDDS-17742; until that is fixed
the fallback can still miss changes once deep cleaning or a snapshot purge has
updated the from snapshot. A diff over pruned files is now correct but costs a
scan of both snapshot tables. Reading the delta from the snapshot checkpoints
instead of the backup directory was not chosen because it does not cover delta
files that exist only in the backup directory.
Covered by {{testDagDiffAfterBackupSstValuePruning}} in the existing
{{TestOptimizedSnapshotDiff}} (native): it compacts, prunes, asserts that files
were pruned, expects the five CREATEs in unordered and ordered reports, and
asserts that the fallback was logged. Without the change both reports are
empty. {{testHasExternalSstFile}} in the existing {{TestSstFileSetReader}}
checks the new method on a file written by the SST file writer (true) and on a
file flushed by a DB (false). With the change {{TestOptimizedSnapshotDiff}},
{{TestRocksDBCheckpointDiffer}}, {{TestSstFileSetReader}}, the snapshot diff
unit suites and {{TestSnapshotDiffManager}} pass, and checkstyle is clean.
Found by code review of the OM optimized snapshot diff paths, as part of the
TLA+ verification effort under HDDS-15926, on commit
d3b75c75bf9ad475551202477c3a21a3f4b968b7. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
