[
https://issues.apache.org/jira/browse/HDDS-17743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17743:
------------------------------
Affects Version/s: 2.3.0
> DAG snapshot diff reads value pruned compaction backup SST files and drops
> every change in them
> -----------------------------------------------------------------------------------------------
>
> Key: HDDS-17743
> URL: https://issues.apache.org/jira/browse/HDDS-17743
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 2.3.0
> Reporter: Siyao Meng
> Priority: Major
> Attachments: CR-5-full-diff-fallback-for-pruned-delta-files.patch,
> TestBugCR5PrunedBackupSst.java
>
>
> h3. Mechanism
> {{RocksDBCheckpointDiffer.getSSTFullPath}} resolves a delta file to its copy
> in the compaction backup directory whenever that copy exists.
> {{RocksDBCheckpointDiffer.pruneSstFileValues}} rewrites the compaction inputs
> in that directory with an SST file writer: empty values and sequence number
> 0. {{DagDiffSequentialReader}} reads the delta through
> {{LatestVersionedKWayMergeIterator.overRawSstFilesFromSequence}}, which keeps
> only rows with a sequence number above the from snapshot's
> {{dbTxSequenceNumber}}, so every row of a pruned file is dropped, tombstones
> included. The values are gone too: even without the sequence check the DAG
> reader could not use these rows, because it parses the values, so relaxing
> the check for sequence 0 rows is not a fix. Nothing falls back to a full
> diff, and the job ends DONE.
> h3. Trigger
> {{ozone.om.snapshot.diff.optimized.enabled=true}}, native library loaded (DAG
> path, the default; the pruner runs whenever the library loads, every 10
> minutes by default):
> # Take snapshot snap1, write keys, take snapshot snap2.
> # A compaction merges the SST file flushed for snap2 with other files. The
> file is kept in the backup directory and later pruned.
> # Request the diff snap1 to snap2, or to a later snapshot.
> A trivial move of the file to another level does not record a compaction and
> does not queue the file for pruning, so tiny test data may need keys on both
> sides of the new ones to get a merging compaction.
> h3. Impact
> The report misses every CREATE, MODIFY, DELETE and RENAME held in the pruned
> files; in the reproduction both reports are empty instead of five CREATEs. A
> diff over a pair whose delta files have since been compacted and pruned is
> affected, whether the file is still in the to snapshot (snap1 to snap2) or
> exists only in the backup directory (snap1 to a snapshot taken after the
> compaction). The report is persisted and returned for every later read of the
> pair.
> h3. Reproduction
> [^TestBugCR5PrunedBackupSst.java] ({{hadoop-ozone/integration-test}}, needs
> {{-Drocks_tools_native}}) writes keys through the client on a mini cluster,
> takes the snapshots, then runs the OM DB compaction and the pruner directly
> for timing and checks that the pruner pruned files. A control diff before
> pruning reports the five CREATEs. After pruning, snap1 to snap2 (ordered) and
> snap1 to snap3 are both empty on the unfixed code ({{Expecting actual: [] to
> contain exactly in any order: [+ ./m0, + ./m1, + ./m2, + ./m3, + ./m4]}});
> with the patch both list the five CREATEs.
> h3. Patch
> [^CR-5-full-diff-fallback-for-pruned-delta-files.patch], against
> d3b75c75bf9ad475551202477c3a21a3f4b968b7. It also applies to master at
> bbc8505e00d.
> New {{SstFileSetReader.hasExternalSstFile}} checks for the
> {{rocksdb.external_sst_file.version}} table property, which the SST file
> writer adds to every file it writes and flush or compaction outputs do not
> have. When a table's DAG delta contains such a file,
> {{SnapshotDiffManager.scanTablesForDiff}} logs it and uses the full diff
> reader for that table, which scans both snapshot tables and does not read the
> delta files. Any delta file written by an SST file writer takes this
> fallback: value pruned backup files, and by code reading also the files that
> snapshot defragmentation ingests (off by default; not run). Delta files
> produced by flush or compaction keep the DAG path. The fallback goes through
> the full diff reader, whose updateID gate has its own defect, HDDS-17742;
> until that is fixed the fallback can still miss changes once deep cleaning or
> a snapshot purge has updated the from snapshot. A diff over pruned files is
> now correct but costs a scan of both snapshot tables. Reading the delta from
> the snapshot checkpoints instead of the backup directory was not chosen
> because it does not cover delta files that exist only in the backup directory.
> Covered by {{testDagDiffAfterBackupSstValuePruning}} in the existing
> {{TestOptimizedSnapshotDiff}} (native): it compacts, prunes, asserts that
> files were pruned, expects the five CREATEs in unordered and ordered reports,
> and asserts that the fallback was logged. Without the change both reports are
> empty. {{testHasExternalSstFile}} in the existing {{TestSstFileSetReader}}
> checks the new method on a file written by the SST file writer (true) and on
> a file flushed by a DB (false). With the change
> {{TestOptimizedSnapshotDiff}}, {{TestRocksDBCheckpointDiffer}},
> {{TestSstFileSetReader}}, the snapshot diff unit suites and
> {{TestSnapshotDiffManager}} pass, and checkstyle is clean.
> Found by code review of the OM optimized snapshot diff paths, as part of the
> TLA+ verification effort under HDDS-15926, on commit
> d3b75c75bf9ad475551202477c3a21a3f4b968b7. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
