[
https://issues.apache.org/jira/browse/HDDS-17747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17747:
------------------------------
Affects Version/s: 2.3.0
> DAG snapshot diff misses the DELETE of a directory that is deleted and
> replaced at the same path within one flush
> -----------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17747
> URL: https://issues.apache.org/jira/browse/HDDS-17747
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 2.3.0
> Reporter: Siyao Meng
> Priority: Major
> Attachments: CR-3-load-from-dir-row-for-every-delta-key.patch,
> TestBugCR3DirectoryReplacedAtSameKey.java
>
>
> h3. Mechanism
> {{DagDiffSequentialReader.scanFromDirectoryTable}} loads a from snapshot
> directory row into the old list only when its objectId was seen as a put in
> the directory delta, or when its table key had a tombstone in the delta.
> After {{rm -r d}} and {{mkdir d}} with no flush in between, both writes hit
> the same {{directoryTable}} key and RocksDB drops the tombstone at flush, so
> the delta holds only the put of the new objectId. The same happens when
> another directory is renamed onto the freed path ({{rm -r d}}, {{mv x d}}).
> The old objectId passes neither check and never reaches the old list. The
> merge join then emits CREATE d without DELETE d. The file path is not
> affected because it looks up every delta key in the from table.
> h3. Trigger
> FSO bucket, {{ozone.om.snapshot.diff.optimized.enabled=true}}, native library
> loaded (DAG path, the default):
> # Create {{/a/d/f}}, take snapshot snap1.
> # {{rm -r /a/d}}, {{mkdir /a/d}}, take snapshot snap2 (no flush in between;
> the snapshot flushes).
> # Request the snapshot diff snap1 to snap2, ordered or unordered.
> Replacing {{/a/d}} by renaming another directory onto it ({{rm -r /a/d}},
> {{mv /a/x /a/d}}) in the same flush has the same effect: the report has
> RENAME a/x to a/d but no DELETE a/d (shown at unit level, see Patch). The
> same change with a snapshot (flush) between the delete and the second write
> is reported correctly.
> h3. Impact
> The report contains {{CREATE a/d}} only. Replaying it onto snap1 keeps the
> old subtree, including {{a/d/f}}, which is gone in snap2. The report is
> persisted and returned for every later read of the pair.
> h3. Reproduction
> [^TestBugCR3DirectoryReplacedAtSameKey.java]
> ({{hadoop-ozone/integration-test}}, needs {{-Drocks_tools_native}}) runs the
> steps for {{a/d}} through the FileSystem and snapshot APIs on a mini cluster,
> plus a control {{b/d}} whose delete and mkdir are separated by a snapshot, in
> the same diff. On the unfixed code it fails with {{Expecting ArrayList: [-
> ./b/d, + ./a/d, + ./b/d] ... could not find the following element(s): [-
> ./a/d]}}; with the patch it passes.
> h3. Patch
> [^CR-3-load-from-dir-row-for-every-delta-key.patch], against
> d3b75c75bf9ad475551202477c3a21a3f4b968b7. It also applies to master at
> bbc8505e00d.
> {{scanDirectoryDeltaEntries}} now records the table key of every directory
> delta entry, put or tombstone, in the key set it already keeps for tombstones
> (the {{SnapDiffJobStore}} Javadocs of that set are updated; the method names
> are kept). The from side row at a replaced key is then loaded into the old
> list and reported as DELETE. For a key whose objectId did not change the row
> was already loaded, so nothing else changes. The merge iterator cannot be the
> place for this, since the tombstone is no longer in any SST file.
> Covered by {{testDirectoryReplacedAtSameKeyWithoutTombstone}} in the existing
> {{TestDagDiffSequentialReader}}, for both shapes: a delta with only the put
> of a new objectId at a key whose from row has another objectId, and a delta
> with the tombstone of a renamed directory plus its put at a key whose from
> row has another objectId. Without the change the replaced objectIds are
> missing from the old list (each shape checked separately). With it the
> snapshot diff unit suites and {{TestSnapshotDiffManager}} (192 tests) and
> {{TestOptimizedSnapshotDiff}} pass, and checkstyle is clean.
> Found by code review of the OM optimized snapshot diff paths, as part of the
> TLA+ verification effort under HDDS-15926, on commit
> d3b75c75bf9ad475551202477c3a21a3f4b968b7. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
