Siyao Meng created HDDS-17747:
---------------------------------
Summary: DAG snapshot diff misses the DELETE of a directory that
is deleted and replaced at the same path within one flush
Key: HDDS-17747
URL: https://issues.apache.org/jira/browse/HDDS-17747
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: CR-3-load-from-dir-row-for-every-delta-key.patch,
TestBugCR3DirectoryReplacedAtSameKey.java
h3. Mechanism
{{DagDiffSequentialReader.scanFromDirectoryTable}} loads a from snapshot
directory row into the old list only when its objectId was seen as a put in the
directory delta, or when its table key had a tombstone in the delta. After {{rm
-r d}} and {{mkdir d}} with no flush in between, both writes hit the same
{{directoryTable}} key and RocksDB drops the tombstone at flush, so the delta
holds only the put of the new objectId. The same happens when another directory
is renamed onto the freed path ({{rm -r d}}, {{mv x d}}). The old objectId
passes neither check and never reaches the old list. The merge join then emits
CREATE d without DELETE d. The file path is not affected because it looks up
every delta key in the from table.
h3. Trigger
FSO bucket, {{ozone.om.snapshot.diff.optimized.enabled=true}}, native library
loaded (DAG path, the default):
# Create {{/a/d/f}}, take snapshot snap1.
# {{rm -r /a/d}}, {{mkdir /a/d}}, take snapshot snap2 (no flush in between; the
snapshot flushes).
# Request the snapshot diff snap1 to snap2, ordered or unordered.
Replacing {{/a/d}} by renaming another directory onto it ({{rm -r /a/d}}, {{mv
/a/x /a/d}}) in the same flush has the same effect: the report has RENAME a/x
to a/d but no DELETE a/d (shown at unit level, see Patch). The same change with
a snapshot (flush) between the delete and the second write is reported
correctly.
h3. Impact
The report contains {{CREATE a/d}} only. Replaying it onto snap1 keeps the old
subtree, including {{a/d/f}}, which is gone in snap2. The report is persisted
and returned for every later read of the pair.
h3. Reproduction
[^TestBugCR3DirectoryReplacedAtSameKey.java]
({{hadoop-ozone/integration-test}}, needs {{-Drocks_tools_native}}) runs the
steps for {{a/d}} through the FileSystem and snapshot APIs on a mini cluster,
plus a control {{b/d}} whose delete and mkdir are separated by a snapshot, in
the same diff. On the unfixed code it fails with {{Expecting ArrayList: [-
./b/d, + ./a/d, + ./b/d] ... could not find the following element(s): [-
./a/d]}}; with the patch it passes.
h3. Patch
[^CR-3-load-from-dir-row-for-every-delta-key.patch], against
d3b75c75bf9ad475551202477c3a21a3f4b968b7. It also applies to master at
bbc8505e00d.
{{scanDirectoryDeltaEntries}} now records the table key of every directory
delta entry, put or tombstone, in the key set it already keeps for tombstones
(the {{SnapDiffJobStore}} Javadocs of that set are updated; the method names
are kept). The from side row at a replaced key is then loaded into the old list
and reported as DELETE. For a key whose objectId did not change the row was
already loaded, so nothing else changes. The merge iterator cannot be the place
for this, since the tombstone is no longer in any SST file.
Covered by {{testDirectoryReplacedAtSameKeyWithoutTombstone}} in the existing
{{TestDagDiffSequentialReader}}, for both shapes: a delta with only the put of
a new objectId at a key whose from row has another objectId, and a delta with
the tombstone of a renamed directory plus its put at a key whose from row has
another objectId. Without the change the replaced objectIds are missing from
the old list (each shape checked separately). With it the snapshot diff unit
suites and {{TestSnapshotDiffManager}} (192 tests) and
{{TestOptimizedSnapshotDiff}} pass, and checkstyle is clean.
Found by code review of the OM optimized snapshot diff paths, as part of the
TLA+ verification effort under HDDS-15926, on commit
d3b75c75bf9ad475551202477c3a21a3f4b968b7. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
