Siyao Meng created HDDS-17745:
---------------------------------
Summary: Optimized FSO snapshot diff drops every file change below
the bucket root when no directory row changed
Key: HDDS-17745
URL: https://issues.apache.org/jira/browse/HDDS-17745
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: MC-5-run-directory-stage-when-only-files-changed.patch,
TestBugMC5EmptyDirectoryDelta.java
h3. Mechanism
{{SnapshotDiffManager.scanTablesForDiff}} returns early when a table has no
delta SST files, before it picks a reader. For an FSO bucket the directory
stage is also what builds the path edges that {{SnapDiffPathResolver}} uses to
resolve file entries. Creating or deleting a file under an existing directory
writes nothing to {{directoryTable}}, so when only files changed the directory
delta is empty, the directory stage is skipped, every file entry whose parent
is not the bucket resolves to null, and {{MergeJoinSnapDiffWriter}} drops it
with a DEBUG log. The job ends DONE.
{{DagDiffSequentialReader.scanDirectoryTables}} already handles an empty delta
by scanning both directory tables; the early return keeps it from running.
h3. Trigger
FSO bucket, {{ozone.om.snapshot.diff.optimized.enabled=true}}, native rocks
tools library loaded (the default, {{ozone.om.snapshot.load.native.lib=true}}):
# Create {{/a/old}}, take snapshot snap1.
# Create {{/a/new}}, delete {{/a/old}}, take snapshot snap2.
# Request the snapshot diff snap1 to snap2, ordered or unordered.
The diff path does not matter once the native library is loaded: the DAG
reader, a forced full diff and the automatic full diff fallback all build the
directory delta only from SST files the two snapshots do not share, and all
pass through the same early return. A forced full diff with the native library
was run and gives the same empty report; the automatic fallback uses the same
code (by reading). Without the native library the optimized path runs only with
a forced full diff, and the non native delta computation then adds every SST
file of the from snapshot to the delta, so the directory delta is never empty
and the bug does not show.
h3. Impact
Both reports are empty instead of {{[CREATE a/new, DELETE a/old]}}. Any diff in
which files were only created, overwritten or deleted, with no directory row
written (no directory created, renamed or deleted, and no file renamed, since a
file rename updates the modification time of both parent directories), loses
every change below the bucket root. The report is persisted and returned for
every later read of the pair.
h3. Reproduction
[^TestBugMC5EmptyDirectoryDelta.java] ({{hadoop-ozone/integration-test}}, needs
{{-Drocks_tools_native}}) runs the steps above through the FileSystem and
snapshot APIs on a mini cluster with the native library, on the default DAG
path (the forced full diff variant was checked separately). On the unfixed code
it fails with {{Expecting actual: [] to contain exactly in any order: [+
./a/new, - ./a/old]}}; with the patch it passes.
h3. Patch
[^MC-5-run-directory-stage-when-only-files-changed.patch], against
d3b75c75bf9ad475551202477c3a21a3f4b968b7. It also applies to master at
bbc8505e00d.
The early return now skips the directory stage only when the earlier file stage
put nothing into the new and old lists ({{SnapDiffJobStore.hasListEntries}},
from the counters the store already keeps). The directory stage then runs with
an empty delta and builds the edges. A diff with no changes at all keeps the
fast path. A diff in which only files changed now scans both directory tables
of the bucket, which a diff with one directory change already does.
Covered by {{testOptimizedDiffWithUnchangedDirectoryTable}} in the existing
{{TestOptimizedSnapshotDiff}} (native), for both report orderings. Without the
change both reports are empty. With it {{TestOptimizedSnapshotDiff}}, the
snapshot diff unit suites and {{TestSnapshotDiffManager}} (191 tests) pass, and
checkstyle is clean.
Found by TLA+ model checking and code review of the OM optimized snapshot diff
paths under HDDS-15926, on commit d3b75c75bf9ad475551202477c3a21a3f4b968b7.
Checked against HDDS issues and apache/ozone pull requests for duplicates
before filing. The attached patch is a proposal for review. Generated with
Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]