Siyao Meng created HDDS-17045:
---------------------------------
Summary: Recon adds a Legacy directory's totals to its ancestors
again on every in place update of the directory key, such as an ACL change
Key: HDDS-17045
URL: https://issues.apache.org/jira/browse/HDDS-17045
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: CR-3-resolve-old-parent-of-legacy-directory-update.patch,
TestBugCR3LegacyDirUpdateDoubleCount.java
h3. Mechanism
For an UPDATE event on a directory key in a LEGACY bucket with
{{ozone.om.enable.filesystem.paths=true}},
{{NSSummaryTaskWithLegacy.processWithFileSystemLayout}} builds the old
directory with {{oldKeyInfo.getParentObjectID()}}. keyTable entries do not
carry a parent object ID (it is 0), which is why the same method looks the
parent up by name with {{setKeyParentID}} for the new value and for files.
{{NSSummaryTaskDbEventHandler.handleDeleteDirEvent}} then finds no summary for
parent 0, logs "The namespace table is not correctly populated." and returns
without subtracting anything, while {{handlePutDirEvent}} adds the directory's
existing totals to its real parent and all ancestors.
FSO and OBS buckets, Legacy buckets without filesystem paths, and Legacy file
keys resolve the old parent correctly.
h3. Trigger
LEGACY bucket, {{ozone.om.enable.filesystem.paths=true}} (not the default).
Create directory {{dir1}} and a file {{dir1/file1}}, then change the ACL of the
key {{dir1/}} with {{ObjectStore.addAcl}}. The other requests that rewrite a
directory key in place ({{setAcl}}, {{removeAcl}}, {{setTimes}}) take the same
path (argued from the code, not run).
h3. Impact
* Each update of a non empty directory key adds the directory's file count and
size again to its parent and every ancestor up to the bucket, and so to the
volume and root totals, which the endpoints compute by adding up the bucket
summaries. In the reproduction one {{addAcl}} turned a bucket with 1 file and
100 bytes into 2 files and 200 bytes.
* The extra amount is never subtracted: later deletes subtract only the real
content. It is served by the namespace summary and disk usage endpoints until a
full NSSummary rebuild. OM data is not affected.
* Since when: the parent id has been read this way since HDDS-7121, but a
missed subtraction only became a double count when HDDS-13432 made directory
updates carry totals up the tree, so 2.1.0 and later are affected.
h3. Reproduction
Deterministic, unmodified source. [^TestBugCR3LegacyDirUpdateDoubleCount.java]
({{hadoop-ozone/integration-test-recon}}) starts a mini cluster with Recon and
filesystem paths enabled and uses only public client calls:
{{createDirectory("dir1")}}, a 100 byte {{dir1/file1}}, a Recon sync,
{{addAcl}} on {{dir1/}}, a 10 byte {{marker}} key, and another sync. The marker
shows when Recon has processed the sync with the ACL change. On the unfixed
code it fails with {{Bucket after ACL change on dir1/ and the marker key:
numOfFiles=3 sizeOfFiles=210 (expected 2 and 110)}}. With the patch it passes.
h3. Patch
[^CR-3-resolve-old-parent-of-legacy-directory-update.patch], against
0665932dd35da4d05087e02748af0a64eb846ad5. It also applies to master at
2264d4d48ac.
The old directory gets the parent already resolved for the new value. For an
UPDATE both values are stored under the same key and have the same key name, so
the parent is the same, and no second lookup is needed. Covered by the new
{{testProcessDirKeyUpdateKeepsAncestorTotals}} in the existing
{{TestNSSummaryTaskWithLegacy}} (an UPDATE on {{dir1/dir2/}} must leave
{{dir1}} and the bucket unchanged), and by {{testProcessUpdateFileSize}}, which
asserted the double count (2 files in a bucket that holds 1) and now expects 1.
Without the change both fail. With it the existing Recon NSSummary, endpoint
and event handler suites pass and checkstyle is clean.
Found by code review of the Recon namespace summary and task sync paths, as
part of the TLA+ verification effort under HDDS-15926, on commit
0665932dd35da4d05087e02748af0a64eb846ad5. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
