Siyao Meng created HDDS-17043:
---------------------------------
Summary: Recon emits a second DELETE event with a stale value when
a key is deleted twice within one delta sync, and subtracts the key twice
Key: HDDS-17043
URL: https://issues.apache.org/jira/browse/HDDS-17043
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: MC-4-drop-repeated-delete-event-in-one-recon-delta.patch,
TestBugMC4DoubleDeleteInOneDelta.java
h3. Mechanism
{{OMDBUpdatesHandler.processEvent}} takes the old value of a key from the
latest event for that key in the current Recon delta when there is one, and
from the Recon OM DB otherwise. After a DELETE event that latest value is the
row that was deleted. The DELETE branch drops an event when the old value is
null, so a second DELETE of the same key in the same delta is emitted again
with the stale row, and {{NSSummaryTaskDbEventHandler.handleDeleteKeyEvent}}
subtracts that key a second time. Without the earlier event the old value would
be read from the Recon OM DB, where the key is already gone, and the event
would be dropped.
OM produces this WAL shape in normal operation: {{OzoneManagerDoubleBuffer}}
writes one flush as one batch, and {{RDBBatchOperation}} keeps only the last
write per key, so a key that is committed and deleted again within one flush
reaches the WAL as a bare DELETE.
h3. Trigger
# Key {{k}} exists when Recon last synced.
# OM flush 1: {{k}} is deleted.
# OM flush 2: {{k}} is created again (commit) and deleted again, both applied
before the same double buffer flush.
# One Recon delta sync covers both flushes.
No fault is needed. Whether both deletes fall into one Recon delta and the
re-create and delete into one OM flush depends on timing and load; I did not
run this on a cluster.
h3. Impact
* The bucket (or the FSO parent and its ancestors) loses the key's file count
and size twice. In the reproduction the bucket ended at 0 files and {{-50}}
bytes instead of 1 file and 50 bytes.
* The wrong totals are served by the namespace summary and disk usage endpoints
and are kept by every later incremental sync until a full NSSummary rebuild. OM
data is not affected.
* Every Recon OM task receives the duplicate DELETE and subtracts it again:
{{FileSizeCountTask}} decrements the file size bin of the stale value again,
with no floor; {{OmTableInsightTask}} decrements the table's object count again
(and, for the open key and deleted tables, the sizes);
{{ContainerKeyMapperTask}} still finds the key's container rows, which are
removed only when the batch is written, and decrements each container's key
count again (argued from the code, not run).
* Since when: OM writes a re-create and delete of a key in one batch as a bare
DELETE since HDDS-8128, so 1.4.0 and later are affected.
h3. Reproduction
Deterministic, unmodified source. [^TestBugMC4DoubleDeleteInOneDelta.java]
({{hadoop-ozone/recon}}) applies real {{OMKeyCommitResponse}} and
{{OMKeyDeleteResponse}} objects to a real OM DB, one batch per flush, reads the
delta with {{RDBStore.getUpdatesSince}}, iterates each WAL batch into
{{OMDBUpdatesHandler}} and applies it to the Recon OM DB as
{{OzoneManagerServiceProviderImpl}} does, and runs {{NSSummaryTaskWithOBS}}.
The expected totals come from a full rebuild of the same Recon OM DB. With the
re-create and second delete in one flush, the handler emits two {{DELETE
/vol1/ob1/k}} events with the same stale value (objectID 2002, 100 bytes) and
the test fails with {{numOfFiles incremental vs rebuild ==> expected: <1> but
was: <0>}}. The second test, with the re-create and second delete in separate
flushes, passes. With the patch both pass.
h3. Patch
[^MC-4-drop-repeated-delete-event-in-one-recon-delta.patch], against
0665932dd35da4d05087e02748af0a64eb846ad5. It also applies to master at
2264d4d48ac.
In the DELETE branch of {{OMDBUpdatesHandler.processEvent}}, a DELETE whose
latest event in this delta is already a DELETE produces no event, as for any
DELETE of a key Recon does not have. Later events for the key are unchanged.
{{TestOMDBUpdatesHandler.testOperateOnSameEntry}} asserted the duplicate DELETE
(7 events); it now expects 6 and keeps its other assertions. Without the change
it fails with {{expected: <6> but was: <7>}}. With it the existing Recon task
and handler suites pass and checkstyle is clean.
Found by TLA+ model checking and code review of the Recon namespace summary and
task sync paths under HDDS-15926, on commit
0665932dd35da4d05087e02748af0a64eb846ad5. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
