[
https://issues.apache.org/jira/browse/HDDS-17043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17043:
------------------------------
Attachment: TestBugMC4DoubleDeleteInOneDelta.java
> Recon emits a second DELETE event with a stale value when a key is deleted
> twice within one delta sync, and subtracts the key twice
> -----------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17043
> URL: https://issues.apache.org/jira/browse/HDDS-17043
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Minor
> Attachments:
> MC-4-drop-repeated-delete-event-in-one-recon-delta.patch,
> TestBugMC4DoubleDeleteInOneDelta.java
>
>
> h3. Mechanism
> {{OMDBUpdatesHandler.processEvent}} takes the old value of a key from the
> latest event for that key in the current Recon delta when there is one, and
> from the Recon OM DB otherwise. After a DELETE event that latest value is the
> row that was deleted. The DELETE branch drops an event when the old value is
> null, so a second DELETE of the same key in the same delta is emitted again
> with the stale row, and {{NSSummaryTaskDbEventHandler.handleDeleteKeyEvent}}
> subtracts that key a second time. Without the earlier event the old value
> would be read from the Recon OM DB, where the key is already gone, and the
> event would be dropped.
> OM produces this WAL shape in normal operation: {{OzoneManagerDoubleBuffer}}
> writes one flush as one batch, and {{RDBBatchOperation}} keeps only the last
> write per key, so a key that is committed and deleted again within one flush
> reaches the WAL as a bare DELETE.
> h3. Trigger
> # Key {{k}} exists when Recon last synced.
> # OM flush 1: {{k}} is deleted.
> # OM flush 2: {{k}} is created again (commit) and deleted again, both applied
> before the same double buffer flush.
> # One Recon delta sync covers both flushes.
> No fault is needed. Whether both deletes fall into one Recon delta and the
> re-create and delete into one OM flush depends on timing and load; I did not
> run this on a cluster.
> h3. Impact
> * The bucket (or the FSO parent and its ancestors) loses the key's file count
> and size twice. In the reproduction the bucket ended at 0 files and {{-50}}
> bytes instead of 1 file and 50 bytes.
> * The wrong totals are served by the namespace summary and disk usage
> endpoints and are kept by every later incremental sync until a full NSSummary
> rebuild. OM data is not affected.
> * Every Recon OM task receives the duplicate DELETE and subtracts it again:
> {{FileSizeCountTask}} decrements the file size bin of the stale value again,
> with no floor; {{OmTableInsightTask}} decrements the table's object count
> again (and, for the open key and deleted tables, the sizes);
> {{ContainerKeyMapperTask}} still finds the key's container rows, which are
> removed only when the batch is written, and decrements each container's key
> count again (argued from the code, not run).
> * Since when: OM writes a re-create and delete of a key in one batch as a
> bare DELETE since HDDS-8128, so 1.4.0 and later are affected.
> h3. Reproduction
> Deterministic, unmodified source. [^TestBugMC4DoubleDeleteInOneDelta.java]
> ({{hadoop-ozone/recon}}) applies real {{OMKeyCommitResponse}} and
> {{OMKeyDeleteResponse}} objects to a real OM DB, one batch per flush, reads
> the delta with {{RDBStore.getUpdatesSince}}, iterates each WAL batch into
> {{OMDBUpdatesHandler}} and applies it to the Recon OM DB as
> {{OzoneManagerServiceProviderImpl}} does, and runs {{NSSummaryTaskWithOBS}}.
> The expected totals come from a full rebuild of the same Recon OM DB. With
> the re-create and second delete in one flush, the handler emits two {{DELETE
> /vol1/ob1/k}} events with the same stale value (objectID 2002, 100 bytes) and
> the test fails with {{numOfFiles incremental vs rebuild ==> expected: <1> but
> was: <0>}}. The second test, with the re-create and second delete in separate
> flushes, passes. With the patch both pass.
> h3. Patch
> [^MC-4-drop-repeated-delete-event-in-one-recon-delta.patch], against
> 0665932dd35da4d05087e02748af0a64eb846ad5. It also applies to master at
> 2264d4d48ac.
> In the DELETE branch of {{OMDBUpdatesHandler.processEvent}}, a DELETE whose
> latest event in this delta is already a DELETE produces no event, as for any
> DELETE of a key Recon does not have. Later events for the key are unchanged.
> {{TestOMDBUpdatesHandler.testOperateOnSameEntry}} asserted the duplicate
> DELETE (7 events); it now expects 6 and keeps its other assertions. Without
> the change it fails with {{expected: <6> but was: <7>}}. With it the existing
> Recon task and handler suites pass and checkstyle is clean.
> Found by TLA+ model checking and code review of the Recon namespace summary
> and task sync paths under HDDS-15926, on commit
> 0665932dd35da4d05087e02748af0a64eb846ad5. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
