[
https://issues.apache.org/jira/browse/HDDS-7483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17641596#comment-17641596
]
Ethan Rose commented on HDDS-7483:
----------------------------------
[~sumitagrawl] approaches that deal with a separate buffer for transactions
will have issues since SCM RocksDB is the Ratis state machine in SCM HA. When a
transaction like delete blocks comes to the leader SCM and goes through Ratis,
it is first appended to the log, then sent to SCMStateMachine#applyTransaction
to commit. SCM applyTransaction will actually buffer the entry in memory on
apply and depend on the log for durability. Ratis snapshot (once every 1000
transactions) or SCM restart will cause the log transactions to be applied to
the state machine in the order present in the log.
With a separate buffer/flush strategy for delete transactions, they still go
through Ratis and show up in the log, but are being added and removed from the
state machine outside of the global apply order established by Ratis. This
could cause divergence among SCM leader and followers or repeat transactions.
For example:
1. Delete txn 1 is committed to the log and DB
2. Delete txn 1 is processed and removed from the DB
3. Ratis snapshot or restart occurs. Delete txn 1 is flushed to the DB from the
log so it reappears in the DB
Since delete transactions are idempotent, this might not cause a correctness
problem from what I can immediately see. However it will be confusing none the
less and probably something we want to avoid. Since delete transactions must go
through Ratis for durability, I think it might be difficult to persist them
with a mechanism outside of Ratis snapshots.
> DN do not remove blocks for deleted content
> -------------------------------------------
>
> Key: HDDS-7483
> URL: https://issues.apache.org/jira/browse/HDDS-7483
> Project: Apache Ozone
> Issue Type: Sub-task
> Reporter: Sumit Agrawal
> Assignee: Sumit Agrawal
> Priority: Major
>
> When delete directory and files is performed, its not deleted in local disk
> of DN.
>
> # SCM keeps transaction information in buffer and commits to DB after a set
> of transaction (like 10,000)
> # DeleteBlockService on SCM gets information from rocks DB and its not
> available as its present in buffer, so delete list size is empty
> This may needs resolve to delete blocks from DN if its already ready for
> deletion.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
