[
https://issues.apache.org/jira/browse/HDDS-7111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sumit Agrawal updated HDDS-7111:
--------------------------------
Parent: HDDS-8485
Issue Type: Sub-task (was: Improvement)
> SCMHADBTransactionBuffer not flushed in time
> --------------------------------------------
>
> Key: HDDS-7111
> URL: https://issues.apache.org/jira/browse/HDDS-7111
> Project: Apache Ozone
> Issue Type: Sub-task
> Reporter: Xu Shao Hong
> Priority: Major
> Labels: pull-request-available
> Fix For: HDDS-7483
>
>
> We recently found that after we deleted all the keys, the cluster will remain
> with some data. One of the reasons is due to this:
> SCM will add deleted blocks into transactions when receiving the request from
> OM. when HA is enabled, DBTransactionbuffer is implemented as the
> SCMHADBTransactionbufferImpl. Inside this, the buffer will not be flushed
> immediately. Normally, it will be flushed when SCM takes a snapshot. The
> snapshot gap threshold is default 1000. If the user has little load pressure
> on the cluster (no writing more ratis logs), the buffer will be always
> pending in the memory. Real deletion happened in SCMBlockDeletingService,
> which will scan the DB and get the transactions, for those txns in-memory
> buffer, it won't find them.
> This is why DN has not yet received these deleted block info.
> We have three solutions:
> # add a global view for all the deleted block transactions, but it will
> bring too much pressure on the memory.
> # use `SCMDBTransactionBufferImpl` instead of SCMHADBTransactionBufferImpl
> in `DeletedBlockLog`. It could work but may be confusing. Why does it need
> the direst flush independently?
> # Add a flush monitor/ check to trigger the non-empty flush, overall, this
> could be a common mechanism if some other cases like the deleted blocks
> appears in the future.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
