[
https://issues.apache.org/jira/browse/HDDS-12829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ivan Andika updated HDDS-12829:
-------------------------------
Description:
Currently, ContainerStateMachine#truncate only remove in-memory write chunk
data from stateMachineDataCache although ContainerStateMachine#write might have
already written the chunks to the disk. Unlike other orphan block cases, these
chunks might not stored in the Datanode BlockDataTable which means that
comparing the OM metadata and DN block metadata might not catch these chunks.
We can add an additional logic to ContainerStateMachine#truncate which will
cleanup uncommitted chunks to reduce the orphan blocks / chunks in datanodes
caused by INCONSISTENCY during writes. The cleanup can be done asynchronously
in another thread (e.g. using chunkExecutor) and can be done in a best-effort
basis.
was:
Currently, ContainerStateMachine#truncate only remove in-memory write chunk
data from stateMachineDataCache although ContainerStateMachine#write might have
already written the chunks to the disk. Unlike other orphan block cases, these
chunks might not stored in the Datanode BlockDataTable which means that
comparing the OM metadata and DN block metadata might not catch these chunks.
We can add an additional logic to ContainerStateMachine#truncate which will
cleanup uncommitted chunks to reduce the orphan blocks / chunks in datanodes
caused by INCONSISTENCY during writes. The cleanup can be done asynchronously
in another thread and can be done in a best-effort basis.
> Cleanup uncommitted chunks during ContainerStateMachine#truncate
> ----------------------------------------------------------------
>
> Key: HDDS-12829
> URL: https://issues.apache.org/jira/browse/HDDS-12829
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: Ozone Datanode
> Reporter: Ivan Andika
> Assignee: Ivan Andika
> Priority: Major
>
> Currently, ContainerStateMachine#truncate only remove in-memory write chunk
> data from stateMachineDataCache although ContainerStateMachine#write might
> have already written the chunks to the disk. Unlike other orphan block cases,
> these chunks might not stored in the Datanode BlockDataTable which means that
> comparing the OM metadata and DN block metadata might not catch these chunks.
> We can add an additional logic to ContainerStateMachine#truncate which will
> cleanup uncommitted chunks to reduce the orphan blocks / chunks in datanodes
> caused by INCONSISTENCY during writes. The cleanup can be done asynchronously
> in another thread (e.g. using chunkExecutor) and can be done in a best-effort
> basis.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
