[
https://issues.apache.org/jira/browse/HDDS-7728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17765859#comment-17765859
]
Ethan Rose commented on HDDS-7728:
----------------------------------
{quote}Hi Stephen O'Donnell, Ethan Rose, I'm not sure if we are thinking the
same case, could you explain a little bit more about what the orphan block and
orphan container situations mentioned,
{quote}
We are trying to handle the cases with block deletes when there are no replicas
of a container left, or when the delete is already processed and a stale
replica with the block comes back.
{quote}and how is it related with the current proposal that if there are 4
replicas of a over replicated container, which one is a better candidate to
delete? So that we can on the same page.
{quote}
This is only relevant if we are using delete transactions in the RM to make
sure all deletes are processed. Stephen and I are saying that this might not be
the best way to handle the problem.
{quote}Unfortunately Recon also become more and more complex now. Is it
possible for Recon to make a better decision than RM without knowing all the
information which are easy to get by RM? I'm kind of doubt that.
{quote}
Recon already has a container -> key mapping exposed through the REST API and
the new OM DB Insights page. This makes orphan containers easy to identify
without adding new background processing to Recon. Let me clarify the proposal:
- When Recon is building the container -> key mapping, it will save a list of
containers that have no keys mapped to them.
- Using a new SCM API, it would instruct SCM to force delete those containers.
This would be the first time Recon is used to modify Ozone's state, but it
would remain optional as orphan blocks/containers do not prevent the system
from functioning.
- When SCM gets this command, it would go through the usual force delete
container flow. Block delete transactions for deleted containers should later
be removed by the SCM block deleting service. I believe the existing code
already handles this case.
*Pros*:
- Missing containers can be removed from the system when all their keys are
deleted.
- Orphan blocks (however they happened) can be removed from the system with
their containers when no data is mapped to the container anymore.
- Leverages existing jobs in Recon and SCM. The only new feature is connecting
them with an API.
*Cons*:
- Recon taking a more active role in cluster management.
- New SCM API could cause data loss if used incorrectly. We would need to be
absolutely sure Recon is the only one who can call it.
- Orphan block cleanup is delayed until the container is effectively empty of
namespace data.
- Orphan container cleanup should not race with the regular block deletion
flow. Just because Recon sees no keys mapped to the container, the block
deletes may still be correctly progressing through the system.
> Block should be safely deleted from the containers if they are instructed
> from OM and containers are in missing state.
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-7728
> URL: https://issues.apache.org/jira/browse/HDDS-7728
> Project: Apache Ozone
> Issue Type: Improvement
> Components: SCM
> Affects Versions: 1.3.0
> Reporter: Uma Maheswara Rao G
> Assignee: Ashish Kumar
> Priority: Major
>
> Currently when OM instructs to delete the blocks and if containers are in
> missing state, deletion may not be processed properly. This Jira to track
> this requirement and implement to safe deletion os blocks what ever state
> they are on. Otherwise containers would never get cleaned up even though all
> blocks in that files deleted.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
