[
https://issues.apache.org/jira/browse/HDDS-8509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17719624#comment-17719624
]
Ethan Rose commented on HDDS-8509:
----------------------------------
[~ghuangups] some things you can check in the cluster to help diagnose the
problem:
# Make sure there is no data in the namespace. Key count in Recon Web UI
should be 0.
# Check the status of the containers with the block files in SCM: {{{}ozone
admin container list{}}}. You can get each replica's state for a given
container with {{{}ozone admin container info --json <container-id>{}}}.
Containers that are not closed or quasi-closed will not have deletes processed.
# Check if the containers are mapped to any keys in OM by curling the Recon
/api/v1/containers endpoint:
[https://ozone.apache.org/docs/current/interface/reconapi.html]. You will need
a new enough build with HDDS-5463 for this to show results for FSO buckets as
well.
# If there is not too much data in the cluster, check the status of various OM
tables that could be referencing the data: keyTable, fileTable, openKeyTable,
openFileTable, deletedTable. {{ozone debug ldb --db=om.db scan
--column-family=<table>}}
# Do a similar dump on the SCM deleted table to look for references to the
blocks: {{ozone debug ldb --db=scm.db scan --column-family=deletedBlocks}}
Devesh is working on Recon improvements that will allow us to better track this
information in the system without so many DB dumps required.
> Some block files were left behind under container chunks dir after deletion
> ----------------------------------------------------------------------------
>
> Key: HDDS-8509
> URL: https://issues.apache.org/jira/browse/HDDS-8509
> Project: Apache Ozone
> Issue Type: Bug
> Components: DN
> Reporter: George Huang
> Assignee: George Huang
> Priority: Major
>
> All keys, buckets and volumes were deleted, essentially no data in Ozone
> namespace. However, some block files were found left behind on some DNs for a
> closed container.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
