[
https://issues.apache.org/jira/browse/HDDS-6441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17506479#comment-17506479
]
Shawn commented on HDDS-6441:
-----------------------------
These are the questions from [~erose]
# Do you have datanode or SCM logs prior to this most recent restart? If there
is info on the history of this container that would be helpful.
2. Do you have an estimate of how many containers are in this state? Maybe
grepping all the DN logs for ‘Missing .container file’?
3. Using 15221 as an example, can you check if it has any keys associated with
it using Recon?
- From the docs in
[https://ozone.apache.org/docs/current/interface/reconapi.html], you can use
the {{GET /api/v1/containers/:id/keys}} API to see if there are any results.
4. Can you check the state of the other replicas of container 15221?
- {{ozone admin container info 15221}} will give the datanodes that have
the copy of that container.
- Can you check the contents of the container’s directory on the
datanodes shown by that command?
> Ozone metadata does not align with underlying blocks when there are many
> incomplete uploads happens
> ---------------------------------------------------------------------------------------------------
>
> Key: HDDS-6441
> URL: https://issues.apache.org/jira/browse/HDDS-6441
> Project: Apache Ozone
> Issue Type: Bug
> Components: Ozone Datanode
> Affects Versions: 1.2.0
> Reporter: Shawn
> Assignee: Ethan Rose
> Priority: Major
>
> Ozone metadata does not align with underlying blocks when there are many
> incomplete uploads happens. I have a cluster which has a very few objects.
> But the datanode usage tells me I almost run out of space.
> ????
> Usage info for datanode with UUID f50108f1-d8bf-44e3-abed-6e77c91f994d:
> Capacity : 8802545958912B
> SCMUsed : 8802128257024B (99.99525%)
> Remaining : 74715136B (0.00085%)
> Usage info for datanode with UUID 2bdb3198-b71f-4153-9663-e3b349c6f82a:
> Capacity : 8802545958912B
> SCMUsed : 8802133102592B (99.99531%)
> Remaining : 76824576B (0.00087%)
> Usage info for datanode with UUID d5644a36-b967-44a6-a736-4bd2013c2b86:
> Capacity : 8793955991552B
> SCMUsed : 8793311227904B (99.99267%)
> Remaining : 291676160B (0.00332%)
> ...
>
> Also I see there are lots of errors in logs, complaining out of disk space
> and also report missing .container files as below:
> ????
> 2022-03-10 03:56:02 ERROR Thread-6 ContainerReader:159 - Missing .container
> file for ContainerID: 15221
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
