[
https://issues.apache.org/jira/browse/HDDS-6441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17533977#comment-17533977
]
Ethan Rose commented on HDDS-6441:
----------------------------------
{quote}
Currently which aws s3cli version does ozone support?
{quote}
I am not sure we have published an official range of supported versions, but it
looks like there has been some discussion on the slack channel based on our
testing ([linking here for reference to future
readers|https://the-asf.slack.com/archives/C5RK7PWA1/p1651873995781939]).
{quote}
I used the aws s3 JDK to list multipart uploads, and abort each of them for my
main bucket (totally 378 MPUs). However, the space usage is not changed at all.
{quote}
Several hours should be plenty of time for deletes to propagate back to the
datanodes, so something may be going wrong here. You may have already checked
this, but you can use {{ozone debug chunkinfo}} to get the specific block
files, and see if those block files are remaining on the datanodes. I will
follow up with you on slack to try to debug the issue and post the resolution
here.
{quote}
For the containers which is missing .container file, is it OK to manually
delete all replicas of this container?
{quote}
Do not delete all replicas, because sometimes container instances that still
have data are deleted from a datanode to compensate for over-replication. Since
we suspect these were caused by HDDS-6449, the containers should be either
empty or the result of over-replication. For each container missing the
.container file, use {{ozone admin container info <containerID>}} to get the
current replica list for that container. If the replica list does not contain
the datanode logging the error, or the container is in a deleted state, you are
safe to remove the artifacts of that container from the disk on that datanode.
If neither of these are true, the issue may be different than HDDS-6449 and we
should investigate more.
> Ozone metadata does not align with underlying blocks when there are many
> incomplete uploads happens
> ---------------------------------------------------------------------------------------------------
>
> Key: HDDS-6441
> URL: https://issues.apache.org/jira/browse/HDDS-6441
> Project: Apache Ozone
> Issue Type: Bug
> Components: Ozone Datanode
> Affects Versions: 1.2.0
> Reporter: Shawn
> Assignee: Ethan Rose
> Priority: Major
>
> Ozone metadata does not align with underlying blocks when there are many
> incomplete uploads happens. I have a cluster which has a very few objects.
> But the datanode usage tells me I almost run out of space.
> ????
> Usage info for datanode with UUID f50108f1-d8bf-44e3-abed-6e77c91f994d:
> Capacity : 8802545958912B
> SCMUsed : 8802128257024B (99.99525%)
> Remaining : 74715136B (0.00085%)
> Usage info for datanode with UUID 2bdb3198-b71f-4153-9663-e3b349c6f82a:
> Capacity : 8802545958912B
> SCMUsed : 8802133102592B (99.99531%)
> Remaining : 76824576B (0.00087%)
> Usage info for datanode with UUID d5644a36-b967-44a6-a736-4bd2013c2b86:
> Capacity : 8793955991552B
> SCMUsed : 8793311227904B (99.99267%)
> Remaining : 291676160B (0.00332%)
> ...
>
> Also I see there are lots of errors in logs, complaining out of disk space
> and also report missing .container files as below:
> ????
> 2022-03-10 03:56:02 ERROR Thread-6 ContainerReader:159 - Missing .container
> file for ContainerID: 15221
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
