[
https://issues.apache.org/jira/browse/HDDS-6301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17499209#comment-17499209
]
Ethan Rose commented on HDDS-6301:
----------------------------------
This turned out to be a byproduct of the existing missing chunks dir bug
HDDS-6235. The block deleting service on the datanode is running and deleting
blocks, but it is operating very slowly. Each run it picks containers to delete
from until the total pending delete block counts meet its threshold, then it
tries to delete them. Some of these empty chunks directory containers have
pending delete blocks (but no actual block files), so they are being included
in the batch. Since there's no chunks dir, deletion fails and the pending
delete block count never gets updated, leading to these same containers to be
picked on the next run and sucking up the block threshold. Here's an extract
from the datanode log that showed this happening:
{code:java}
Plan to choose 5000 blocks for block deletion, actually deleting 3141 blocks.
...
Container: 266899, deleted blocks: 7, task elapsed time: 8ms
{code}
As a result block deletion on this datanode is effectively operating at 7
deletions per run on this cluster, when ideally it would be hitting 5000 per
run.
> containerDir left over after volume/bucket/key deletion
> -------------------------------------------------------
>
> Key: HDDS-6301
> URL: https://issues.apache.org/jira/browse/HDDS-6301
> Project: Apache Ozone
> Issue Type: Bug
> Components: Ozone Manager, SCM
> Reporter: George Huang
> Assignee: Ethan Rose
> Priority: Major
>
> Ozone volume/bucket/key were deleted via command line using command like:
> ozone fs -rm -skipTrash -r ofs://ozone1/vol-1-49431
> Now check data nodes, many containerDir### were left over on the data node:
> [root@ve1135 current]# pwd
> /data/1/hadoop-ozone/datanode/data8/hdds/CID-f72ad861-4e70-4fe5-982b-79ce831c6d31/current
> [root@ve1135 current]# ls -al
> ./containerDir1/262719/chunks/107544261459853384.block
> -rw-r--r-- 1 hdfs hdfs 10240 Jan 31 22:23
> ./containerDir1/262719/chunks/107544261459853384.block
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
