[
https://issues.apache.org/jira/browse/HDFS-12354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16143402#comment-16143402
]
Yiqun Lin commented on HDFS-12354:
----------------------------------
Thanks for the comments, [~cheersyang].
bq. Do you think if we can extend ContainerStatus to have another field such
as numOfPendingDeletionBlocks?
Adding new field in ContainerStatus seems need some more adjustment work for
this.
I noticed one detail that you mentioned there might more than 4000 containers
in a datanode. Like this scenery, we will still meets some problems if we
support get a shuffle list. Each time we only chose specified number of
containers, there is a chance that some containers will not be chosen in a long
time when the specified number is small while the whole containers is large. So
based on this, I am thinking can we use the round-robin way? We can pass
previous key and invoke {{containerMap#tailMap}} to select the next containers
that these container name is great than previous key.
For the pending under blocks based way, I prefer to implement this later and
firstly implement above two ways (The randomly choosing way and round-robin
way). These two ways only need minor change but will have a good effect on
container choosing.
> Ozone: Shuffle container list for datanode BlockDeletingService
> ---------------------------------------------------------------
>
> Key: HDFS-12354
> URL: https://issues.apache.org/jira/browse/HDFS-12354
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, ozone
> Reporter: Weiwei Yang
> Assignee: Yiqun Lin
>
> {{BlockDeletingService}} is a per-datanode container block deleting service
> takes in charge of the "real" deletion of ozone blocks. It spawns a worker
> thread per container and delete blocks/chunks from disk as background
> threads. The number of threads currently is throttled by
> {{ozone.block.deleting.container.limit.per.interval}}, but there is a
> potential problem. Containers are sorted so it always fetch same of
> containers, we need to fix this by creating an API in
> {{ContainerManagerImpl}} to get a shuffled list of containers.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
