[
https://issues.apache.org/jira/browse/HDFS-12443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16233972#comment-16233972
]
Weiwei Yang commented on HDFS-12443:
------------------------------------
Hi [~linyiqun]
bq. add datanode info into delLog
This might not be a good option. It is not flexible if we have fixed
containerName to datanode mapping, because a container might be replicated to
another nodes if the original DN is lost.
bq. Scan the entire delLog from the beginning to end, getting blocks list info
for each node. If one node reach maximum container number, then its record will
be skipped.
I think this approach is the best for now. How you plan to define the max
number of containers for each node? Actually I am fine with a fixed number, e.g
50 for simplify the problem.
bq. If not, keep scanning log until it reach the maximum value.
Yes, good idea. I think we need a in-memory data structure to handle this. It
maintains a map, key is datanodeID and value is the a list of
{{DeletedBlocksTransaction}}, e.g DatanodeBlockDeletionTransactions. Each
datanodeID is bounded with a max size for the length of the
{{DeletedBlocksTransaction}}, it behaves like:
# a KV entry is full once the value reaches the max length, add more element to
this datanodeID will be skipped
# the map is full only when all KV entries are full
# each value has no duplicate element distinguished by the TXID
and each time we ensure the DelLog at most can be scanned once. Suggest to
write a separate test case to test such structure, to ensure the behavior is
well tested. then the implementation in SCMBlockDeletingService will be
straightforward.
Thanks for driving this forward, appreciate.
> Ozone: Improve SCM block deletion throttling algorithm
> -------------------------------------------------------
>
> Key: HDFS-12443
> URL: https://issues.apache.org/jira/browse/HDFS-12443
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone, scm
> Reporter: Weiwei Yang
> Assignee: Yiqun Lin
> Priority: Major
> Labels: OzonePostMerge
> Attachments: HDFS-12443-HDFS-7240.001.patch,
> HDFS-12443-HDFS-7240.002.patch, HDFS-12443-HDFS-7240.002.patch,
> HDFS-12443-SCM-blockdeletion-throttle.pdf
>
>
> Currently SCM scans delLog to send deletion transactions to datanode
> periodically, the throttling algorithm is simple, it scans at most
> {{BLOCK_DELETE_TX_PER_REQUEST_LIMIT}} (by default 50) at a time. This is
> non-optimal, worst case it might cache 50 TXs for 50 different DNs so each DN
> will only get 1 TX to proceed in an interval, this will make the deletion
> slow. An improvement to this is to make this throttling by datanode, e.g 50
> TXs per datanode per interval.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
