[
https://issues.apache.org/jira/browse/HDFS-12443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16235120#comment-16235120
]
Yiqun Lin commented on HDFS-12443:
----------------------------------
Thanks comments, [~cheersyang]. I think we are in the same direction now. Some
details I'd like to confirm with you.
bq. How you plan to define the max number of containers for each node?
I'd like to calculated this based on container, block size that was configured.
The Calculation way I had mentioned in above comment. Please have a look.
bq. I think we need a in-memory data structure to handle this...
For this new data structure, I'd like to make a change based on current class
{{DatanodeBlockDeletionTransactions}} and to make this being a independent
class. That will be convenient for us to test.
Please see if it looks good to you or any suggestion. Then I will start work on
this. Thank you.
> Ozone: Improve SCM block deletion throttling algorithm
> -------------------------------------------------------
>
> Key: HDFS-12443
> URL: https://issues.apache.org/jira/browse/HDFS-12443
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone, scm
> Reporter: Weiwei Yang
> Assignee: Yiqun Lin
> Priority: Major
> Labels: OzonePostMerge
> Attachments: HDFS-12443-HDFS-7240.001.patch,
> HDFS-12443-HDFS-7240.002.patch, HDFS-12443-HDFS-7240.002.patch,
> HDFS-12443-SCM-blockdeletion-throttle.pdf
>
>
> Currently SCM scans delLog to send deletion transactions to datanode
> periodically, the throttling algorithm is simple, it scans at most
> {{BLOCK_DELETE_TX_PER_REQUEST_LIMIT}} (by default 50) at a time. This is
> non-optimal, worst case it might cache 50 TXs for 50 different DNs so each DN
> will only get 1 TX to proceed in an interval, this will make the deletion
> slow. An improvement to this is to make this throttling by datanode, e.g 50
> TXs per datanode per interval.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
