[
https://issues.apache.org/jira/browse/HDFS-12443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16238852#comment-16238852
]
Yiqun Lin edited comment on HDFS-12443 at 11/4/17 9:03 AM:
-----------------------------------------------------------
Thanks for the review, [~cheersyang]!
bq. Second thought, I think we can use
OZONE_BLOCK_DELETING_CONTAINER_LIMIT_PER_INTERVAL for calculation...
Good idea. Adressed.
bq. when checking the limit, we need to make sure we are checking the number of
containers of all TXs.
Addressed. Use container name as the check condition instead of TX id.
bq. line 133: maybe we should replace getAllNodes with get all healthy nodes by
getNodes(NodeState.HEALTHY).
Fixed.
bq. line 46: consider to replace hashmap to Multimap?
Done.
bq. Can we verify the deletion throttling by capture the DeleteBlocksCommand
SCM sent to DN and verify its size is expected?
Addressed.
Attach the updated patch.
was (Author: linyiqun):
Thanks for the review, [~cheersyang]!
bq. Second thought, I think we can use
OZONE_BLOCK_DELETING_CONTAINER_LIMIT_PER_INTERVAL for calculation...
Good idea. Adressed.
bq. when checking the limit, we need to make sure we are checking the number of
containers of all TXs.
Addressed. Use container name as the check condition instead of TX id.
bq. line 133: maybe we should replace getAllNodes with get all healthy nodes by
getNodes(NodeState.HEALTHY).
Fixed.
bq, line 46: consider to replace hashmap to Multimap?
Done.
bq. Can we verify the deletion throttling by capture the DeleteBlocksCommand
SCM sent to DN and verify its size is expected?
Addressed.
Attach the updated patch.
> Ozone: Improve SCM block deletion throttling algorithm
> -------------------------------------------------------
>
> Key: HDFS-12443
> URL: https://issues.apache.org/jira/browse/HDFS-12443
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone, scm
> Reporter: Weiwei Yang
> Assignee: Yiqun Lin
> Priority: Major
> Labels: OzonePostMerge
> Attachments: HDFS-12443-HDFS-7240.001.patch,
> HDFS-12443-HDFS-7240.002.patch, HDFS-12443-HDFS-7240.002.patch,
> HDFS-12443-HDFS-7240.003.patch, HDFS-12443-HDFS-7240.004.patch,
> HDFS-12443-SCM-blockdeletion-throttle.pdf
>
>
> Currently SCM scans delLog to send deletion transactions to datanode
> periodically, the throttling algorithm is simple, it scans at most
> {{BLOCK_DELETE_TX_PER_REQUEST_LIMIT}} (by default 50) at a time. This is
> non-optimal, worst case it might cache 50 TXs for 50 different DNs so each DN
> will only get 1 TX to proceed in an interval, this will make the deletion
> slow. An improvement to this is to make this throttling by datanode, e.g 50
> TXs per datanode per interval.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
