[
https://issues.apache.org/jira/browse/HDFS-15640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17217678#comment-17217678
]
Yiqun Lin commented on HDFS-15640:
----------------------------------
[~LiJinglun] , use distcp execution time as fedbalance threshold is not an
appropriate way. The execution time can be impacted by other aspects, like no
enough resource to schedule task or slow rpc calls.
I prefer to use the snapshot diff entries number as the threshold here. We
could use getSnapshotDiffReport API to get this info. If snapshot diff entries
reduced to a very low number value, that means only few files/dirs needed be
synced. And then we can prepare to do the final distcp copy.
> RBF: Add fast distcp threshold to FedBalance.
> ---------------------------------------------
>
> Key: HDFS-15640
> URL: https://issues.apache.org/jira/browse/HDFS-15640
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Jinglun
> Assignee: Jinglun
> Priority: Major
> Attachments: HDFS-15640.001.patch
>
>
> Currently in the DistCpProcedure it must submit distcp round by round until
> there is no diff to go to the final distcp stage. The condition is very
> strict. If the distcp could finish in an acceptable period then we don't need
> to wait for no diff. For example if 3 consecutive distcp jobs all finish
> within 10 minutes then we can predict the final distcp could also finish
> within 10 minutes. So we can start the final distcp directly.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
