[
https://issues.apache.org/jira/browse/HDFS-11104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15659720#comment-15659720
]
Ajith S commented on HDFS-11104:
--------------------------------
Would like to work on this, can u please assign to me
> BlockPlacementPolicyDefault choose favoredNodes in turn which may cause
> imbalance
> ---------------------------------------------------------------------------------
>
> Key: HDFS-11104
> URL: https://issues.apache.org/jira/browse/HDFS-11104
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: Doris Gu
>
> if client transfer favoredNodes when it writes files into hdfs,chooseTarget
> in BlockPlacementPolicyDefault prior chooseTarget in turn:
> {quote}
> DatanodeStorageInfo[] chooseTarget(String src,
> int numOfReplicas,
> Node writer,
> Set<Node> excludedNodes,
> long blocksize,
> List<DatanodeDescriptor> favoredNodes,
> BlockStoragePolicy storagePolicy) {
> try {
> ...
> *for (int i = 0; i < favoredNodes.size() && results.size() <
> numOfReplicas; i++)* {
> DatanodeDescriptor favoredNode = favoredNodes.get(i);
> // Choose a single node which is local to favoredNode.
> // 'results' is updated within chooseLocalNode
> final DatanodeStorageInfo target = chooseLocalStorage(favoredNode,
> favoriteAndExcludedNodes, blocksize, maxNodesPerRack,
> results, avoidStaleNodes, storageTypes, false);
> ...
> {quote}
> why not shuffle it here? Make block more balanced, save the cost balancer
> will pay and make cluster more stable.
> {quote}
> for (DatanodeDescriptor favoredNode :
> DFSUtil.shuffle(favoredNodes.toArray(new
> DatanodeDescriptor[favoredNodes.size()])))
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
