[
https://issues.apache.org/jira/browse/HDFS-8875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14662231#comment-14662231
]
Chris Trezzo commented on HDFS-8875:
------------------------------------
[~mingma] Couple of thoughts about this issue:
# For the {{Collections.shuffle(connectors);}} call: I can see this being
advantageous in the scenario where the balancer is constantly behind. With the
shuffle, you won't always start with the same namespace.
# For the sleep time between iterations: Even with federation, we still might
run into the case where we would want to sleep between iterations. For example,
lets say that only one namespace still needs balancing. The other namespaces
quickly finish their iteration and we are back to the same namespace. Maybe we
can check for the amount of time each iteration took. If that time is greater
than sleep time, then we skip the sleep.
> Optimize the wait time in Balancer for federation scenario
> ----------------------------------------------------------
>
> Key: HDFS-8875
> URL: https://issues.apache.org/jira/browse/HDFS-8875
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Ming Ma
> Assignee: Chris Trezzo
>
> Balancer has wait time between two consecutive iterations. That is to give
> some time for block movement to be fully committed ( return from replaceBlock
> doesn't mean the NN's blockmap has been updated and the block has been
> invalidated on the source node.).
> This wait time could be 23 seconds if {{dfs.heartbeat.interval}} is set to 10
> and {{dfs.namenode.replication.interval}} is to 3. In the case of federation,
> given we iterate through all namespaces in each iteration, this wait time
> becomes unnecessary as while balancer is processing the next namespace, it
> gives the previous namespace it just finished time to commit.
> In addition, Balancer calls {{Collections.shuffle(connectors);}} It doesn't
> seem necessary.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
