[
https://issues.apache.org/jira/browse/HDFS-11742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16049905#comment-16049905
]
Kihwal Lee commented on HDFS-11742:
-----------------------------------
bq. But may be we want to understand more closely at the root cause of the
behavior before we commit?
HDFS-8818 made a thread pool to be created per target. The size of each thread
pool is fixed to {{dfs.datanode.balance.max.concurrent.moves}}(default=50).
When the number of targets * max concurrent moves exceeds the configured mover
thread limit (default=1000), it stops creating thread pools. The end result is
that not all calculated moves are executed.
So when the default config is used, if an iteration involves more than 20
targets (1000/50), some blocks won't be moved. We have clusters where
balancing involves hundreds of targets and the effect of this limitation is
very visible. My patch dynamically sizes each thread pool based on the number
of targets in the iteration and the configured maximum mover threads to
mitigate this problem. It does not limit the capability of HDFS-8818. It simply
makes it possible to continue to use the default value or adjust it without
much side effect.
> Improve balancer usability after HDFS-8818
> ------------------------------------------
>
> Key: HDFS-11742
> URL: https://issues.apache.org/jira/browse/HDFS-11742
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: Kihwal Lee
> Assignee: Kihwal Lee
> Priority: Blocker
> Labels: release-blocker
> Attachments: balancer2.8.png, balancer_fix.png,
> HDFS-11742.branch-2.8.patch, HDFS-11742.branch-2.patch,
> HDFS-11742.trunk.patch, HDFS-11742.v2.trunk.patch, replaceBlockNumOps-8w.jpg
>
>
> We ran 2.8 balancer with HDFS-8818 on a 280-node and a 2,400-node cluster. In
> both cases, it would hang forever after two iterations. The two iterations
> were also moving things at a significantly lower rate. The hang itself is
> fixed by HDFS-11377, but the design limitation remains, so the balancer
> throughput ends up actually lower.
> Instead of reverting HDFS-8188 as originally suggested, I am making a small
> change to make it less error prone and more usable.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
