[
https://issues.apache.org/jira/browse/HDFS-8876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14662306#comment-14662306
]
Tsz Wo Nicholas Sze commented on HDFS-8876:
-------------------------------------------
Thanks for filing the JIRAs. I already have HDFS-8818 to speed up Balancer.
In my tests, it can speed up from 4GB per iteration to 800GB per iteration
while the duration of each iteration is only increased from ~1 minutes to 1-2
minutes.
I also have patches to remove some useless hard coded parameters and make the
other hard coded parameters configurable; see HDFS-8818 and HDFS-8824.
> Make hard coded parameters used by balancer and other tools configurable
> ------------------------------------------------------------------------
>
> Key: HDFS-8876
> URL: https://issues.apache.org/jira/browse/HDFS-8876
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Ming Ma
> Assignee: Chris Trezzo
>
> During investigation of how to speed up balancer, at least to the level
> specified by {{dfs.datanode.balance.bandwidthPerSec}}, we found that
> parameters such as {{MAX_BLOCKS_SIZE_TO_FETCH}} and
> {{SOURCE_BLOCKS_MIN_SIZE}} are hard coded. These parameters are related to
> block size and other configurable parameters used by balancer. So least we
> should make it configurable. In the longer term, it might be interesting to
> understand if we simplify all these related configurations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
