[
https://issues.apache.org/jira/browse/HDFS-11384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15891198#comment-15891198
]
yunjiong zhao commented on HDFS-11384:
--------------------------------------
Thank you [~benoyantony] for your time to review this patch.
{quote}Sleeping inside the Synchronized block should be avoided as it will
prevent other threads from obtaining the lock while the thread is sleeping.
{quote}
I did it on purpose for sleeping inside the Synchronized block.
In balancer there are multiple threads (by default 200) that may call getBlocks
at same time, if user need to set dfs.balancer.getBlocks.interval.millis to
slow down balancer, without a lock it won't work well due to at worst case
there are still 200 getBlocks send to NameNode at same time.
{quote}It will be better to keep track of the interval between successive
getBlocks and sleep only for the required time. {quote}
Since by default, this patch doesn't change anything, only add a option let
user slow down balancer send getBlocks to NameNode, so I'd like to keep it as
simple as possible.
> Add option for balancer to disperse getBlocks calls to avoid NameNode's
> rpc.CallQueueLength spike
> -------------------------------------------------------------------------------------------------
>
> Key: HDFS-11384
> URL: https://issues.apache.org/jira/browse/HDFS-11384
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer & mover
> Affects Versions: 2.7.3
> Reporter: yunjiong zhao
> Assignee: yunjiong zhao
> Attachments: balancer.day.png, balancer.week.png, HDFS-11384.001.patch
>
>
> When running balancer on hadoop cluster which have more than 3000 Datanodes
> will cause NameNode's rpc.CallQueueLength spike. We observed this situation
> could cause Hbase cluster failure due to RegionServer's WAL timeout.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
