[ 
https://issues.apache.org/jira/browse/HDFS-11015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15586821#comment-15586821
 ] 

Zhe Zhang commented on HDFS-11015:
----------------------------------

Thanks Kihwal for sharing the results; they look really interesting!

bq. However, the removal of the long timeout wasn't necessary. The timeout is 
still useful for avoiding hung nodes and does not abort slow moves.
I agree. We should use a combination of {{IN_PROGRESS}} heartbeating and socket 
timeout.

Regarding {{DFS_BALANCER_BLOCK_MOVE_TIMEOUT}}:
# Since it's named and documented as a Balancer config knob, I guess we should 
initiate it inside {{Balancer}} and pass it to {{Dispatcher}}? Otherwise 
{{Mover}} is silently affected by this config.
# Nit: "3 to 5 minute" should be "3 to 5 minutes"

Regarding the 20 mins timeout:
# There are only two {{while}} loops in {{Dispatcher}} so I think your patch 
does a pretty good job in enforcing the timeout. Can we simplify the loops by 
merging the conditions into the {{while}} statement? Especially the one in 
{{dispatchBlocks}}, {{isTimeUp}} isn't necessary.

Overall the patch looks pretty good. I realized that it doesn't really cover 
the HDFS-10977 scenario. May I suggest we update the subject to "Enforce 
iteration time limit and slow block move timeout in Balancer"? It doesn't look 
easy to add a unit test and I'm OK in this case because the added logic is 
pretty straightforward. I'm happy to open a separate test JIRA though.

> Enforce timeout in balancer
> ---------------------------
>
>                 Key: HDFS-11015
>                 URL: https://issues.apache.org/jira/browse/HDFS-11015
>             Project: Hadoop HDFS
>          Issue Type: Bug
>            Reporter: Kihwal Lee
>            Assignee: Kihwal Lee
>         Attachments: HDFS-11015-1.patch, balancer.png
>
>
> 1) Hung node detection: HDFS-6247 has removed the socket read timeout while 
> adding the periodic response for slow block moves. However, the removal of 
> the long timeout wasn't necessary.  The timeout is still useful for avoiding 
> hung nodes and does not abort slow moves.
> 2) Enforcing the iteration limit:The 20 minute iteration limit is supposed to 
> be enforced, but it is not. An iteration can easily stretch to 30 to 40 
> minutes with a long tail. Because of the long tails, the balancer throughput 
> does not reach its full potential.
> 3) Slow move detection: For improved throughput, imposing block move timeout 
> is sometimes necessary.  We have seen an iteration taking over 2 hours 
> because of one slow block move.  This is mainly for catching exceptionally 
> slow moves.  Even if the balancer stops waiting, the move will continue and 
> finish.
> In order to not undo what  HDFS-6247 tried to achieve, it should be possible 
> to configure off 3).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

Reply via email to