[
https://issues.apache.org/jira/browse/HDFS-1312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15068911#comment-15068911
]
Lei (Eddy) Xu commented on HDFS-1312:
-------------------------------------
Hey, [~anu].
Thanks a lot for working on this. It is a very demanding feature.
I have a few questions regarding the design, much appreciated if you can answer
them to help me to better understand the design.
* Will disk balancer be a daemon or a CLI tool, that waits the process to
finishes?
* Where is the Planner executed? A DiskBalancer daemon / CLI tool or NN?
* When copying a replica from one volume to another, how to prevent it to have
concurrent issues with the {{DirectoryScanner}} in background?
* Is there a limitation of how many such disk balancer jobs can run in the
cluster? Could other job queries the status of a running job?
Thanks again!
> Re-balance disks within a Datanode
> ----------------------------------
>
> Key: HDFS-1312
> URL: https://issues.apache.org/jira/browse/HDFS-1312
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode
> Reporter: Travis Crawford
> Assignee: Anu Engineer
> Attachments: Architecture_and_testplan.pdf, disk-balancer-proposal.pdf
>
>
> Filing this issue in response to ``full disk woes`` on hdfs-user.
> Datanodes fill their storage directories unevenly, leading to situations
> where certain disks are full while others are significantly less used. Users
> at many different sites have experienced this issue, and HDFS administrators
> are taking steps like:
> - Manually rebalancing blocks in storage directories
> - Decomissioning nodes & later readding them
> There's a tradeoff between making use of all available spindles, and filling
> disks at the sameish rate. Possible solutions include:
> - Weighting less-used disks heavier when placing new blocks on the datanode.
> In write-heavy environments this will still make use of all spindles,
> equalizing disk use over time.
> - Rebalancing blocks locally. This would help equalize disk use as disks are
> added/replaced in older cluster nodes.
> Datanodes should actively manage their local disk so operator intervention is
> not needed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
