[
https://issues.apache.org/jira/browse/HDFS-1312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13463005#comment-13463005
]
Allen Wittenauer commented on HDFS-1312:
----------------------------------------
Since someone (off-jira) asked:
* Yes, I think this should be fixed.
* No, I don't think this is as big of an issue as most people think.
* At 70-80% full, you start to run the risk that the NN is going to have
trouble placing blocks, esp if . Also, if you are like most places and put the
MR spill space on the same file system as HDFS, that 70-80% is more like 100%,
especially if you don't clean up after MR. (Thus why I always put MR area on a
separate file system...)
* As you scale, you care less about the health of individual nodes and more
about total framework health.
* 1PB isn't that big. At 12 drives per node, we're looking at ~50-60 nodes.
> Re-balance disks within a Datanode
> ----------------------------------
>
> Key: HDFS-1312
> URL: https://issues.apache.org/jira/browse/HDFS-1312
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: data-node
> Reporter: Travis Crawford
>
> Filing this issue in response to ``full disk woes`` on hdfs-user.
> Datanodes fill their storage directories unevenly, leading to situations
> where certain disks are full while others are significantly less used. Users
> at many different sites have experienced this issue, and HDFS administrators
> are taking steps like:
> - Manually rebalancing blocks in storage directories
> - Decomissioning nodes & later readding them
> There's a tradeoff between making use of all available spindles, and filling
> disks at the sameish rate. Possible solutions include:
> - Weighting less-used disks heavier when placing new blocks on the datanode.
> In write-heavy environments this will still make use of all spindles,
> equalizing disk use over time.
> - Rebalancing blocks locally. This would help equalize disk use as disks are
> added/replaced in older cluster nodes.
> Datanodes should actively manage their local disk so operator intervention is
> not needed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
