[
https://issues.apache.org/jira/browse/HDFS-329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192670#comment-13192670
]
Suresh Srinivas commented on HDFS-329:
--------------------------------------
Balancer tool can balance space utiliization. Is there any thing really needs
to be done for this bug? Closing it for now. If any one wants to pursue this,
please reopen the bug.
> separate space reservation for hdfs blocks and intermediate storage
> -------------------------------------------------------------------
>
> Key: HDFS-329
> URL: https://issues.apache.org/jira/browse/HDFS-329
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Joydeep Sen Sarma
> Priority: Critical
>
> both dfs client buffering (and i imagine map-reduce intermediate data) and
> datanode try to honor the same space reservation (dfs.du.reserved). But this
> is problematic because once hdfs/data-node fill up a node - there's no space
> left for map-reduce computations.
> ideally - hdfs should be allowed to consume upto some watermark (say 60%) and
> then dfs buffering/intermediate storage should be allowed to consume space
> upto some higher watermark (say 90%). this way the node will always remain
> usable.
> we are hitting this problem in a cluster where a few nodes have lower amount
> of space. while the cluster overall has space left, these nodes are hitting
> their space limits. but now tasks scheduled on these nodes fail because dfs
> client does not find space to buffer to. there's no workaround really i can
> think of.
> another option would be to globally allocate hdfs blocks based on space
> availability (keep all nodes at the same space utilization % approx.).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
