separate space reservation for hdfs blocks and intermediate storage
-------------------------------------------------------------------
Key: HADOOP-2677
URL: https://issues.apache.org/jira/browse/HADOOP-2677
Project: Hadoop
Issue Type: Improvement
Components: dfs
Affects Versions: 0.14.4
Reporter: Joydeep Sen Sarma
Priority: Critical
both dfs client buffering (and i imagine map-reduce intermediate data) and
datanode try to honor the same space reservation (dfs.du.reserved). But this is
problematic because once hdfs/data-node fill up a node - there's no space left
for map-reduce computations.
ideally - hdfs should be allowed to consume upto some watermark (say 60%) and
then dfs buffering/intermediate storage should be allowed to consume space upto
some higher watermark (say 90%). this way the node will always remain usable.
we are hitting this problem in a cluster where a few nodes have lower amount of
space. while the cluster overall has space left, these nodes are hitting their
space limits. but now tasks scheduled on these nodes fail because dfs client
does not find space to buffer to. there's no workaround really i can think of.
another option would be to globally allocate hdfs blocks based on space
availability (keep all nodes at the same space utilization % approx.).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
