We ran into issue as well on our cluster. +1 for JIRA for that Alexander, could you please create a JIRA in https://issues.apache.org/jira/browse/HDFS for that (it is your observation, so that you should get credit ;). Otherwise, I can do that.
2013/2/12 Alexander Fahlke <alexander.fahlke.mailingli...@googlemail.com> > Hi! > > I'm using hadoop-0.20.2 on Debian Squeeze and ran into the same confusion > as many others with the parameter for dfs.datanode.du.reserved. > One day some data nodes got out of disk errors although there was space > left on the disks. > > The following values are rounded to make the problem more clear: > > - the disk for the DFS data has 1000GB and only one Partition (ext3) for > DFS data > - you plan to set the dfs.datanode.du.reserved to 20GB > - the reserved reserved-blocks-percentage by tune2fs is 5% (the default) > > That gives all users, except root, 5% less capacity that they can use. > Although the System reports the total of 1000GB as usable for all users via > df. > The hadoop-deamons are not running as root. > > > If i read it right, than hadoop get's the free capacity via df. > > Starting in /src/hdfs/org/apache/hadoop/hdfs/server/datanode/FSDataset.java > on line 350: > return usage.getCapacity()-reserved; > > going to /src/core/org/apache/hadoop/fs/DF.java which says: > "Filesystem disk space usage statistics. Uses the unix 'df' program" > > > When you have 5% reserved by tune2fs (in our case 50GB) and you give > dfs.datanode.du.reserved only 20GB, than you can possibly ran into out of > disk errors that hadoop can't handle. > > In this case you must add the planned 20GB du reserved to the reserved > capacity by tune2fs. This results in (at least) 70GB > for dfs.datanode.du.reserved in my case. > > > Two ideas: > > 1. The documentation must be clear at this point to avoid this problem. > 2. Hadoop could check for reserved space by tune2fs (or other tools) and > add this value to the dfs.datanode.du.reserved parameter. > > > -- > BR > > Alexander Fahlke > Software Development > www.fahlke.org >
