Hi Raghu, Each DN machine has 3 partitions, e.g.:
Filesystem Size Used Avail Use% Mounted on /dev/sda1 20G 8.0G 11G 44% / /dev/sda3 1.4T 756G 508G 60% /data tmpfs 3.9G 0 3.9G 0% /dev/shm All of the paths in hadoop-site.xml point to /data, which is the partition that filled up to 100% (I deleted a bunch of files from HDFS since then). So I guess the question is whether the DN looks at just the partition its data directory is on, or all partitions when it determines disk usage. -- Stefan > From: Raghu Angadi <[EMAIL PROTECTED]> > Reply-To: <[email protected]> > Date: Wed, 29 Oct 2008 11:57:07 -0700 > To: <[email protected]> > Subject: Re: Datanode not detecting full disk > > Stefan Will wrote: >> Hi Jeff, >> >> Yeah, it looks like I'm running into the issues described in the bug. I'm >> running 0.18.1 on CentOS 5 by the way. Measuring available disk space >> appears to be harder than I thought ... and here I was under the impression >> the percentage in df was a pretty clear indicator of how full the disk is >> ;-) >> >> How did you guys solve/work around this ? > > How many partitions do you have? If it is just one and NameNode thinks > it has space though 'available' in df shows very less or no space, then > you need to file a jira. There should be no case where DN reports more > space than what 'available' field in 'df' shows. > > But if you have more partitions and only some of them are full, then it > is a different issue.. which should still be fixed. > > Raghu. > >> -- Stefan >> >> >>> From: Jeff Hammerbacher <[EMAIL PROTECTED]> >>> Reply-To: <[email protected]> >>> Date: Mon, 27 Oct 2008 12:40:08 -0700 >>> To: <[email protected]> >>> Subject: Re: Datanode not detecting full disk >>> >>> Hey Stefan, >>> >>> We used to have trouble with this issue at Facebook. What version are >>> you running? You might get more information on this ticket: >>> https://issues.apache.org/jira/browse/HADOOP-2991. >>> >>> Regards, >>> Jeff >>> >>> On Mon, Oct 27, 2008 at 10:00 AM, Stefan Will <[EMAIL PROTECTED]> wrote: >>>> Each of my datanodes has a system and a data partition, with dfs.data.dir >>>> pointed to the data partition. The data partition just filled up to 100% on >>>> all of my nodes (as evident via df), but the NameNode web ui still shows >>>> them only 88-94% full (interestingly, the numbers differ even though the >>>> machines are configured identically). I thought the datanodes used df to >>>> determine free space ? How is the storage utilization determined ? >>>> >>>> -- Stefan >>>> >> >>
