Great thanks for the info! Right after I finished my last question I started to think about how Hadoop measures data allocation. Are the figures presented actually the size of HDFS on each machine or the amount of disk allocated and measured by issuing something like "df".
The reason why I am asking is that df -h is quite close to the figures presented in the GUI but it could be a coincidence. //Marcus On Tue, Apr 7, 2009 at 4:02 PM, Koji Noguchi <[email protected]> wrote: > Marcus, > > One known issue in 0.18.3 is HADOOP-5465. > > Copy&Paste from > https://issues.apache.org/jira/browse/HADOOP-4489?focusedCommentId=12693 > 956&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpa > nel#action_12693956<https://issues.apache.org/jira/browse/HADOOP-4489?focusedCommentId=12693%0A956&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpa%0Anel#action_12693956> > > Hairong said: > " This bug might be caused by HADOOP-5465. Once a datanode hits > HADOOP-5465, NameNode sends an empty replication request to the data > node on every reply to a heartbeat, thus not a single scheduled block > deletion request can be sent to the data node." > > (Also, if you're always writing from one of the nodes, that node is more > likely to get full.) > > > > Nigel, not sure if this is the issue, but it would be nice to have > 0.18.4 out. > > > Koji > > > > -----Original Message----- > From: Marcus Herou [mailto:[email protected]] > Sent: Tuesday, April 07, 2009 12:45 AM > To: [email protected] > Subject: Very assymetric data allocation > > Hi. > > We are running Hadoop 0.18.3 and noticed a strange issue when one of our > machines went out of disk yesterday. > If you can see the table below it would display that the server > "mapredcoord" is 66.91% allocated and the others are almost empty. > How can that be ? > > Any information about this would be very helpful. > > mapredcoord is as well our jobtracker. > > //Marcus > > Node Last Contact Admin State Size (GB) Used (%) Used (%) Remaining (GB) > Blocks > mapredcoord<http://mapredcoord:50076/browseDirectory.jsp?namenodeInfoPor > t=50070&dir=%2F<http://mapredcoord:50076/browseDirectory.jsp?namenodeInfoPor%0At=50070&dir=%2F> > >2In > Service416.6966.91 > > 90.9419806 > mapreduce2<http://mapreduce2:50076/browseDirectory.jsp?namenodeInfoPort= > 50070&dir=%2F<http://mapreduce2:50076/browseDirectory.jsp?namenodeInfoPort=%0A50070&dir=%2F> > >2In > Service416.696.71 > > 303.54456 > mapreduce3<http://mapreduce3:50076/browseDirectory.jsp?namenodeInfoPort= > 50070&dir=%2F<http://mapreduce3:50076/browseDirectory.jsp?namenodeInfoPort=%0A50070&dir=%2F> > >2In > Service416.690.44 > 351.693975 > mapreduce4<http://mapreduce4:50076/browseDirectory.jsp?namenodeInfoPort= > 50070&dir=%2F<http://mapreduce4:50076/browseDirectory.jsp?namenodeInfoPort=%0A50070&dir=%2F> > >0In > Service416.690.25 > 355.821549 > mapreduce5<http://mapreduce5:50076/browseDirectory.jsp?namenodeInfoPort= > 50070&dir=%2F<http://mapreduce5:50076/browseDirectory.jsp?namenodeInfoPort=%0A50070&dir=%2F> > >2In > Service416.690.42 > 347.683995 > mapreduce6<http://mapreduce6:50076/browseDirectory.jsp?namenodeInfoPort= > 50070&dir=%2F<http://mapreduce6:50076/browseDirectory.jsp?namenodeInfoPort=%0A50070&dir=%2F> > >0In > Service416.690.43 > 352.73982 > mapreduce7<http://mapreduce7:50076/browseDirectory.jsp?namenodeInfoPort= > 50070&dir=%2F<http://mapreduce7:50076/browseDirectory.jsp?namenodeInfoPort=%0A50070&dir=%2F> > >0In > Service416.690.5 > 351.914079 > mapreduce8<http://mapreduce8:50076/browseDirectory.jsp?namenodeInfoPort= > 50070&dir=%2F<http://mapreduce8:50076/browseDirectory.jsp?namenodeInfoPort=%0A50070&dir=%2F> > >1In > Service416.690.48 > 350.154169 > > > -- > Marcus Herou CTO and co-founder Tailsweep AB > +46702561312 > [email protected] > http://www.tailsweep.com/ > http://blogg.tailsweep.com/ > -- Marcus Herou CTO and co-founder Tailsweep AB +46702561312 [email protected] http://www.tailsweep.com/ http://blogg.tailsweep.com/
