I guess this is exactly the problem is! Is there any way I could do "Greping archives" inside the MP program ? Or some hadoop command that could combine all small pieces files into a big one ?
Thanks, -Kun --- On Mon, 10/19/09, Ashutosh Chauhan <[email protected]> wrote: > From: Ashutosh Chauhan <[email protected]> > Subject: Re: Hadoop dfs can't allocate memory with enough hard disk space > when data gets huge > To: [email protected] > Date: Monday, October 19, 2009, 3:30 PM > You might be hitting into the problem > of "small-files". This has been > discussed multiple times on the list. Greping through > archives will help. > Also http://www.cloudera.com/blog/2009/02/02/the-small-files-problem/ > > Ashutosh > > On Sun, Oct 18, 2009 at 22:57, Kunsheng Chen <[email protected]> > wrote: > > > I and running a hadoop program to perform MapReduce > work on files inside a > > folder. > > > > My program is basically doing Map and Reduce work, > each line of any file is > > a pair of string, and the result is a string associate > with occurence inside > > all files. > > > > The program works fine until the number of files grow > to about 80,000,then > > the 'cannot allocate memory' error occur for some > reason. > > > > Each of the file contains around 50 lines, but the > total size of all files > > is no more than 1.5 GB. There are 3 datanodes > performing calculation,each of > > them have more than 10GB hd left. > > > > I am wondering if that is normal for Hadoop because > the data is too large ? > > Or it might be my programs problem ? > > > > It is really not supposed to be since Hadoop was > developed for processing > > large data sets. > > > > > > Any idea is well appreciated > > > > > > > > > > > > > > > > > > > > > __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com
