Do they only exist during the map/reduce process and will be removed after the MR finished？
When the reduce finished，I only see part-m-0000, part-m-0001 ...., which are reduce results. [email protected] From: Susheel Kumar Gadalay Date: 2014-12-16 13:05 To: user Subject: Re: Where the output of mappers are saved ? Map outputs will be in hdfs under your user name and output directory. They will have name like part-m-0000, part-m-0001 .... On 12/16/14, Abdul Navaz <[email protected]> wrote: > Hello, > > > Second Try ! > > > I have created a directory to store this mapper output as below. > <property> > <name>mapred.local.dir</name> > <value>/app/hadoop/tmp/myoutput</value> > </property> > and i looked at > hduser@dn4:/app/hadoop/tmp/myoutput$ ls -lrt > total 16 > drwxr-xr-x 2 hduser hadoop 4096 Dec 12 10:50 tt_log_tmp > drwx------ 3 hduser hadoop 4096 Dec 12 10:53 ttprivate > drwxr-xr-x 3 hduser hadoop 4096 Dec 12 10:53 taskTracker > drwxr-xr-x 4 hduser hadoop 4096 Dec 12 13:25 userlogs > and i couldnot find anything here when i run the map reduce job . Where by > default mapper output is saved and how can I get the size of mapper output > in bytes > > > Thanks. > > > From: Abdul Navaz <[email protected]> > Date: Friday, December 12, 2014 at 12:36 AM > To: "[email protected]" <[email protected]> > Subject: Where the output of mappers are saved ? > > Hello, > > > I am interested in efficiently manage the Hadoop shuffling traffic and > utilize the network bandwidth effectively. To do this I want to know how > much shuffling traffic generated by each Datanodes ? Shuffling traffic is > nothing but the output of mappers. So where this mapper output is saved ? > How can i get the size of mapper output from each datanodes in a real time > ? > Appreciate your help. > > Thanks & Regards, > > Abdul Navaz > > > >
