Your diagnose sounds reasonable. Since the mappers of your optimized solution outputs 3 key/value pairs for each input key/value pair, the map output size may be three times of the input size for each mapper. That size map exceeds the value of io.sort.mb in your configuration. If so, the mappers have to spill the map output onto disk in multiple segments and merge them at the end. That is very costly.
Runping > -----Original Message----- > From: novice user [mailto:[EMAIL PROTECTED] > Sent: Wednesday, May 14, 2008 2:45 AM > To: [email protected] > Subject: Performance difference over two map-reduce solutions of same > problem in different cluster sizes > > > Hi, > I have been working on a problem where I have to process a particular > data > and return three varieties of data and then, I have to process each of > them > and store each variety of data into separate files. > In order to solve the above problem, I have proposed two solutions. One I > called un-optimized, the other one, I called optimized. > > The unoptimized flow, goes in this way: > > I created three map-reduce jobs. In the first map-reduce job, > 1. During map phase, I processed the data and pass one variety of it to > reduce phase and write the remaining two varieties in different files > explicitly (explicitly means using Sequence.writer). > 2. During the reduce phase of this job, I process the first variety of > data > passed to this phase and the processed data will be stored by reduce task. > > 2. Secomd map-reduce job, takes one of the variety written to files in > above map phase as input and in map phase, it > process this data and send it to reduce phase and the rest of the > processing > is done in reduce phase and output is stored in files. > 3. Third map-reduce job is similar to above one but works on the other set > of data. > > Optimized flow go in this way: > 1. During map phase, the data is processed and output all three varieties > of > data to reduce phase > 2. In the reduce phase, depending on which type of data it is, the data is > processed and one of it will be outputted by reduce phase and remaining > two > types will be written to files explicitly. > > And, I assumed that by optimizing the 3 map-reduce jobs to single map- > reduce > job, the process should be faster and should end faster than un-optimized > one. > > But, what I figured out that, the optimized one run faster than > unoptimized > one when I run on a cluster of size=1. If I run the same on a cluster of > size=9, then unoptimized one took less time (1 minute difference) than > optimized one. > > After looking at the analysis of each job, I figured out that, the map > phase > in PLSI is taking too much time because of outputting three values at a > time > for each key,value pair. > > Can any one please let me know if any one figured out anything suspicious > in > above flow which would have caused this? > > Thanks > > -- > View this message in context: http://www.nabble.com/Performance- > difference-over-two-map-reduce-solutions-of-same-problem-in-different- > cluster-sizes-tp17227312p17227312.html > Sent from the Hadoop core-user mailing list archive at Nabble.com.
