Hi,
 I have been working on a problem where I have to process a particular data
and return three varieties of data and then, I have to process each of them
and store each variety of data into separate files.
In order to solve the above problem, I have proposed two solutions. One I
called un-optimized, the other one, I called optimized.

The unoptimized flow, goes in this way:

I created three map-reduce jobs. In the first map-reduce job,
 1. During map phase, I processed the data and pass one variety of it to
reduce phase and write the remaining two varieties in different files
explicitly (explicitly means using Sequence.writer).
2. During the reduce phase of this job, I process the first variety of data
passed to this phase and the processed data will be stored by reduce task.

 2. Secomd map-reduce job, takes one of the variety written to files in
above map phase as input and in map phase, it 
process this data and send it to reduce phase and the rest of the processing
is done in reduce phase and output is stored in files.
3. Third map-reduce job is similar to above one but works on the other set
of data.

Optimized flow go in this way:
1. During map phase, the data is processed and output all three varieties of
data to reduce phase
2. In the reduce phase, depending on which type of data it is, the data is
processed and one of it will be outputted by reduce phase and remaining two
types will be written to files explicitly.

And, I assumed that by optimizing the 3 map-reduce jobs to single map-reduce
job, the process should be faster and should end faster than un-optimized
one.

But, what I figured out that, the optimized one run faster than unoptimized
one when I run on a cluster of size=1. If I run the same on a cluster of
size=9, then unoptimized one took less time (1 minute difference) than
optimized one.

After looking at the analysis of each job, I figured out that, the map phase
in PLSI is taking too much time because of outputting three values at a time
for each key,value pair. 

Can any one please let me know if any one figured out anything suspicious in
above flow which would have caused this?

Thanks

-- 
View this message in context: 
http://www.nabble.com/Performance-difference-over-two-map-reduce-solutions-of-same-problem-in-different-cluster-sizes-tp17227312p17227312.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.

Reply via email to