Hi, I have been working on a problem where I have to process a particular data and return three varieties of data and then, I have to process each of them and store each variety of data into separate files. In order to solve the above problem, I have proposed two solutions. One I called un-optimized, the other one, I called optimized.
The unoptimized flow, goes in this way: I created three map-reduce jobs. In the first map-reduce job, 1. During map phase, I processed the data and pass one variety of it to reduce phase and write the remaining two varieties in different files explicitly (explicitly means using Sequence.writer). 2. During the reduce phase of this job, I process the first variety of data passed to this phase and the processed data will be stored by reduce task. 2. Secomd map-reduce job, takes one of the variety written to files in above map phase as input and in map phase, it process this data and send it to reduce phase and the rest of the processing is done in reduce phase and output is stored in files. 3. Third map-reduce job is similar to above one but works on the other set of data. Optimized flow go in this way: 1. During map phase, the data is processed and output all three varieties of data to reduce phase 2. In the reduce phase, depending on which type of data it is, the data is processed and one of it will be outputted by reduce phase and remaining two types will be written to files explicitly. And, I assumed that by optimizing the 3 map-reduce jobs to single map-reduce job, the process should be faster and should end faster than un-optimized one. But, what I figured out that, the optimized one run faster than unoptimized one when I run on a cluster of size=1. If I run the same on a cluster of size=9, then unoptimized one took less time (1 minute difference) than optimized one. After looking at the analysis of each job, I figured out that, the map phase in PLSI is taking too much time because of outputting three values at a time for each key,value pair. Can any one please let me know if any one figured out anything suspicious in above flow which would have caused this? Thanks -- View this message in context: http://www.nabble.com/Performance-difference-over-two-map-reduce-solutions-of-same-problem-in-different-cluster-sizes-tp17227312p17227312.html Sent from the Hadoop core-user mailing list archive at Nabble.com.
