Hi, Reduce task looks at map tasks for the partition it requires, and pulls it ( the number of parallel copies is controlled by reduce.parallel.copies ). As partitions are taken in by reduce task, it performs a merge sort, this forms your S&S phase. Typically your mappers / reducers are O(n) , S&S is O(nlogn), so if the amount of intermediate data is huge you will see a relative drop in performance.
Amogh On 10/27/09 6:35 AM, "Jeff Zhang" <[email protected]> wrote: Hi all, I'd like to know does the map task push map output to reduce task or reduce task pull it from map task ? Which way is real in hadoop ? Thank you very much. Jeff zhang
