Hi Sandy, As i understood map task involves these phases.1) Map processing 2) spill buffer contents to disk 3) partitioning 4) sorting 5) merging spill files into single file MM maybe i am wrong but i thought outputs are grouped in partitioning phase and after that it will be sorted in sort phase before sending to reducer. Is that what happens in mapper phase ?
Regarding to your question ,actually I think sort phase is one of the time consuming phase in mapper , what i am trying to do is to know how much percentage of mapper time is spent on sort phase and investigate if it is possible to skip sort in some cases.For example if we have only one reducer is it possible to skip the sorting and just flush the data directly to the reducer ? Samaneh On Thu, Apr 18, 2013 at 8:46 PM, Sandy Ryza <sandy.r...@cloudera.com> wrote: > Hi Samaneh, > > If you want to see the map outputs post sort/shuffle, the easiest way is > probably to use an IdentityReducer and inspect the job. > > Can you be more specific on what you need to disable the sort phase for? > Sorting is used in part to group map outputs and route them to the correct > reducer. > > -Sandy > > > On Thu, Apr 18, 2013 at 1:53 AM, Samaneh Shokuhi > <samaneh.shok...@gmail.com>wrote: > > > Hello All, > > I am doing some experiments with WordCount example running on hadoop > > cluster. I have some questions : > > > > 1) How can i monitor the output from mapper before flushing to reducer? ( > > Infact i want to see how the keys are sorted.) > > > > 2) In one of my experiments i need to disable the sort phase in Mapper > and > > send unsorted data to reducer. Is there any way to disable this sort in > > mapper ? or i need to modify hadoop to disable it ? > > As i undestood in MapTask.java this functionality implemented. > > And ofcourse i dont want to set number of reducer to zero becouse i need > to > > have atleast one reducer. > > > > So any idea how to disable the sort phase in mapper and monitor the > output > > ? > > > > Best, > > Samaneh > > >
