Hi Kai Thank you for the reply.
The reduce() will not start because the shuffle phase does not finish. And the shuffle phase will not finish untill alll mapper end. I am curious about the design purpose about overlapping the map and reduce stage. Was this only for saving shuffling time? Or there are some other reasons. Best wishes! Chen On Mon, Sep 19, 2011 at 12:36 AM, Kai Voigt <[email protected]> wrote: > Hi Chen, > > the times when nodes running instances of the map and reduce nodes overlap. > But map() and reduce() execution will not. > > reduce nodes will start copying data from map nodes, that's the shuffle > phase. And the map nodes are still running during that copy phase. My > observation had been that if the map phase progresses from 0 to 100%, it > matches with the reduce phase progress from 0-33%. For example, if you map > progress shows 60%, reduce might show 20%. > > But the reduce() will not start until all the map() code has processed the > entire input. So you will never see the reduce progress higher than 66% when > map progress didn't reach 100%. > > If you see map phase reaching 100%, but reduce phase not making any higher > number than 66%, it means your reduce() code is broken or slow because it > doesn't produce any output. An infinitive loop is a common mistake. > > Kai > > Am 19.09.2011 um 07:29 schrieb He Chen: > > > Hi Arun > > > > I have a question. Do you know what is the reason that hadoop allows the > map > > and the reduce stage overlap? Or anyone knows about it. Thank you in > > advance. > > > > Chen > > > > On Sun, Sep 18, 2011 at 11:17 PM, Arun C Murthy <[email protected]> > wrote: > > > >> Nan, > >> > >> The 'phase' is implicitly understood by the 'progress' (value) made by > the > >> map/reduce tasks (see o.a.h.mapred.TaskStatus.Phase). > >> > >> For e.g. > >> Reduce: > >> 0-33% -> Shuffle > >> 34-66% -> Sort (actually, just 'merge', there is no sort in the reduce > >> since all map-outputs are sorted) > >> 67-100% -> Reduce > >> > >> With 0.23 onwards the Map has phases too: > >> 0-90% -> Map > >> 91-100% -> Final Sort/merge > >> > >> Now,about starting reduces early - this is done to ensure shuffle can > >> proceed for completed maps while rest of the maps run, there-by > pipelining > >> shuffle and map completion. There is a 'reduce slowstart' feature to > control > >> this - by default, reduces aren't started until 5% of maps are complete. > >> Users can set this higher. > >> > >> Arun > >> > >> On Sep 18, 2011, at 7:24 PM, Nan Zhu wrote: > >> > >>> Hi, all > >>> > >>> recently, I was hit by a question, "how is a hadoop job divided into 2 > >>> phases?", > >>> > >>> In textbooks, we are told that the mapreduce jobs are divided into 2 > >> phases, > >>> map and reduce, and for reduce, we further divided it into 3 stages, > >>> shuffle, sort, and reduce, but in hadoop codes, I never think about > >>> this question, I didn't see any variable members in JobInProgress class > >>> to indicate this information, > >>> > >>> and according to my understanding on the source code of hadoop, the > >> reduce > >>> tasks are unnecessarily started until all mappers are finished, in > >>> constract, we can see the reduce tasks are in shuffle stage while there > >> are > >>> mappers which are still in running, > >>> So how can I indicate the phase which the job is belonging to? > >>> > >>> Thanks > >>> -- > >>> Nan Zhu > >>> School of Electronic, Information and Electrical Engineering,229 > >>> Shanghai Jiao Tong University > >>> 800,Dongchuan Road,Shanghai,China > >>> E-Mail: [email protected] > >> > >> > > -- > Kai Voigt > [email protected] > > > > >
