Can you post the snippet of logs which can throw some light on the reason for failure?
-Chaitanya. On Tue, Sep 28, 2010 at 3:32 PM, Pramy Bhats <[email protected]>wrote: > Hi all, > > It turns out the problem of memory consumption leading to too many > fetch-failures. > > I have to kill the child process spawned because of previous runs $pkill > java > > restart the cluster. > > thanks > > On Mon, Sep 27, 2010 at 9:31 PM, Pramy Bhats <[email protected] > >wrote: > > > Hi Jon, > > > > I am not sure, If i get the last part correctly. The job doesn't fail > > consistently. However, DNS setup seems perfectly fine. > > > > Could you give me some pointers to debug the problem for the exact cause. > > > > thanks in advance. > > > > > > On Mon, Sep 27, 2010 at 9:01 PM, Joe Stein < > [email protected] > > > wrote: > > > >> I have seen this before if your hosts are not setup so every data node > can > >> contact every other data node by the host name registered. Typically > when > >> adding a new node ( or initial cluster setup ) making sure the internal > DNS > >> is updated or that the hosts file is updated on all data nodes as such > with > >> the list of datanodes with the name and ip. > >> > >> Since a or some data nodes can not contact each other ( maybe your > >> situation has another reason for this than the one I gave above ) then > when > >> mappers need data not local or while reducing the data fetch fails from > the > >> node that it cannot contact the datanodes it is trying to get data from. > >> > >> /* > >> Joe Stein, 973-944-0094 > >> http://www.medialets.com > >> Twitter: @allthingshadoop > >> */ > >> > >> On Sep 27, 2010, at 2:50 PM, Pramy Bhats <[email protected]> > >> wrote: > >> > >> > Hello, > >> > > >> > I am trying to run a biagram count on a 12-node cluster setup. For an > >> input > >> > file of 135 splits (around 7.5 GB), the job fails for some of the > runs. > >> > > >> > The error that I get on the jobtracker that out of 135 mappers, 1 of > the > >> > mapper fails because of > >> > > >> > > >> > "Too many fetch-failures > >> > > >> > Too many fetch-failures > >> > Too many fetch-failures > >> > Too many fetch-failures " > >> > > >> > > >> > > >> > As a result of this mapper failure, whole job fails -- the reducers > >> > which are making progress also stalls. > >> > > >> > > >> > Could anyone please help in regard of solving this error. > >> > > >> > > >> > thanks in advance. > >> > > > > >
