Re: Reduce task hang[EMERGENCE]

2014-01-03 Thread Azuryy Yu
Hi Harsh, There is only three warnnings in stderr *stderr logs* log4j:WARN No appenders could be found for logger (org.apache.hadoop.mapred.Child). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. On

Re: Reduce task hang[EMERGENCE]

2014-01-03 Thread Azuryy Yu
actuall these are all logs in the stderr, and stdout is empty On Fri, Jan 3, 2014 at 4:12 PM, Azuryy Yu azury...@gmail.com wrote: Hi Harsh, There is only three warnnings in stderr *stderr logs* log4j:WARN No appenders could be found for logger (org.apache.hadoop.mapred.Child).

Reduce task hang[EMERGENCE]

2014-01-02 Thread Azuryy Yu
Hi, Our prod cluster met some issues recently, All map tasks finished successfully, but reduce task hanged. but It's not happened on all TaskTrackers, only sometimes. we used mapred-1.0.4 There is 0.0% reduce copy forever until kill task manually. reduce logs on the TaskTracker:

Re: Reduce task hang[EMERGENCE]

2014-01-02 Thread Azuryy Yu
Add addtional: Our MR version is 1.2.1, not 1.0.4 There is no useful information in the JT log. On Fri, Jan 3, 2014 at 12:20 PM, Azuryy Yu azury...@gmail.com wrote: Hi, Our prod cluster met some issues recently, All map tasks finished successfully, but reduce task hanged. but It's not

Re: Reduce task hang[EMERGENCE]

2014-01-02 Thread Harsh J
Does the Reduce task log (of attempt_201312201200_34795_r_00_0) show any errors in trying to communicate with the various TaskTrackers in trying to obtain the data? On Fri, Jan 3, 2014 at 9:54 AM, Azuryy Yu azury...@gmail.com wrote: Add addtional: Our MR version is 1.2.1, not 1.0.4 There

Re: Reduce task hang[EMERGENCE]

2014-01-02 Thread Azuryy Yu
Hi Harsh, Thanks. There is no any error logs for attempt_201312201200_34795_r_00_0 in the tasktracker log. only '0.0% reduce copy ' I configured all hosts in all slaves and master. This job has only one reduce. it hanged. but I configured everybody's max job running to '1' in the Fair

Re: Reduce task hang[EMERGENCE]

2014-01-02 Thread Azuryy Yu
In detail: 'and these people's job never hanged...' these people's map and reduce tasks never hanged. On Fri, Jan 3, 2014 at 1:46 PM, Azuryy Yu azury...@gmail.com wrote: Hi Harsh, Thanks. There is no any error logs for attempt_201312201200_34795_r_00_0 in the tasktracker log. only