Mathijs, Your reduce task may fail because there are too many values associated with some key and it takes more than 10 minutes to process the key. Please try to let your reduce task explicitly notify the task tracker that "I am alive" by doing report.setStatus(String) once, for example, every 100 or 1000 values.
Hairong -----Original Message----- From: Mathijs Homminga [mailto:[EMAIL PROTECTED] Sent: Tuesday, April 03, 2007 3:27 AM To: [email protected] Subject: Re: Re-reduce, without re-map Each reduce task (Nutch indexing job) gets as far as 66%, and then fails with the following error: "Task failed to report status for 600 seconds. Killing." In the end, no reduce task completes successfully. Besides solves this issue, I was wondering if I could update code and configuration and start the reduce phase again without the need to redo all map tasks (that saves me 2 hours). Assuming of course that the output of the map tasks has not changed. Mathijs Arun C Murthy wrote: > Hi Mathijs, > > Mathijs Homminga wrote: >> >> We have some troubles with the reduce phase of our job. >> Is it possible to re-execute the reduce tasks without the need to do >> all map tasks again? >> > > That the MR-framework already does... you don't have to re-execute > the maps for the *failed* reduces. Are you noticing something else? > > What are the 'troubles' you allude to? Also with once we get > HADOOP-1127 in, you should try turing on 'speculative execution' - > that helps when some tasks are very slow w.r.t other similar tasks. > > Arun > >> Thanks! >> Mathijs Homminga >
