Hi, Begin with decreasing the number of records your process per mapper and increase the number of mappers. Better parallelism and less recovery work in case of problems. We usually don't do more than 25k URL's per mapper but do a lot of mappers instead! Easier to control and debug.
Cheers. -----Original message----- > From:Amit Sela <[email protected]> > Sent: Saturday 6th July 2013 10:43 > To: [email protected] > Subject: Long crawl keeps failing in fetch phase > > Hi all, > I'm trying to execute a long crawl with no depth. I am crawling over a seed > list of ~15M sites with depth 1 because I just want to index these sites > and nothing else. > I'm running with nutch 1.6 over a 3 machines cluster. > Once I get to 79% (takes a few days...) my map and reduce tasks get killed > with "Lost task tracker: ..." and after that I start getting tasks failed > with: "Child Error...IOException: Task process exit with nonzero status of > 255" and "Child Error... IOException: Creation of symlink from [path to > log].cleanup to [path ti mapred-tmp-dir log].cleanup failed. > > I read about "mapred.task.timeout" might be the cause of "Lost task > tracker" so I changed it to 10800000 (default is 600000) but it keeps > happening. 600000 is 10 minuites and 10800000 is 3 hours.. maybe increase > even more ? > > Thanks for the help, I'm really puzzled here. > > Amit. >
