Hi all, I'm trying to execute a long crawl with no depth. I am crawling over a seed list of ~15M sites with depth 1 because I just want to index these sites and nothing else. I'm running with nutch 1.6 over a 3 machines cluster. Once I get to 79% (takes a few days...) my map and reduce tasks get killed with "Lost task tracker: ..." and after that I start getting tasks failed with: "Child Error...IOException: Task process exit with nonzero status of 255" and "Child Error... IOException: Creation of symlink from [path to log].cleanup to [path ti mapred-tmp-dir log].cleanup failed.
I read about "mapred.task.timeout" might be the cause of "Lost task tracker" so I changed it to 10800000 (default is 600000) but it keeps happening. 600000 is 10 minuites and 10800000 is 3 hours.. maybe increase even more ? Thanks for the help, I'm really puzzled here. Amit.
