Forgot to mention that this happens in the fetch job. Thanks. On Jul 6, 2013 11:43 AM, "Amit Sela" <[email protected]> wrote:
> Hi all, > I'm trying to execute a long crawl with no depth. I am crawling over a > seed list of ~15M sites with depth 1 because I just want to index these > sites and nothing else. > I'm running with nutch 1.6 over a 3 machines cluster. > Once I get to 79% (takes a few days...) my map and reduce tasks get killed > with "Lost task tracker: ..." and after that I start getting tasks failed > with: "Child Error...IOException: Task process exit with nonzero status of > 255" and "Child Error... IOException: Creation of symlink from [path to > log].cleanup to [path ti mapred-tmp-dir log].cleanup failed. > > I read about "mapred.task.timeout" might be the cause of "Lost task > tracker" so I changed it to 10800000 (default is 600000) but it keeps > happening. 600000 is 10 minuites and 10800000 is 3 hours.. maybe increase > even more ? > > Thanks for the help, I'm really puzzled here. > > Amit. >
