Shailendra Mudgal wrote:
Hi ,
We have upgraded our code to nutch-0.9 with hadoop-0.12.2-core.jar. After
running say 50 nutch jobs(which includes inject/generate/fetch/parse etc.)
we start getting "Too many open files" error on our cluster. We are using
Linux box with kernel 2.6.9 and the open files number is 1024 on these
machine which is default. I read several mails from nutch-user, or
hadoop-user mailing lists. And i found only way was to increase the number
of open files using ulimit. Is there any other solution for this problem at
code level. BTW the value for io.sort.factor is 8 in our hadoop-site.xml.
Is anybody having any idea in this regard? Any help will be appreciated.
Apparently datanodes that perform intensive IO operations need a higher
limit. Try increasing this number to 16k or so.
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
