Hi ,
We have upgraded our code to nutch-0.9 with hadoop-0.12.2-core.jar. After
running say 50 nutch jobs(which includes inject/generate/fetch/parse etc.)
we start getting "Too many open files" error on our cluster. We are using
Linux box with kernel 2.6.9 and the open files number is 1024 on these
machine which is default. I read several mails from nutch-user, or
hadoop-user mailing lists. And i found only way was to increase the number
of open files using ulimit. Is there any other solution for this problem at
code level. BTW the value for io.sort.factor is 8 in our hadoop-site.xml.
Is anybody having any idea in this regard? Any help will be appreciated.
Here is the stacktrace of this error :
Error initializing task_0055_m_000000_3:java.io.FileNotFoundException :
/data/nutch/clean-hadoop5/tmp/hadoop-nutch/mapred/local/taskTracker/jobcache/job_0055/job.xml
(Too many open files) at java.io.FileOutputStream.open(Native Method) at
java.io.FileOutputStream.(FileOutputStream.java:179) at
java.io.FileOutputStream.(FileOutputStream.java:131) at
org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.(
RawLocalFileSystem.java:152) at
org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:191)
at org.apache.hadoop.fs.ChecksumFileSystem$FSOutputSummer.(
ChecksumFileSystem.java:363) at
org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:438)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:346) at
org.apache.hadoop.fs.FileSystem.create(FileSystem.java:253) at
org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:84) at
org.apache.hadoop.fs.ChecksumFileSystem.copyToLocalFile(
ChecksumFileSystem.java:577) at
org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:766) at
org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:352) at
org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:863) at
org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:531) at
org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:899) at
org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:1589)
Regards,
Shailendra
-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
Nutch-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nutch-general
