Sumanth, You may want to check ulimit setting for open files.
Set it to a higher value if it is at default value of 1024. Regards, Sreedhar ________________________________ From: Sumanth V <[email protected]> To: [email protected] Sent: Thursday, February 16, 2012 6:25 PM Subject: ENOENT: No such file or directory Hi, We have a 20 node hadoop cluster running CDH3 U2. Some of our jobs are failing with the following errors. We noticed that we are consistently hitting this error condition when the total number of map tasks in a particular job exceeds the total map task capacity of the cluster. Other jobs where the number of map tasks are lower than the total map task capacity fares well. Here are the lines from Job Tracker log file - 2012-02-16 15:05:28,695 INFO org.apache.hadoop.mapred.TaskInProgress: Error from attempt_201202161408_0004_m_000169_0: ENOENT: No such file or directory at org.apache.hadoop.io.nativeio.NativeIO.open(Native Method) at org.apache.hadoop.io.SecureIOUtils.createForWrite(SecureIOUtils.java: 172) at org.apache.hadoop.mapred.TaskLog.writeToIndexFile(TaskLog.java:215) at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:288) at org.apache.hadoop.mapred.Child.main(Child.java:245) Here is the task tracker log - 2012-02-16 15:05:22,126 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201202161408_0004_m_1467721896 exited with exit code 0. Number of tasks it ran: 1 2012-02-16 15:05:22,127 WARN org.apache.hadoop.mapred.TaskLogsTruncater: Exception in truncateLogs while getting allLogsFileDetails(). Ignoring the truncation of logs of this process. java.io.FileNotFoundException: /usr/lib/hadoop-0.20/logs/userlogs/ job_201202161408_0004/attempt_201202161408_0004_m_000112_1/log.index (No such file or directory) at java.io.FileInputStream.open(Native Method) at java.io.FileInputStream.<init>(FileInputStream.java:120) at java.io.FileReader.<init>(FileReader.java:55) at org.apache.hadoop.mapred.TaskLog.getAllLogsFileDetails(TaskLog.java: 110) at org.apache.hadoop.mapred.TaskLogsTruncater.getAllLogsFileDetails(TaskLogsTr uncater.java: 353) at org.apache.hadoop.mapred.TaskLogsTruncater.shouldTruncateLogs(TaskLogsTrunc ater.java: 98) at org.apache.hadoop.mapreduce.server.tasktracker.userlogs.UserLogManager.doJv mFinishedAction(UserLogManager.java: 163) at org.apache.hadoop.mapreduce.server.tasktracker.userlogs.UserLogManager.proc essEvent(UserLogManager.java: 137) at org.apache.hadoop.mapreduce.server.tasktracker.userlogs.UserLogManager.moni tor(UserLogManager.java: 132) at org.apache.hadoop.mapreduce.server.tasktracker.userlogs.UserLogManager $1.run(UserLogManager.java:66) 2012-02-16 15:05:22,228 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201202161408_0004_m_000006_0 0.0% 2012-02-16 15:05:22,228 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201202161408_0004_m_000053_0 0.0% 2012-02-16 15:05:22,329 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201202161408_0004_m_000057_0 0.0% Any help in resolving this issue would be highly appreciated! Let me know if any other config info is needed. Thanks, Sumanth
