ulimit values are set to much higher values than the default values Here is the /etc/security/limits.conf contents - * - nofile 64000 hdfs - nproc 32768 hdfs - stack 10240 hbase - nproc 32768 hbase - stack 10240 mapred - nproc 32768 mapred - stack 10240
Sumanth On Thu, Feb 16, 2012 at 6:48 PM, Sree K <[email protected]> wrote: > Sumanth, > > You may want to check ulimit setting for open files. > > > Set it to a higher value if it is at default value of 1024. > > Regards, > Sreedhar > > > > > ________________________________ > From: Sumanth V <[email protected]> > To: [email protected] > Sent: Thursday, February 16, 2012 6:25 PM > Subject: ENOENT: No such file or directory > > Hi, > > We have a 20 node hadoop cluster running CDH3 U2. Some of our jobs > are failing with the following errors. We noticed that we are > consistently hitting this error condition when the total number of map > tasks in a particular job exceeds the total map task capacity of the > cluster. > Other jobs where the number of map tasks are lower than the total map task > capacity fares well. > > Here are the lines from Job Tracker log file - > > 2012-02-16 15:05:28,695 INFO org.apache.hadoop.mapred.TaskInProgress: > Error from attempt_201202161408_0004_m_000169_0: ENOENT: No such file or > directory > at org.apache.hadoop.io.nativeio.NativeIO.open(Native Method) > > at org.apache.hadoop.io.SecureIOUtils.createForWrite(SecureIOUtils.java: > 172) > > at org.apache.hadoop.mapred.TaskLog.writeToIndexFile(TaskLog.java:215) > at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:288) > at org.apache.hadoop.mapred.Child.main(Child.java:245) > > Here is the task tracker log - > > 2012-02-16 15:05:22,126 INFO org.apache.hadoop.mapred.JvmManager: JVM : > jvm_201202161408_0004_m_1467721896 exited with exit code 0. Number of tasks > it ran: 1 > 2012-02-16 15:05:22,127 WARN org.apache.hadoop.mapred.TaskLogsTruncater: > Exception in truncateLogs while getting allLogsFileDetails(). Ignoring the > truncation of logs of this process. > java.io.FileNotFoundException: > /usr/lib/hadoop-0.20/logs/userlogs/ > job_201202161408_0004/attempt_201202161408_0004_m_000112_1/log.index > (No > such file or directory) > at java.io.FileInputStream.open(Native Method) > at java.io.FileInputStream.<init>(FileInputStream.java:120) > at java.io.FileReader.<init>(FileReader.java:55) > > at org.apache.hadoop.mapred.TaskLog.getAllLogsFileDetails(TaskLog.java: > 110) > > at > org.apache.hadoop.mapred.TaskLogsTruncater.getAllLogsFileDetails(TaskLogsTr > uncater.java: 353) > > at > org.apache.hadoop.mapred.TaskLogsTruncater.shouldTruncateLogs(TaskLogsTrunc > ater.java: 98) > > at > org.apache.hadoop.mapreduce.server.tasktracker.userlogs.UserLogManager.doJv > mFinishedAction(UserLogManager.java: 163) > > at > org.apache.hadoop.mapreduce.server.tasktracker.userlogs.UserLogManager.proc > essEvent(UserLogManager.java: 137) > > at > org.apache.hadoop.mapreduce.server.tasktracker.userlogs.UserLogManager.moni > tor(UserLogManager.java: 132) > > at org.apache.hadoop.mapreduce.server.tasktracker.userlogs.UserLogManager > $1.run(UserLogManager.java:66) > 2012-02-16 15:05:22,228 INFO > org.apache.hadoop.mapred.TaskTracker: attempt_201202161408_0004_m_000006_0 > 0.0% > 2012-02-16 15:05:22,228 INFO > org.apache.hadoop.mapred.TaskTracker: attempt_201202161408_0004_m_000053_0 > 0.0% > 2012-02-16 15:05:22,329 INFO > org.apache.hadoop.mapred.TaskTracker: attempt_201202161408_0004_m_000057_0 > 0.0% > > Any help in resolving this issue would be highly appreciated! Let me > know if any other config info is needed. > > Thanks, > Sumanth >
