[ http://issues.apache.org/jira/browse/HADOOP-242?page=comments#action_12412986 ]
Owen O'Malley commented on HADOOP-242: -------------------------------------- It is 4 failures of the _same_ fragment of work. So it takes map-0000000 failing 4 times before the job is killed. The job tracker does know which machines that each task has failed on and prefers to run other tasks on that node. > job fails because of "No valid local directories in property: " exception > ------------------------------------------------------------------------- > > Key: HADOOP-242 > URL: http://issues.apache.org/jira/browse/HADOOP-242 > Project: Hadoop > Type: Bug > Components: mapred > Reporter: Yoram Arnon > Assignee: Sameer Paranjpye > > when running a fairly large job, of 70+K map tasks, I get many exceptions as > shown below, and eventually the job failes when a task fails four times. > The exception doesn't really tell us enough information to debug this > properly, so the first thing to do would be to add more information (path) to > the exception. > The path indicated in the config file exists, is writable and valid, though > 'path' may be anything. > the exception: > java.io.IOException: No valid local directories in property: mapred.local.dir > at org.apache.hadoop.conf.Configuration.getLocalPath(Configuration.java:293) > at org.apache.hadoop.mapred.JobConf.getLocalPath(JobConf.java:153) at > org.apache.hadoop.mapred.TaskTracker$TaskInProgress.localizeTask(TaskTracker.java:523) > at > org.apache.hadoop.mapred.TaskTracker$TaskInProgress.launchTask(TaskTracker.java:572) > at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:389) > at org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:303) at > org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:418) at > org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:920) > the code: > public Path getLocalPath(String dirsProp, String path) > throws IOException { > String[] dirs = getStrings(dirsProp); > int hashCode = path.hashCode(); > FileSystem fs = FileSystem.getNamed("local", this); > for (int i = 0; i < dirs.length; i++) { // try each local dir > int index = (hashCode+i & Integer.MAX_VALUE) % dirs.length; > Path file = new Path(dirs[index], path); > Path dir = file.getParent(); > if (fs.exists(dir) || fs.mkdirs(dir)) { > return file; > } > } > throw new IOException("No valid local directories in property: > "+dirsProp); > } -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
