[ http://issues.apache.org/jira/browse/HADOOP-242?page=comments#action_12412984 ]
eric baldeschwieler commented on HADOOP-242: -------------------------------------------- perhaps we should not record failure until a job has been tried on two nodes? Then we could collect stats on frequency that a job fails on a given node. Also 4 failures seems odd. Shouldn't we just have a configurable allowable % failure? That will scale evenly. > job fails because of "No valid local directories in property: " exception > ------------------------------------------------------------------------- > > Key: HADOOP-242 > URL: http://issues.apache.org/jira/browse/HADOOP-242 > Project: Hadoop > Type: Bug > Components: mapred > Reporter: Yoram Arnon > Assignee: Sameer Paranjpye > > when running a fairly large job, of 70+K map tasks, I get many exceptions as > shown below, and eventually the job failes when a task fails four times. > The exception doesn't really tell us enough information to debug this > properly, so the first thing to do would be to add more information (path) to > the exception. > The path indicated in the config file exists, is writable and valid, though > 'path' may be anything. > the exception: > java.io.IOException: No valid local directories in property: mapred.local.dir > at org.apache.hadoop.conf.Configuration.getLocalPath(Configuration.java:293) > at org.apache.hadoop.mapred.JobConf.getLocalPath(JobConf.java:153) at > org.apache.hadoop.mapred.TaskTracker$TaskInProgress.localizeTask(TaskTracker.java:523) > at > org.apache.hadoop.mapred.TaskTracker$TaskInProgress.launchTask(TaskTracker.java:572) > at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:389) > at org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:303) at > org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:418) at > org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:920) > the code: > public Path getLocalPath(String dirsProp, String path) > throws IOException { > String[] dirs = getStrings(dirsProp); > int hashCode = path.hashCode(); > FileSystem fs = FileSystem.getNamed("local", this); > for (int i = 0; i < dirs.length; i++) { // try each local dir > int index = (hashCode+i & Integer.MAX_VALUE) % dirs.length; > Path file = new Path(dirs[index], path); > Path dir = file.getParent(); > if (fs.exists(dir) || fs.mkdirs(dir)) { > return file; > } > } > throw new IOException("No valid local directories in property: > "+dirsProp); > } -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
