job fails because of "No valid local directories in property: " exception
-------------------------------------------------------------------------
Key: HADOOP-242
URL: http://issues.apache.org/jira/browse/HADOOP-242
Project: Hadoop
Type: Bug
Components: mapred
Reporter: Yoram Arnon
Assigned to: Sameer Paranjpye
when running a fairly large job, of 70+K map tasks, I get many exceptions as
shown below, and eventually the job failes when a task fails four times.
The exception doesn't really tell us enough information to debug this properly,
so the first thing to do would be to add more information (path) to the
exception.
The path indicated in the config file exists, is writable and valid, though
'path' may be anything.
the exception:
java.io.IOException: No valid local directories in property: mapred.local.dir
at org.apache.hadoop.conf.Configuration.getLocalPath(Configuration.java:293) at
org.apache.hadoop.mapred.JobConf.getLocalPath(JobConf.java:153) at
org.apache.hadoop.mapred.TaskTracker$TaskInProgress.localizeTask(TaskTracker.java:523)
at
org.apache.hadoop.mapred.TaskTracker$TaskInProgress.launchTask(TaskTracker.java:572)
at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:389) at
org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:303) at
org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:418) at
org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:920)
the code:
public Path getLocalPath(String dirsProp, String path)
throws IOException {
String[] dirs = getStrings(dirsProp);
int hashCode = path.hashCode();
FileSystem fs = FileSystem.getNamed("local", this);
for (int i = 0; i < dirs.length; i++) { // try each local dir
int index = (hashCode+i & Integer.MAX_VALUE) % dirs.length;
Path file = new Path(dirs[index], path);
Path dir = file.getParent();
if (fs.exists(dir) || fs.mkdirs(dir)) {
return file;
}
}
throw new IOException("No valid local directories in property: "+dirsProp);
}
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
