initTasks method in the JobInProgress class does not correctly configure the
classloader with the custom map/reduce jar
-----------------------------------------------------------------------------------------------------------------------
Key: HADOOP-509
URL: http://issues.apache.org/jira/browse/HADOOP-509
Project: Hadoop
Issue Type: Bug
Components: mapred
Affects Versions: 0.5.0
Reporter: Thomas FRIOL
In this part of code :
JobConf jd = new JobConf(localJobFile);
FileSystem fs = FileSystem.get(conf);
String ifClassName = jd.get("mapred.input.format.class");
InputFormat inputFormat;
if (ifClassName != null && localJarFile != null) {
try {
ClassLoader loader =
new URLClassLoader(new URL[]{
localFs.pathToFile(localJarFile).toURL() });
Class inputFormatClass = Class.forName(ifClassName, true, loader);
inputFormat = (InputFormat)inputFormatClass.newInstance();
} catch (Exception e) {
throw new IOException(e.toString());
}
} else {
inputFormat = jd.getInputFormat();
}
I think that the URLClassLoader should be added the URLs to all the jars
contained in the lib directory of the mapred jar. Or maybe, the jar should be
unjarred and the classpath should be updated as well as it is done into the
TaskRunner class :
String jar = conf.getJar();
if (jar != null) { // if jar exists, it into workDir
RunJar.unJar(new File(jar), workDir);
File[] libs = new File(workDir, "lib").listFiles();
if (libs != null) {
for (int i = 0; i < libs.length; i++) {
classPath.append(sep); // add libs from jar to classpath
classPath.append(libs[i]);
}
}
classPath.append(sep);
classPath.append(new File(workDir, "classes"));
classPath.append(sep);
classPath.append(workDir);
}
Actually, it is not possible to load a custom InputFormat class. Any thoughts ?
PS : This issue comes following a discussion on the issue
https://issues.apache.org/jira/browse/HADOOP-366
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
