i am porting a map-reduce library from CDH3 to apache hadoop 1.0.4. the
unit tests used to run fine but are now misbehaving. i do a simple setup in
the unit test like this:
private static MiniDFSCluster dfsCluster;
private static FileSystem fs;
private static MiniMRCluster mrCluster;
@BeforeClass
public static void oneTimeSetUp() throws Exception {
new File("tmp/logs").mkdirs();
System.setProperty("hadoop.log.dir", "tmp/logs");
System.setProperty("javax.xml.parsers.SAXParserFactory",
"com.sun.org.apache.xerces.internal.jaxp.SAXParserFactoryImpl");
final Configuration conf = new Configuration();
conf.set("io.skip.checksum.errors", "true");
dfsCluster = new MiniDFSCluster(conf, 2, true, null);
fs = dfsCluster.getFileSystem();
mrCluster = new MiniMRCluster(2,
dfsCluster.getFileSystem().getUri().toString(), 1, null, null, new
org.apache.hadoop.mapred.JobConf(conf));
}
however with hadoop 1.0.4 this results in an exception where the jobtracker
dies:
12/11/10 02:38:29 ERROR mapred.MiniMRCluster: Job tracker crashed
java.lang.NoSuchFieldError: IS_SECURITY_ENABLED
at
org.apache.jasper.compiler.JspRuntimeContext.<init>(JspRuntimeContext.java:197)
at org.apache.jasper.servlet.JspServlet.init(JspServlet.java:150)
at
org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:440)
not sure what this error is about, but i managed to get around it for now
by removing the 2 hadoop apache jasper dependencies which seem to have
something to do with it: jasper-compiler-5.5.12.jar and
jasper-runtime-5.5.12.jar. with this "fix" i get further and my jobs
starts, but now all the tasks fail with messages like this:
attempt_20121110024037623_0001_r_000002_0: Exception in thread "main"
java.lang.NoClassDefFoundError: org/apache/hadoop/mapred/Child
am i doing something completely wrong, and was i just lucky that this used
to run with CDH3? any ideas? thanks! koert
