On 10 November 2012 07:43, Koert Kuipers <[email protected]> wrote: > i am porting a map-reduce library from CDH3 to apache hadoop 1.0.4. the > unit tests used to run fine but are now misbehaving. i do a simple setup in > the unit test like this: > > private static MiniDFSCluster dfsCluster; > private static FileSystem fs; > private static MiniMRCluster mrCluster; > > @BeforeClass > public static void oneTimeSetUp() throws Exception { > new File("tmp/logs").mkdirs(); > System.setProperty("hadoop.log.dir", "tmp/logs"); > System.setProperty("javax.xml.parsers.SAXParserFactory", > > "com.sun.org.apache.xerces.internal.jaxp.SAXParserFactoryImpl"); > final Configuration conf = new Configuration(); > conf.set("io.skip.checksum.errors", "true"); > dfsCluster = new MiniDFSCluster(conf, 2, true, null); > fs = dfsCluster.getFileSystem(); > mrCluster = new MiniMRCluster(2, > dfsCluster.getFileSystem().getUri().toString(), 1, null, null, new > org.apache.hadoop.mapred.JobConf(conf)); > } > > however with hadoop 1.0.4 this results in an exception where the > jobtracker dies: > > 12/11/10 02:38:29 ERROR mapred.MiniMRCluster: Job tracker crashed > java.lang.NoSuchFieldError: IS_SECURITY_ENABLED > at > org.apache.jasper.compiler.JspRuntimeContext.<init>(JspRuntimeContext.java:197) > at org.apache.jasper.servlet.JspServlet.init(JspServlet.java:150) > at > org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:440) > > not sure what this error is about, but i managed to get around it for now > by removing the 2 hadoop apache jasper dependencies which seem to have > something to do with it: jasper-compiler-5.5.12.jar and > jasper-runtime-5.5.12.jar. with this "fix" i get further and my jobs > starts, but now all the tasks fail with messages like this: > > attempt_20121110024037623_0001_r_000002_0: Exception in thread "main" > java.lang.NoClassDefFoundError: org/apache/hadoop/mapred/Child > > am i doing something completely wrong, and was i just lucky that this used > to run with CDH3? any ideas? thanks! koert > > This sounds like some classpath/versioning thing -MiniMR cluster works for me; I even fixed that system property setup in trunk as I got fed up with having to do it myself.
What is your CP in your test run?
