ant sets classpath for me but i have same issues when i set it myself. for example i can run it like this: java -cp lib/test/*:build/classes:build/test com.tresata.hadoop.mapred.MapRedTest
my dependencies are hadoop-core, hadoop-test and i also had to add jsr311api to make it work. ls lib/test gives: ant-1.6.5.jar commons-beanutils-1.7.0.jar commons-beanutils-core-1.8.0.jar commons-cli-1.2.jar commons-codec-1.4.jar commons-collections-3.2.1.jar commons-configuration-1.6.jar commons-digester-1.8.jar commons-el-1.0.jar commons-httpclient-3.0.1.jar commons-lang-2.4.jar commons-logging-1.1.1.jar commons-math-2.1.jar commons-net-1.4.1.jar core-3.1.1.jar ftplet-api-1.0.0.jar ftpserver-core-1.0.0.jar ftpserver-deprecated-1.0.0-M2.jar guava-r06.jar hadoop-core-1.0.4.jar hadoop-test-1.0.4.jar hsqldb-1.8.0.10.jar jackson-core-asl-1.0.1.jar jackson-mapper-asl-1.0.1.jar jasper-compiler-5.5.12.jar jasper-runtime-5.5.12.jar jets3t-0.7.1.jar jetty-6.1.26.jar jetty-util-6.1.26.jar joda-time-2.1.jar jsp-2.1-6.1.14.jar jsp-api-2.1-6.1.14.jar jsr311-api-1.1.1.jar junit-3.8.1.jar kfs-0.3.jar log4j-1.2.16.jar mina-core-2.0.0-M5.jar oro-2.0.8.jar servlet-api-2.5-20081211.jar servlet-api-2.5-6.1.14.jar slf4j-api-1.6.1.jar slf4j-log4j12-1.6.1.jar xmlenc-0.52.jar On Sat, Nov 10, 2012 at 5:24 AM, Steve Loughran <[email protected]>wrote: > > > On 10 November 2012 07:43, Koert Kuipers <[email protected]> wrote: > >> i am porting a map-reduce library from CDH3 to apache hadoop 1.0.4. the >> unit tests used to run fine but are now misbehaving. i do a simple setup in >> the unit test like this: >> >> private static MiniDFSCluster dfsCluster; >> private static FileSystem fs; >> private static MiniMRCluster mrCluster; >> >> @BeforeClass >> public static void oneTimeSetUp() throws Exception { >> new File("tmp/logs").mkdirs(); >> System.setProperty("hadoop.log.dir", "tmp/logs"); >> System.setProperty("javax.xml.parsers.SAXParserFactory", >> >> "com.sun.org.apache.xerces.internal.jaxp.SAXParserFactoryImpl"); >> final Configuration conf = new Configuration(); >> conf.set("io.skip.checksum.errors", "true"); >> dfsCluster = new MiniDFSCluster(conf, 2, true, null); >> fs = dfsCluster.getFileSystem(); >> mrCluster = new MiniMRCluster(2, >> dfsCluster.getFileSystem().getUri().toString(), 1, null, null, new >> org.apache.hadoop.mapred.JobConf(conf)); >> } >> >> however with hadoop 1.0.4 this results in an exception where the >> jobtracker dies: >> >> 12/11/10 02:38:29 ERROR mapred.MiniMRCluster: Job tracker crashed >> java.lang.NoSuchFieldError: IS_SECURITY_ENABLED >> at >> org.apache.jasper.compiler.JspRuntimeContext.<init>(JspRuntimeContext.java:197) >> at org.apache.jasper.servlet.JspServlet.init(JspServlet.java:150) >> at >> org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:440) >> >> not sure what this error is about, but i managed to get around it for now >> by removing the 2 hadoop apache jasper dependencies which seem to have >> something to do with it: jasper-compiler-5.5.12.jar and >> jasper-runtime-5.5.12.jar. with this "fix" i get further and my jobs >> starts, but now all the tasks fail with messages like this: >> >> attempt_20121110024037623_0001_r_000002_0: Exception in thread "main" >> java.lang.NoClassDefFoundError: org/apache/hadoop/mapred/Child >> >> am i doing something completely wrong, and was i just lucky that this >> used to run with CDH3? any ideas? thanks! koert >> >> > This sounds like some classpath/versioning thing -MiniMR cluster works for > me; I even fixed that system property setup in trunk as I got fed up with > having to do it myself. > > What is your CP in your test run? >
