Well, I passed the main class to the JobConf which should take care of
it. JobConf jconf = new JobConf(MyRunner.class);
However, I tried your suggestion anyway:
JobConf jconf = new JobConf(MyRunner.class);
jconf.setJarByClass(MyMapper.class);
and I get the same results, with the warning about the missing jar and
everything.
- Jonathan
On Tue, 2007-11-13 at 13:56 -0800, Aaron Kimball wrote:
> It doesn't know which jar to load your class from...
>
> Try adding jconf.setJarByClass(MyMapper.class); to your code before the
> call to runJob() and see what happens.
>
> - Aaron
>
>
> jonathan doklovic wrote:
> > Hi,
> >
> > I've been testing a map/reduce on a local hadoop cluster on my machine.
> > It was working fine yesterday, and now it keeps throwing
> > ClassNotFoundExceptions if I run it as a java app via eclipse.
> >
> > If I choose "run on hadoop" it seems to run ok, but I don't see any
> > messages in the console from my System.out.println statements.
> >
> > When I run as a plain java app, I pass my hadoop-site.xml from my conf
> > dir to my configuration.
> >
> > Here's my job code:
> >
> > public static void main(String[] args)
> > {
> > Configuration conf = new Configuration();
> > if(args.length>0) conf.addResource(new Path(args[0]));
> > try
> > {
> > Path path = new Path("/accuracy-test/content.seq");
> > Path out = new Path("/accuracy-out/scores");
> > FileSystem fs = FileSystem.get(conf);
> >
> > if(fs.exists(path))
> > {
> > fs.delete(path);
> > }
> >
> > if(fs.exists(out))
> > {
> > fs.delete(out);
> > }
> > Item item = new Item();
> > item.setDate(new Date());
> > item.setTitle("some title");
> > item.setUrl("http://www.junk.com";);
> > item.setContent("blah blah");
> > MapWritable content = item.toMapWritable();
> >
> > SequenceFile.Writer writer =
> > SequenceFile.createWriter(fs, conf,
> > path, IntWritable.class, MapWritable.class);
> >
> > writer.append(new IntWritable(1), content);
> > writer.close();
> >
> >
> > JobConf jconf = new JobConf(MyRunner.class);
> > if(args.length > 0) jconf.addResource(new
> > Path(args[0]));
> > jconf.setInputPath(new Path("/accuracy-test/"));
> > jconf.setOutputPath(out);
> > jconf.setInputFormat(new
> > SequenceFileInputFormat<IntWritable,
> > MapWritable>().getClass());
> > jconf.setOutputFormat(SequenceFileOutputFormat.class);
> > jconf.setOutputKeyClass(Text.class);
> > jconf.setOutputValueClass(MapWritable.class);
> >
> > jconf.setMapperClass(MyMapper.class);
> > jconf.setNumMapTasks(10);
> >
> > JobClient.runJob(jconf);
> >
> >
> > } catch (IOException e)
> > {
> > // TODO Auto-generated catch block
> > e.printStackTrace();
> > }
> > }
> >
> > And here's the output:
> >
> > 07/11/13 15:26:37 WARN util.NativeCodeLoader: Unable to load
> > native-hadoop library for your platform... using builtin-java classes
> > where applicable
> > 07/11/13 15:26:37 WARN mapred.JobClient: No job jar file set. User
> > classes may not be found. See JobConf(Class) or JobConf#setJar(String).
> > 07/11/13 15:26:37 INFO mapred.FileInputFormat: Total input paths to
> > process : 1
> > 07/11/13 15:26:37 INFO mapred.JobClient: Running job:
> > job_200711131515_0008
> > 07/11/13 15:26:38 INFO mapred.JobClient: map 0% reduce 0%
> > 07/11/13 15:26:47 INFO mapred.JobClient: Task Id :
> > task_200711131515_0008_m_000000_0, Status : FAILED
> > java.lang.RuntimeException: java.lang.RuntimeException:
> > java.lang.ClassNotFoundException: com.ibsys.hadoop.mapper.MyMapper
> > at
> > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:576)
> > at org.apache.hadoop.mapred.JobConf.getMapperClass(JobConf.java:745)
> > at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:36)
> > at
> > org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
> > at
> > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:82)
> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:188)
> > at org.apache.hadoop.mapred.TaskTracker
> > $Child.main(TaskTracker.java:1760)
> > Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
> > com.ibsys.hadoop.mapper.MyMapper
> > at
> > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:544)
> > at
> > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:568)
> > ... 6 more
> > Caused by: java.lang.ClassNotFoundException:
> > com.ibsys.hadoop.mapper.MyMapper
> > at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
> > at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
> > at java.lang.Class.forName0(Native Method)
> > at java.lang.Class.forName(Class.java:247)
> > at
> > org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:524)
> > at
> > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:542)
> > ... 7 more
> >
> > 07/11/13 15:26:51 INFO mapred.JobClient: Task Id :
> > task_200711131515_0008_m_000000_1, Status : FAILED
> > java.lang.RuntimeException: java.lang.RuntimeException:
> > java.lang.ClassNotFoundException: com.ibsys.hadoop.mapper.MyMapper
> > at
> > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:576)
> > at org.apache.hadoop.mapred.JobConf.getMapperClass(JobConf.java:745)
> > at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:36)
> > at
> > org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
> > at
> > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:82)
> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:188)
> > at org.apache.hadoop.mapred.TaskTracker
> > $Child.main(TaskTracker.java:1760)
> > Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
> > com.ibsys.hadoop.mapper.MyMapper
> > at
> > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:544)
> > at
> > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:568)
> > ... 6 more
> > Caused by: java.lang.ClassNotFoundException:
> > com.ibsys.hadoop.mapper.MyMapper
> > at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
> > at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
> > at java.lang.Class.forName0(Native Method)
> > at java.lang.Class.forName(Class.java:247)
> > at
> > org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:524)
> > at
> > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:542)
> > ... 7 more
> >
> > 07/11/13 15:26:54 INFO mapred.JobClient: Task Id :
> > task_200711131515_0008_m_000000_2, Status : FAILED
> > java.lang.RuntimeException: java.lang.RuntimeException:
> > java.lang.ClassNotFoundException: com.ibsys.hadoop.mapper.MyMapper
> > at
> > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:576)
> > at org.apache.hadoop.mapred.JobConf.getMapperClass(JobConf.java:745)
> > at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:36)
> > at
> > org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
> > at
> > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:82)
> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:188)
> > at org.apache.hadoop.mapred.TaskTracker
> > $Child.main(TaskTracker.java:1760)
> > Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
> > com.ibsys.hadoop.mapper.MyMapper
> > at
> > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:544)
> > at
> > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:568)
> > ... 6 more
> > Caused by: java.lang.ClassNotFoundException:
> > com.ibsys.hadoop.mapper.MyMapper
> > at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
> > at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
> > at java.lang.Class.forName0(Native Method)
> > at java.lang.Class.forName(Class.java:247)
> > at
> > org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:524)
> > at
> > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:542)
> > ... 7 more
> >
> > 07/11/13 15:26:58 INFO mapred.JobClient: map 100% reduce 100%
> > java.io.IOException: Job failed!
> > at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:831)
> > at com.ibsys.hadoop.temp.StoryCreator.main(MyRunner.java:67)
> >
