It doesn't know which jar to load your class from...
Try adding jconf.setJarByClass(MyMapper.class); to your code before the
call to runJob() and see what happens.
- Aaron
jonathan doklovic wrote:
Hi,
I've been testing a map/reduce on a local hadoop cluster on my machine.
It was working fine yesterday, and now it keeps throwing
ClassNotFoundExceptions if I run it as a java app via eclipse.
If I choose "run on hadoop" it seems to run ok, but I don't see any
messages in the console from my System.out.println statements.
When I run as a plain java app, I pass my hadoop-site.xml from my conf
dir to my configuration.
Here's my job code:
public static void main(String[] args)
{
Configuration conf = new Configuration();
if(args.length>0) conf.addResource(new Path(args[0]));
try
{
Path path = new Path("/accuracy-test/content.seq");
Path out = new Path("/accuracy-out/scores");
FileSystem fs = FileSystem.get(conf);
if(fs.exists(path))
{
fs.delete(path);
}
if(fs.exists(out))
{
fs.delete(out);
}
Item item = new Item();
item.setDate(new Date());
item.setTitle("some title");
item.setUrl("http://www.junk.com";);
item.setContent("blah blah");
MapWritable content = item.toMapWritable();
SequenceFile.Writer writer =
SequenceFile.createWriter(fs, conf,
path, IntWritable.class, MapWritable.class);
writer.append(new IntWritable(1), content);
writer.close();
JobConf jconf = new JobConf(MyRunner.class);
if(args.length > 0) jconf.addResource(new
Path(args[0]));
jconf.setInputPath(new Path("/accuracy-test/"));
jconf.setOutputPath(out);
jconf.setInputFormat(new
SequenceFileInputFormat<IntWritable,
MapWritable>().getClass());
jconf.setOutputFormat(SequenceFileOutputFormat.class);
jconf.setOutputKeyClass(Text.class);
jconf.setOutputValueClass(MapWritable.class);
jconf.setMapperClass(MyMapper.class);
jconf.setNumMapTasks(10);
JobClient.runJob(jconf);
} catch (IOException e)
{
// TODO Auto-generated catch block
e.printStackTrace();
}
}
And here's the output:
07/11/13 15:26:37 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes
where applicable
07/11/13 15:26:37 WARN mapred.JobClient: No job jar file set. User
classes may not be found. See JobConf(Class) or JobConf#setJar(String).
07/11/13 15:26:37 INFO mapred.FileInputFormat: Total input paths to
process : 1
07/11/13 15:26:37 INFO mapred.JobClient: Running job:
job_200711131515_0008
07/11/13 15:26:38 INFO mapred.JobClient: map 0% reduce 0%
07/11/13 15:26:47 INFO mapred.JobClient: Task Id :
task_200711131515_0008_m_000000_0, Status : FAILED
java.lang.RuntimeException: java.lang.RuntimeException:
java.lang.ClassNotFoundException: com.ibsys.hadoop.mapper.MyMapper
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:576)
at org.apache.hadoop.mapred.JobConf.getMapperClass(JobConf.java:745)
at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:36)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:82)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:188)
at org.apache.hadoop.mapred.TaskTracker
$Child.main(TaskTracker.java:1760)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
com.ibsys.hadoop.mapper.MyMapper
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:544)
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:568)
... 6 more
Caused by: java.lang.ClassNotFoundException:
com.ibsys.hadoop.mapper.MyMapper
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276)
at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:524)
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:542)
... 7 more
07/11/13 15:26:51 INFO mapred.JobClient: Task Id :
task_200711131515_0008_m_000000_1, Status : FAILED
java.lang.RuntimeException: java.lang.RuntimeException:
java.lang.ClassNotFoundException: com.ibsys.hadoop.mapper.MyMapper
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:576)
at org.apache.hadoop.mapred.JobConf.getMapperClass(JobConf.java:745)
at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:36)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:82)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:188)
at org.apache.hadoop.mapred.TaskTracker
$Child.main(TaskTracker.java:1760)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
com.ibsys.hadoop.mapper.MyMapper
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:544)
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:568)
... 6 more
Caused by: java.lang.ClassNotFoundException:
com.ibsys.hadoop.mapper.MyMapper
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276)
at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:524)
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:542)
... 7 more
07/11/13 15:26:54 INFO mapred.JobClient: Task Id :
task_200711131515_0008_m_000000_2, Status : FAILED
java.lang.RuntimeException: java.lang.RuntimeException:
java.lang.ClassNotFoundException: com.ibsys.hadoop.mapper.MyMapper
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:576)
at org.apache.hadoop.mapred.JobConf.getMapperClass(JobConf.java:745)
at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:36)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:82)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:188)
at org.apache.hadoop.mapred.TaskTracker
$Child.main(TaskTracker.java:1760)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
com.ibsys.hadoop.mapper.MyMapper
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:544)
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:568)
... 6 more
Caused by: java.lang.ClassNotFoundException:
com.ibsys.hadoop.mapper.MyMapper
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276)
at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:524)
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:542)
... 7 more
07/11/13 15:26:58 INFO mapred.JobClient: map 100% reduce 100%
java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:831)
at com.ibsys.hadoop.temp.StoryCreator.main(MyRunner.java:67)
