On Fri, 2006-08-18 at 15:03 -0700, Benjamin Reed wrote: > Are you constructing the JobConf with a Class?
Yeah, I subclass JobConf, and instantiate that subclass in my jars main. > And is your class file in > a directory on the classpath (rather than the Jar file)? If it is, your > jar file will NOT be submitted with the job. (I opened a bug on this a > while ago.) The jar file which also contains my JobConf subclass was not in the class path. However I symlinked it in hadoop/lib/ and I was able to get a little further. > The easiest way to make sure your jar gets sent is to use the setJar() > method on JobConf. I use setJar to point the job at the absolute path of the jar. More or less I have this working now. My problems are in another area now :-) > > > Mikkel Kamstrup Erlandsen wrote: > > Im using a custom Writable in a proxy server feeding records to > > hadoop.mapred.RecordReaders. The proxy talks hadoop rpc with the record > > reader. > > > > This works fine when I run with mapred.job.tracker="local", but I get a > > ClassNotFoundException when i run distributed: > > > > 282106-2006-08-17 15:02:34,609 INFO org.apache.hadoop.ipc.Client: Client > > connection to 127.0.0.1:19093 caught: java.lang.RuntimeException: > > readObject can't find class > > 282266-java.lang.RuntimeException: readObject can't find class > > 282322- at > > org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:178) > > 282398- at > > org.apache.hadoop.io.ObjectWritable.readFields(ObjectWritable.java:60) > > 282473- at org.apache.hadoop.ipc.Client$Connection.run(Client.java:170) > > 282538:Caused by: java.lang.ClassNotFoundException: > > dk.statsbiblioteket.sbandex.mapred.WritableRecord > > 282633- at java.net.URLClassLoader$1.run(URLClassLoader.java:200) > > 282692- at java.security.AccessController.doPrivileged(Native Method) > > 282755- at java.net.URLClassLoader.findClass(URLClassLoader.java:188) > > 282818- at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > > 282876- at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:268) > > 282942- at java.lang.ClassLoader.loadClass(ClassLoader.java:251) > > 283000- at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319) > > 283066- at java.lang.Class.forName0(Native Method) > > 283110- at java.lang.Class.forName(Class.java:242) > > 283154- at > > org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:174) > > 283230- ... 2 more > > > > > > Do I have to add the module containing > > dk.statsbiblioteket.sbandex.mapred.WritableRecord to Hadoops classpath? > > I would assume that "hadoop jar myPackage.jar" would import everything > > from myPackage.jar... > > > > Cheers, > > Mikkel > > > > >
