Hi,
I'm attempting to use Jython to do some one-off map-reduce scripts. When I
try to run a JobConf which makes use of python classes defined, I get
ClassNotFoundExceptions. Even using WordCount.py that's shipped with Hadoop
gives me the problem:
~ jython WordCount.py in out
08/09/29 00:40:24 INFO jvm.JvmMetrics: Initializing JVM Metrics with
processName=JobTracker, sessionId=
08/09/29 00:40:24 WARN mapred.JobClient: No job jar file set. User classes
may not be found. See JobConf(Class) or JobConf#setJar(String).
08/09/29 00:40:24 INFO mapred.FileInputFormat: Total input paths to process
: 1
08/09/29 00:40:24 INFO mapred.JobClient: Running job: job_local_1
08/09/29 00:40:24 INFO mapred.MapTask: numReduceTasks: 1
08/09/29 00:40:24 WARN mapred.LocalJobRunner: job_local_1
*java.lang.RuntimeException: java.lang.RuntimeException:
java.lang.ClassNotFoundException: org.python.proxies.__main__$WordCountMap$0
*
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:639)
at org.apache.hadoop.mapred.JobConf.getMapperClass(JobConf.java:728)
at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:36)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:82)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:204)
at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:132)
In the original script I was using, I don't get the "No job jar file set"
warning. My suspicion is that the problem is either classpath-related (how
does Hadoop know about org.python.proxies.__main__?) or related to the fact
that we're trying to reference a python class from inside of Java (although
I think this should be ok because the python class being called inherits
from Mapper and MapReduceBase). Do I need to compile with jythonc first to
get this work? I've experimented around with that a bit, but no luck thus
far - and the need to compile would eliminate much of the time I hoped to
gain by writing one-offs in jython instead of Java.
Any help would be greatly appreciated!
Best,
John
