On 29-Sep-08, at 1:01 AM, John Thompson wrote:
Hi,
I'm attempting to use Jython to do some one-off map-reduce scripts.
When I
try to run a JobConf which makes use of python classes defined, I get
ClassNotFoundExceptions. Even using WordCount.py that's shipped
with Hadoop
gives me the problem:
[...]
In the original script I was using, I don't get the "No job jar file
set"
warning. My suspicion is that the problem is either classpath-
related (how
does Hadoop know about org.python.proxies.__main__?) or related to
the fact
that we're trying to reference a python class from inside of Java
(although
I think this should be ok because the python class being called
inherits
from Mapper and MapReduceBase). Do I need to compile with jythonc
first to
get this work? I've experimented around with that a bit, but no
luck thus
far - and the need to compile would eliminate much of the time I
hoped to
gain by writing one-offs in jython instead of Java.
In my experience, you have to compile. There may be a way around
this, but I don't know of it. Does your class work when you follow
the instructions in the wiki, which include compiling? You also need
to perform the Hadoop build step.
When you compile, make sure you have hadoop's build/classes, lib/
*.jar, and lib/jetty-ext/*.jar in your $CLASSPATH.
I recommend using streaming instead if you can, much easier to develop
and debug. It's also nice to not get that "stop doing that, jythonc
is going away" message each time you compile :) Also check out the
recently announced Happy framework for Hadoop and Jython, it looks
interesting.
I didn't mind the few seconds that compiling took, since I found
realistic testing to only be possible within a real Hadoop cluster in
any case (pseudo-distributed wouldn't always reproduce errors for
me). However, I was never able to use Jython to unit test my mappers
and reducers - I would get similar class not found exceptions, even
though they worked within the Hadoop framework. This was probably a
problem with my method signatures; there isn't much guidance on the
web for how to figure this problem out. I also found it much easier
to develop in a memory-efficient way with a modern Python and
generators.
Karl Anderson
[EMAIL PROTECTED]
http://monkey.org/~kra
