This is a bit scattered but I wanted to post this in case it might
help someone...
Here's a little more detail on the loading problems I've been having.
For now, I'm just trying to call some ruby from the reduce method of
my map/reduce job. I want to move to a more general setup, like the
one James Moore proposes above, but I'm taking baby steps due to my
general lack of knowledge regarding hadoop and jruby.
The first problem I encountered was that, from within hadoop, was
unable to load the scripting framework (JSR223) at all. I was getting
this exception (Using JRubyScriptEngineManager):
Exception in thread "main" java.lang.NullPointerException
at org.jruby.runtime.load.LoadService.findFile(LoadService.java:476)
at org.jruby.runtime.load.LoadService.findLibrary(LoadService.java:394)
at org.jruby.runtime.load.LoadService.smartLoad(LoadService.java:259)
at org.jruby.runtime.load.LoadService.require(LoadService.java:349)
at
com.sun.script.jruby.JRubyScriptEngine.init(JRubyScriptEngine.java:484)
at
com.sun.script.jruby.JRubyScriptEngine.<init>(JRubyScriptEngine.java:96)
at
com.sun.script.jruby.JRubyScriptEngineFactory.getScriptEngine(JRubyScriptEngineFactory.java:134)
at
com.sun.script.jruby.JRubyScriptEngineManager.registerEngineNames(JRubyScriptEngineManager.java:95)
at
com.sun.script.jruby.JRubyScriptEngineManager.init(JRubyScriptEngineManager.java:72)
at
com.sun.script.jruby.JRubyScriptEngineManager.<init>(JRubyScriptEngineManager.java:66)
at
com.sun.script.jruby.JRubyScriptEngineManager.<init>(JRubyScriptEngineManager.java:61)
at com.talentspring.TestMapreduce.dump(TestMapreduce.java:236)
at com.talentspring.TestMapreduce.main(TestMapreduce.java:432)
Poking around the JRubyScriptEngine source
(https://scripting.dev.java.net/source/browse/scripting/engines/jruby/src/com/sun/script/jruby/)
it looks like it uses the property "com.sun.script.jruby.loadpath" and
not "jruby.home" as suggested by
http://wiki.jruby.org/wiki/Java_Integration#Java_6_.28using_JSR_223:_Scripting.29
. hmmm.
I added -Dcom.sun.script.jruby.loadpath=$JRUBY_HOME to my invocation
and it worked... sort of. I found that by the time execution reached
the 'configure' method, the load path property was null. Odd. Does
anybody know why this might be? In any case, I saved the value in my
JobConf before submitting the job, like so:
jobConf.set("jruby.load_path",
System.getProperty("com.sun.script.jruby.loadpath"));
Then, in the configure method I have:
System.setProperty("com.sun.script.jruby.loadpath",
jobConf.get("jruby.load_path"));
I then load the script engine and everything works...
So: Does anybody have any idea of why i might be losing the system
load path property when I get to the configure method?
Cheers,
-lincoln
--
lincolnritter.com
On Fri, Jul 25, 2008 at 10:22 AM, Lincoln Ritter
<[EMAIL PROTECTED]> wrote:
> I was using BSF to avoid java 6 issues. However I'm having similar
> issues using both systems. Basically, I can't load the scripting
> engine from within hadoop. I have successfully compiled and run some
> stand-alone test examples but am having trouble getting anything to
> work from hadoop. One confounding factor is that my development
> machine is OS X 10.5 with the stock 1.5 JDK. On the surface this
> doesn't seem to be a problem given the success I've had at creating
> small stand-alone tests... I run the stand-alone stuff with exactly
> the same classpath and environment so it seems that something weird is
> going on. Additionally, as a sanity check, I've tried loading the
> javascript engine and that does work from within hadoop.
>
> All the JSR jars are on the classpath and I'm kinking off the hadoop
> process using the -Djruby.home=... option. Did you have to do
> anything special here?
>
> -lincoln
>
> --
> lincolnritter.com
>
>
>
> On Thu, Jul 24, 2008 at 7:00 PM, James Moore <[EMAIL PROTECTED]> wrote:
>> On Thu, Jul 24, 2008 at 3:51 PM, Lincoln Ritter
>> <[EMAIL PROTECTED]> wrote:
>>> Well that sounds awesome! It would be simply splendid to see what
>>> you've got if you're willing to share.
>>
>> I'll be happy to share, but it's pretty much in pieces, not ready for
>> release. I'll put it out with whatever license Hadoop itself uses
>> (presumably Apache).
>>
>>>
>>> Are you going the 'direct' embedding route or using a scripting frame
>>> work (BSF or javax.script)?
>>
>> JSR233 is the way to go according to the JRuby guys at RailsConf last
>> month. It's pretty straightforward - see
>> http://wiki.jruby.org/wiki/Java_Integration#Java_6_.28using_JSR_223:_Scripting.29
>>
>> --
>> James Moore | [EMAIL PROTECTED]
>> Ruby and Ruby on Rails consulting
>> blog.restphone.com
>>
>
