Hi Lu, folks,
I've been investigating how to run Java actions in Sahara EDP that
depend on
HBase libraries (see snippet from the original question by Lu below).
In a nutshell, we need to use Oozie sharelibs for this. I am working on
a spec now, thinking
about the best way to support this in Sahara, but here is a semi-manual
intermediate solution
that will work if you would like to run such a job from Sahara.
1) Create your own Oozie sharelib that contains the HBase jars.
This ultimately is just an HDFS dir holding the jars. On any node in
your cluster with
HBase installed, run the attached script or something like it (I like
Python better than bash :) )
It simply separates the classpath and uploads all the jars to the
specified HDFS dir.
$ parsePath.py /user/myhbaselib
2) Run your Java action from EDP, but use the oozie.libpath
configuration value when you
launch the job. For example, on the job configure tab set oozie.libpath
like this:
Name Value
oozie.libpath hdfs://namenode:8020/user/myhbaselib
(note, support for this was added in
https://review.openstack.org/#/c/154214/)
That's it! In general, you can add any jars that you want to a sharelib
and then set the
oozie.libpath for the job to access them.
Here is a good blog entry about sharelibs and extra jars in Oozie jobs:
http://blog.cloudera.com/blog/2014/05/how-to-use-the-sharelib-in-apache-oozie-cdh-5/
Best,
Trevor
--- original question
(1) EDP job in Java action
The background is that we want write integration test case for newly
added services like HBase, zookeeper just like the way the edp-examples
does( sample code under sahara/etc/edp-examples/). So I thought I can
wrote an example via edp job by Java action to test HBase Service, then
I wrote the HBaseTest.java and packaged as a jar file, and run this jar
manually with the command "java -cp `hbase classpath` HBaseTest.jar
HBaseTest", it works well in the vm(provisioned by sahara with cdh
plugin).
“/usr/lib/jvm/java-7-oracle-cloudera/bin/java -cp "HBaseTest.jar:`hbase
classpath`" HBaseTest”
So I want run this job via horizon in sahara job execution page, but
found no place to pass the `hbase classpath` parameter.(I have tried
java_opt and configuration and args, all failed). When I pass the “-cp
`hbase classpath`” to java_opts in horizon job execution page. Oozie
raise this error as below
#!/usr/bin/python
import sys
import os
import subprocess
def main():
subprocess.Popen("hadoop fs -mkdir %s" % sys.argv[1], shell=True).wait()
cp, stderr = subprocess.Popen("hbase classpath", shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()
paths = cp.split(':')
for p in paths:
if p.endswith(".jar"):
print(p)
subprocess.Popen("hadoop fs -put %s %s" % (os.path.realpath(p), sys.argv[1]), shell=True).wait()
if __name__ == "__main__":
main()
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
