Hi Lu, folks, I've been investigating how to run Java actions in Sahara EDP that depend on HBase libraries (see snippet from the original question by Lu below).
In a nutshell, we need to use Oozie sharelibs for this. I am working on a spec now, thinking about the best way to support this in Sahara, but here is a semi-manual intermediate solution that will work if you would like to run such a job from Sahara. 1) Create your own Oozie sharelib that contains the HBase jars. This ultimately is just an HDFS dir holding the jars. On any node in your cluster with HBase installed, run the attached script or something like it (I like Python better than bash :) ) It simply separates the classpath and uploads all the jars to the specified HDFS dir. $ parsePath.py /user/myhbaselib 2) Run your Java action from EDP, but use the oozie.libpath configuration value when you launch the job. For example, on the job configure tab set oozie.libpath like this: Name Value oozie.libpath hdfs://namenode:8020/user/myhbaselib (note, support for this was added in https://review.openstack.org/#/c/154214/) That's it! In general, you can add any jars that you want to a sharelib and then set the oozie.libpath for the job to access them. Here is a good blog entry about sharelibs and extra jars in Oozie jobs: http://blog.cloudera.com/blog/2014/05/how-to-use-the-sharelib-in-apache-oozie-cdh-5/ Best, Trevor --- original question (1) EDP job in Java action The background is that we want write integration test case for newly added services like HBase, zookeeper just like the way the edp-examples does( sample code under sahara/etc/edp-examples/). So I thought I can wrote an example via edp job by Java action to test HBase Service, then I wrote the HBaseTest.java and packaged as a jar file, and run this jar manually with the command "java -cp `hbase classpath` HBaseTest.jar HBaseTest", it works well in the vm(provisioned by sahara with cdh plugin). “/usr/lib/jvm/java-7-oracle-cloudera/bin/java -cp "HBaseTest.jar:`hbase classpath`" HBaseTest” So I want run this job via horizon in sahara job execution page, but found no place to pass the `hbase classpath` parameter.(I have tried java_opt and configuration and args, all failed). When I pass the “-cp `hbase classpath`” to java_opts in horizon job execution page. Oozie raise this error as below
#!/usr/bin/python import sys import os import subprocess def main(): subprocess.Popen("hadoop fs -mkdir %s" % sys.argv[1], shell=True).wait() cp, stderr = subprocess.Popen("hbase classpath", shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate() paths = cp.split(':') for p in paths: if p.endswith(".jar"): print(p) subprocess.Popen("hadoop fs -put %s %s" % (os.path.realpath(p), sys.argv[1]), shell=True).wait() if __name__ == "__main__": main()
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev