Ryan Brush created OOZIE-2277:
---------------------------------
Summary: Honor oozie.action.sharelib.for.spark in Spark jobs
Key: OOZIE-2277
URL: https://issues.apache.org/jira/browse/OOZIE-2277
Project: Oozie
Issue Type: Improvement
Reporter: Ryan Brush
Priority: Minor
Shared libraries specified by oozie.action.sharelib.for.spark are not visible
in the Spark job itself. For instance, setting oozie.action.sharelib.for.spark
to "spark,hcat" will not make the hcat jars usable in the Spark job. This is
inconsistent with other actions (such as Java and MapReduce actions).
Since the Spark action just calls SparkSubmit, it looks like we would need to
explicitly pass the jars for the specified sharelibs into the SparkSubmit
operation so they are available to the Spark operation itself.
One option: we can just pass the HDFS URLs to that command via the --jars
parameter. This is actually what I've done to work around this issue; it makes
for a long SparkSubmit command but works.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
