[
https://issues.apache.org/jira/browse/OOZIE-3472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16820843#comment-16820843
]
Junfan Zhang commented on OOZIE-3472:
-------------------------------------
The problem is not the user spark jar conflicts with Oozie but the user spark
jar is in conflict with the Hadoop. cause the Oozie mr launcher failed. The
root cause details have been clarified in the description.
Maybe i should upload the patch to illustrate. Give me some time to adjust our
production code to upload.
> Improve Spark Action compatibility with Oozie launcher
> ------------------------------------------------------
>
> Key: OOZIE-3472
> URL: https://issues.apache.org/jira/browse/OOZIE-3472
> Project: Oozie
> Issue Type: Improvement
> Components: action
> Affects Versions: 5.1.0
> Reporter: Junfan Zhang
> Assignee: Junfan Zhang
> Priority: Major
>
> In the production environment, when using the spark action, our users often
> encounter conflicts between the user jar and the launcher, causing the
> launcher to fail to start.
> To do this we have a maven plugin to guide the user to remove Hadoop related
> dependencies from the user jar. But the user jar is more complicated and
> sometimes not easy to remove. Therefore, it is appropriate to solve this
> problem from the oozie side.
> We research code found that the spark action is inherited to the Java action.
> The reason for the conflict is because the Java action will put the user jar
> into the cache before the mr starts (related
> [link|https://github.com/apache/oozie/blob/b91457edd2a76f94f41a89ec718eec574c200c71/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java#L722]).
> If there is a Hadoop dependency in the user jar and the version is
> incompatible, a conflict will occur.
> From the root cause analysis, the spark action just uses the map node in mr
> as a spark submit client, and does not need to add the user jar to the mr
> distributed cache. We solved this conflict by using spark submit sdk to load
> the user jar from HDFS directly. It currently works well in our production
> environment. :)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
