[
https://issues.apache.org/jira/browse/OOZIE-2601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16291532#comment-16291532
]
Rohini Palaniswamy edited comment on OOZIE-2601 at 12/14/17 8:13 PM:
---------------------------------------------------------------------
I think this can be simplified a lot by just adding /usr/lib/hive/lib/*
directly to yarn classpath setting. Listing all the jars in that directory from
the Oozie node and adding that to classpath seems wasteful.
was (Author: rohini):
I think this can be simplified a lot by just adding file:///usr/lib/hive/lib/*
directly to yarn classpath setting. Listing all the jars in that directory from
the Oozie node and adding that to classpath seems wasteful.
> Ability to use local paths for the sharelib
> -------------------------------------------
>
> Key: OOZIE-2601
> URL: https://issues.apache.org/jira/browse/OOZIE-2601
> Project: Oozie
> Issue Type: New Feature
> Affects Versions: oya
> Reporter: Robert Kanter
> Assignee: Attila Sasvari
> Fix For: oya, 5.0.0b1
>
> Attachments: OOZIE-2601-01.patch, OOZIE-2601-03.patch,
> OOZIE-2601-05.patch, OOZIE-2601-06.patch, OOZIE-2601-07.patch,
> OOZIE-2601-09.patch, OOZIE-2601-10.patch, OOZIE-2601-11.patch
>
>
> With OOZIE-2590, as part of OOZIE-1770 Oozie on Yarn work, Oozie now has full
> control over the classpath given to the Launcher AM. In a cluster where all
> nodes have everything installed locally (in the same paths), it should be
> possible to have the Launcher AM reference the local jars instead of having
> to localize them from HDFS.
> For example, if you have Hive installed on all nodes at {{/usr/lib/hive/}}
> and all Hive jars under {{/usr/lib/hive/lib/}}, we could have the Launcher AM
> add {{/usr/lib/hive/lib}} to its classpath. This saves on the overhead of
> localizing the same jars from the hive sharelib in HDFS.
> I think the best way to implement this is to augment the [Sharelib Mapping
> File|https://oozie.apache.org/docs/4.2.0/AG_Install.html#Oozie_Share_Lib]
> feature to accept {{file:///}} paths.
> If we had this also work with the "oozie" sharelib and the Oozie jars in the
> individual sharelibs (e.g. have the Mapping file take comma-separated
> dirs/jars), then in a cluster with everything installed on all of the nodes,
> you wouldn't need to bother with the sharelib at all!
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
