[
https://issues.apache.org/jira/browse/OOZIE-2601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16295475#comment-16295475
]
Rohini Palaniswamy commented on OOZIE-2601:
-------------------------------------------
bq. to set the classpath via JavaActionExecutor prior to application submission
in a different way (now we are adding them via "yarn.application.classpath")
and the improvement of ShareLibService
I am not suggesting changing setting of yarn.application.classpath
bq. Right now, ShareLibService checks the existence of files either on HDFS or
local filesystem (symlinks followed too). It could help users (i.e. warning
them if specified dependencies are not present).
I am actually asking to skip this step. What it is doing is actually
checking the files or resolving symlinks in the Oozie server, while the actual
files are in the task nodes. Instead of letting it go through regular code
path, I would just suggest adding a new getShareLibJarsLocal(String actiontype)
method, which will return the local files path as is (if it is a file with
extension) or return both the path and /* appended to the path (/* works only
for jars. The original directory path is required if there are config files
under the directory). That can be then added directly to
yarn.application.classpath setting.
> Ability to use local paths for the sharelib
> -------------------------------------------
>
> Key: OOZIE-2601
> URL: https://issues.apache.org/jira/browse/OOZIE-2601
> Project: Oozie
> Issue Type: New Feature
> Affects Versions: oya
> Reporter: Robert Kanter
> Assignee: Attila Sasvari
> Fix For: oya, 5.0.0b1
>
> Attachments: OOZIE-2601-01.patch, OOZIE-2601-03.patch,
> OOZIE-2601-05.patch, OOZIE-2601-06.patch, OOZIE-2601-07.patch,
> OOZIE-2601-09.patch, OOZIE-2601-10.patch, OOZIE-2601-11.patch
>
>
> With OOZIE-2590, as part of OOZIE-1770 Oozie on Yarn work, Oozie now has full
> control over the classpath given to the Launcher AM. In a cluster where all
> nodes have everything installed locally (in the same paths), it should be
> possible to have the Launcher AM reference the local jars instead of having
> to localize them from HDFS.
> For example, if you have Hive installed on all nodes at {{/usr/lib/hive/}}
> and all Hive jars under {{/usr/lib/hive/lib/}}, we could have the Launcher AM
> add {{/usr/lib/hive/lib}} to its classpath. This saves on the overhead of
> localizing the same jars from the hive sharelib in HDFS.
> I think the best way to implement this is to augment the [Sharelib Mapping
> File|https://oozie.apache.org/docs/4.2.0/AG_Install.html#Oozie_Share_Lib]
> feature to accept {{file:///}} paths.
> If we had this also work with the "oozie" sharelib and the Oozie jars in the
> individual sharelibs (e.g. have the Mapping file take comma-separated
> dirs/jars), then in a cluster with everything installed on all of the nodes,
> you wouldn't need to bother with the sharelib at all!
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
