[
https://issues.apache.org/jira/browse/OOZIE-2601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Sasvari updated OOZIE-2601:
----------------------------------
Attachment: OOZIE-2601-01.patch
Tests
- new test in TestShareLibService
- executed the example pig job using local filesystem resources on a pseudo
hadoop cluster
{code}
$ cat mapping_file.prop
oozie.oozie=file:///Users/asasvari/workspace/apache/oozie_dup/distro/target/oozie-5.0.0-SNAPSHOT-distro/oozie-5.0.0-SNAPSHOT/share/lib/oozie
oozie.pig=file:///Users/asasvari/workspace/apache/oozie_dup/distro/target/oozie-5.0.0-SNAPSHOT-distro/oozie-5.0.0-SNAPSHOT/share/lib/pig
{code}
and adjusted {{oozie-site.xml}} so that it ships the launcher jar
{code}
<property>
<name>oozie.action.ship.launcher.jar</name>
<value>true</value>
</property>
{code}
Oozie sharelib:
{code}
bin/oozie admin -shareliblist oozie
log4j:WARN No appenders could be found for logger
(org.apache.hadoop.security.authentication.client.KerberosAuthenticator).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more
info.
[Available ShareLib]
oozie
file:/Users/asasvari/workspace/apache/oozie_dup/distro/target/oozie-5.0.0-SNAPSHOT-distro/oozie-5.0.0-SNAPSHOT/share/lib/oozie/json-simple-1.1.jar
file:/Users/asasvari/workspace/apache/oozie_dup/distro/target/oozie-5.0.0-SNAPSHOT-distro/oozie-5.0.0-SNAPSHOT/share/lib/oozie/oozie-sharelib-oozie-5.0.0-SNAPSHOT.jar
{code}
{code}
bin/oozie job -oozie http://localhost:11000/oozie -config
examples/apps/pig/job.properties -run -DnameNode=hdfs://localhost:9000
-DjobTracker=localhost:8032
{code}
Generated YARN launcher shell script,
{{nm-local-dir/nmPrivate/application_1499613972796_0007/container_1499613972796_0007_01_000001/launch_container.sh}}:
{code}
export
CLASSPATH="$PWD:$PWD/*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/hadoop/common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_YARN_HOME/share/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:file:/Users/asasvari/workspace/apache/oozie_dup/distro/target/oozie-5.0.0-SNAPSHOT-distro/oozie-5.0.0-SNAPSHOT/share/lib/pig/hive-hcatalog-pig-adapter-1.2.0.jar:file:/Users/asasvari/workspace/apache/oozie_dup/distro/target/oozie-5.0.0-SNAPSHOT-distro/oozie-5.0.0-SNAPSHOT/share/lib/pig/jetty-all-7.6.0.v20120127.jar
...
{code}
Pig jars are not to the distributed cache.
> Ability to use local paths for the sharelib
> -------------------------------------------
>
> Key: OOZIE-2601
> URL: https://issues.apache.org/jira/browse/OOZIE-2601
> Project: Oozie
> Issue Type: New Feature
> Affects Versions: oya
> Reporter: Robert Kanter
> Assignee: Attila Sasvari
> Fix For: oya
>
> Attachments: OOZIE-2601-01.patch
>
>
> With OOZIE-2590, as part of OOZIE-1770 Oozie on Yarn work, Oozie now has full
> control over the classpath given to the Launcher AM. In a cluster where all
> nodes have everything installed locally (in the same paths), it should be
> possible to have the Launcher AM reference the local jars instead of having
> to localize them from HDFS.
> For example, if you have Hive installed on all nodes at {{/usr/lib/hive/}}
> and all Hive jars under {{/usr/lib/hive/lib/}}, we could have the Launcher AM
> add {{/usr/lib/hive/lib}} to its classpath. This saves on the overhead of
> localizing the same jars from the hive sharelib in HDFS.
> I think the best way to implement this is to augment the [Sharelib Mapping
> File|https://oozie.apache.org/docs/4.2.0/AG_Install.html#Oozie_Share_Lib]
> feature to accept {{file:///}} paths.
> If we had this also work with the "oozie" sharelib and the Oozie jars in the
> individual sharelibs (e.g. have the Mapping file take comma-separated
> dirs/jars), then in a cluster with everything installed on all of the nodes,
> you wouldn't need to bother with the sharelib at all!
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
