[
https://issues.apache.org/jira/browse/YARN-1151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16409013#comment-16409013
]
Xuan Gong commented on YARN-1151:
---------------------------------
Uploaded a poc patch for the fix.
In the fix, we created a new configuration:
"yarn.nodemanager.aux-services.%s.remote-classpath". The value for this
configuration would be the path of the jar file in HDFS. Currently, we only
support the Archive file format (zip, tar, jar,etc).
The basic workflow will be:
* At the beginning of NM starts, it would create a sub-dir named
(nmAuxService) under the NM local directories, and the permission would be 700
* When we initiate the AuxServices, we would download the jar(zip, tar) files
from HDFS (the location is configurated by the newly added configuration), and
we will automatically unpack the files into the path
${NM_LOCAL}/nmAuxService/${aux_class_name}
> Ability to configure auxiliary services from HDFS-based JAR files
> -----------------------------------------------------------------
>
> Key: YARN-1151
> URL: https://issues.apache.org/jira/browse/YARN-1151
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager
> Affects Versions: 2.1.0-beta, 2.9.0
> Reporter: john lilley
> Assignee: Xuan Gong
> Priority: Major
> Labels: auxiliary-service, yarn
> Attachments: YARN-1151.1.patch, YARN-1151.branch-2.poc.patch,
> [YARN-1151] [Design] Configure auxiliary services from HDFS-based JAR
> files.pdf
>
>
> I would like to install an auxiliary service in Hadoop YARN without actually
> installing files/services on every node in the system. Discussions on the
> user@ list indicate that this is not easily done. The reason we want an
> auxiliary service is that our application has some persistent-data components
> that are not appropriate for HDFS. In fact, they are somewhat analogous to
> the mapper output of MapReduce's shuffle, which is what led me to
> auxiliary-services in the first place. It would be much easier if we could
> just place our service's JARs in HDFS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
