[
https://issues.apache.org/jira/browse/YARN-1151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15283701#comment-15283701
]
Vinod Kumar Vavilapalli commented on YARN-1151:
-----------------------------------------------
General thought: Instead of a new bootstrap-path configuration, how about we
simply extend the existing {{.%s.classpath}} property to also support other
Hadoop FileSystem schemes?
In that case, the user can set, for e.g.,
{{mapreduce_shuffle.classpath=/my/path/to/aux/service/mapreduce-shuffle.jar:/my/path/to/aux/service/dependency.jar}}
assuming we keep the default scheme for classpath to be the local FS, and when
he/she wants to configure via HDFS, it can be set like this:
{{mapreduce_shuffle.classpath=hdfs:///my/remotepath/to/aux/service/mapreduce-shuffle.jar:hdfs:///my/remotepath/to/aux/service/dependency.jar}}.
One can mix and match file-systems too, but that may be a bit too much.
Thoughts? /cc [~hitesh] [~sjlee0]
> Ability to configure auxiliary services from HDFS-based JAR files
> -----------------------------------------------------------------
>
> Key: YARN-1151
> URL: https://issues.apache.org/jira/browse/YARN-1151
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager
> Affects Versions: 2.1.0-beta, 2.9.0
> Reporter: john lilley
> Assignee: Xuan Gong
> Labels: auxiliary-service, yarn
> Attachments: YARN-1151.1.patch
>
>
> I would like to install an auxiliary service in Hadoop YARN without actually
> installing files/services on every node in the system. Discussions on the
> user@ list indicate that this is not easily done. The reason we want an
> auxiliary service is that our application has some persistent-data components
> that are not appropriate for HDFS. In fact, they are somewhat analogous to
> the mapper output of MapReduce's shuffle, which is what led me to
> auxiliary-services in the first place. It would be much easier if we could
> just place our service's JARs in HDFS.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
