[
https://issues.apache.org/jira/browse/YARN-1151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16419936#comment-16419936
]
Robert Kanter commented on YARN-1151:
-------------------------------------
Overall seems good. A few comments/concerns:
# It looks like it currently only supports archives because it's using
{{LocalResourceType.ARCHIVE}} and then also {{new Path(download.call() +
Path.SEPARATOR + "*");}} for the classpath. If it's a regular jar, there's no
reason to extract it.
# There's no documentation. It looks like there's no documentation on the
original local version either :/ I did find YARN-6412 was filed to add docs
though. At the very least, we should update YARN-6412 to also mention that
{{yarn.nodemanager.aux-services.%s.remote-classpath}} needs to be documented.
# [~vinodkv] asked about this in a previous comment: currently the NM will
repeatedly download the tar again and again on every restart. We should try to
figure out a way not to do that.
# We also need a way for these files to be cleaned up if they're not being used
anymore. For instance, if the jar's name is versioned (e.g.
"my-aux-service-1.1.jar"), then every time you update the jar and restart the
NM, it's going to create new local files and never clean up the old ones. We
can probably have it delete anything in the {{nmAuxService}} dir that it's not
loading.
# We need some unit tests.
> Ability to configure auxiliary services from HDFS-based JAR files
> -----------------------------------------------------------------
>
> Key: YARN-1151
> URL: https://issues.apache.org/jira/browse/YARN-1151
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager
> Affects Versions: 2.1.0-beta, 2.9.0
> Reporter: john lilley
> Assignee: Xuan Gong
> Priority: Major
> Labels: auxiliary-service, yarn
> Attachments: YARN-1151.1.patch, YARN-1151.2.patch, YARN-1151.3.patch,
> YARN-1151.branch-2.poc.2.patch, YARN-1151.branch-2.poc.3.patch,
> YARN-1151.branch-2.poc.patch, [YARN-1151] [Design] Configure auxiliary
> services from HDFS-based JAR files.pdf
>
>
> I would like to install an auxiliary service in Hadoop YARN without actually
> installing files/services on every node in the system. Discussions on the
> user@ list indicate that this is not easily done. The reason we want an
> auxiliary service is that our application has some persistent-data components
> that are not appropriate for HDFS. In fact, they are somewhat analogous to
> the mapper output of MapReduce's shuffle, which is what led me to
> auxiliary-services in the first place. It would be much easier if we could
> just place our service's JARs in HDFS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
