[
https://issues.apache.org/jira/browse/YARN-1151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410206#comment-16410206
]
Wangda Tan commented on YARN-1151:
----------------------------------
Thanks [~xgong],
Inside AuxServices#init:
- Rename {{appClassPath}} to {{appLocalClassPath}}
- Instead of warning:
{code}
// load AuxiliaryService from remote class path
if (appClassPath != null && !appClassPath.isEmpty()) {
LOG.warn("The aux serivce:" + sName + " has local classpath:"
+ appClassPath + " and remote classpath:"
+ appRemoteClassPath + ". Using remote classpath.");
}
{code}
Is it better to throw exception?
- Do we require this directory (parent dir of all aux service classpathes)
created prior to using this feature?
{code}
Path(dirsHandler.getLocalPathForWrite(
"." + Path.SEPARATOR + NM_AUX_SERVICE_DIR)
{code}
- Instead of hard coding remoteFs, is it better to load FS according to URI?
For example user can put aux tars to HDFS/S3/WASB, etc:
{code}
this.remoteLFS = getRemoteFileContext(this.conf);
{code}
- Could you add a comment about why {{LocalResourceVisibility.APPLICATION}} is
being used. I can understand it is not PUBLIC for sure. But maybe {{PRIVATE}}
should be used?
- Maybe it's better to cache exception here:
{code}
Path dest = new Path(download.call() + Path.SEPARATOR + "*");
{code}
And add some information, like "exception happend while downloading files for
aux-service=x and remote-file-path=y, etc."
- Following code is commented:
{code}
// createAuxServiceDir(className);
{code}
Add by mistake?
- IIUC, local directories of AuxService is initialized inside
{{ResourceLocalizationService}}, we need to make sure
{{ResourceLocalizationService}} is initialized prior to AuxServices.
> Ability to configure auxiliary services from HDFS-based JAR files
> -----------------------------------------------------------------
>
> Key: YARN-1151
> URL: https://issues.apache.org/jira/browse/YARN-1151
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager
> Affects Versions: 2.1.0-beta, 2.9.0
> Reporter: john lilley
> Assignee: Xuan Gong
> Priority: Major
> Labels: auxiliary-service, yarn
> Attachments: YARN-1151.1.patch, YARN-1151.branch-2.poc.patch,
> [YARN-1151] [Design] Configure auxiliary services from HDFS-based JAR
> files.pdf
>
>
> I would like to install an auxiliary service in Hadoop YARN without actually
> installing files/services on every node in the system. Discussions on the
> user@ list indicate that this is not easily done. The reason we want an
> auxiliary service is that our application has some persistent-data components
> that are not appropriate for HDFS. In fact, they are somewhat analogous to
> the mapper output of MapReduce's shuffle, which is what led me to
> auxiliary-services in the first place. It would be much easier if we could
> just place our service's JARs in HDFS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
